Secondary visual inspection apparatus, visual inspection system, and secondary visual inspection method

Abstract

To suppress generation of wrong determinations when a machine is used to execute a secondary determination of a target object.SOLUTION: There is acquired a primary determination result R showing a defective region An of a plurality of inspection regions A(N) which has been determined to be defective by a primary determination (step S201). The abnormality degree which shows an abnormality in a solder image Is of a solder S (a target object) in the defective region An shown by the primary determination result R, that is, the abnormality degree of a defective region image In is calculated. On the basis of the abnormality degree, the state of the solder S is determined (a secondary determination).SELECTED DRAWING: Figure 5

Description

Technical Field

[0004] , , , , , ,

[0001] The present invention relates to a technique for accurately determining the state of an object by performing a secondary determination for determining the state of the object on an object determined to be defective by a primary determination for determining the state of the object in each of a plurality of inspection regions.

Background Art

[0002] There is known an appearance inspection apparatus that determines the state of solder by imaging the solder joining a component to a substrate. In such an appearance inspection apparatus, in order to determine the state of the solder in detail, determination may be performed for each of a plurality of different inspection regions. For example, three inspection regions having different distances from the electrodes of the component are set. In the inspection region closest to the electrode, the presence or absence of a solder shape defect (unsoldered) in which the solder is recessed in front of the electrode of the component is determined. In the inspection region second closest to the electrode, the presence or absence of a solder shortage defect (small solder) in which the amount of solder is insufficient is determined. Further, in the inspection region third closest to the electrode, the presence or absence of a metal foil exposure defect (red eye) in which the metal foil (copper foil) to be covered by the solder is exposed is determined.

[0003] However, in such an appearance inspection apparatus, there are cases where it is misjudged as defective even though it is actually good. Therefore, a secondary determination that complements the determination (primary determination) in the appearance inspection apparatus is appropriately performed by the user. That is, in the primary determination of the appearance inspection apparatus, when it is determined that there is a defect in any one of the plurality of inspection regions, an image or the like obtained by imaging the solder that is the target of the primary determination is displayed on the display. As a result, the user can perform a final determination (secondary determination) based on visual inspection using the display.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] Incidentally, it is conceivable that the secondary judgment, which was previously performed by the user, could be automated to reduce manpower. For example, Patent Document 1 proposes a neural network that determines the state of an object based on abnormalities in pixels contained in the image of the object. However, there were the following problems when it came to having a machine perform the secondary judgment that complements the primary judgment by the visual inspection device.

[0006] In other words, the secondary judgment performed by the user is intended to confirm the status of the inspection areas that were determined to be defective in the primary judgment among multiple inspection areas. However, if the image of the object contains multiple inspection areas, the machine may calculate the abnormalities contained in the image, which could lead to over-judgment, where inspection areas that were not determined to be defective in the primary judgment are incorrectly judged as defective.

[0007] This invention has been made in view of the above-mentioned problems, and aims to suppress the occurrence of over-judgments when a machine performs a secondary judgment to determine the state of an object that has been determined to be defective by a primary judgment that determines the state of the object in each of multiple inspection areas. [Means for solving the problem]

[0008] The secondary visual inspection apparatus according to the present invention comprises: an information acquisition unit that acquires defective area information indicating a defective area among a plurality of inspection areas in a primary determination that determines the state of an object in each of a plurality of different inspection areas; an abnormality data calculation unit that calculates defective area abnormality data indicating abnormalities contained in the image of the object in the defective area indicated by the defective area information; and a secondary determination execution unit that performs a secondary determination that determines the state of the object based on the defective area abnormality data.

[0009] The visual inspection system according to the present invention comprises a primary visual inspection device that performs a primary determination to determine the state of an object in each of a plurality of different inspection areas and outputs defective area information indicating the defective area determined to be defective in the primary determination, and the secondary visual inspection device described above, and the information acquisition unit acquires the defective area information output from the primary visual inspection device.

[0010] The secondary visual inspection method according to the present invention comprises the steps of: acquiring defective area information indicating a defective area among a plurality of inspection areas, in a primary determination in which the state of an object in each of a plurality of different inspection areas is determined to be in a defective area; calculating defective area anomaly data indicating anomalies contained in the image of the object in the defective area indicated by the defective area information; and performing a secondary determination in which the state of the object is determined based on the defective area anomaly data.

[0011] In the present invention (secondary visual inspection apparatus, visual inspection system, and secondary visual inspection method) configured as described above, defective area information is acquired that indicates the defective area among multiple inspection areas that was determined to be defective in the primary inspection. Then, defective area anomaly data is calculated that indicates the abnormality contained in the image of the object in the defective area indicated by the defective area information, and the state of the object is determined based on this defective area anomaly data (secondary inspection). In other words, inspection areas that were not determined to be defective in the primary inspection are excluded from the criteria for determining the state of the object in the secondary inspection. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas that were not determined to be defective in the primary inspection are judged to be defective.

[0012] Furthermore, the abnormality data calculation unit may be configured such that the secondary visual inspection device includes an image acquisition unit that acquires an object image, which is an image of the object in an imaging area containing multiple inspection areas, captured in the primary judgment; multiple machine learning models, each provided for a multiple inspection area; and a data calculation unit that calculates defective area abnormality data using a corresponding model from among the multiple machine learning models that corresponds to the defective area. Each of the multiple machine learning models outputs inspection area abnormality data indicating abnormalities contained in the image captured in the corresponding inspection area, and the data calculation unit extracts images contained in the defective area from the object image as defective area images, and calculates the inspection area abnormality data output by the corresponding model for the defective area images as defective area abnormality data. In such a configuration, inspection areas that were not judged as defective in the primary judgment are excluded from the criteria for determining the state of the object in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas that were not judged as defective in the primary judgment are judged as defective.

[0013] Furthermore, the abnormality data calculation unit may be configured such that it includes an image acquisition unit that acquires an object image, which is an image of the object in an imaging area containing multiple inspection areas, captured in the primary judgment; a machine learning model that outputs imaging area abnormality data indicating abnormalities contained in the image captured in the imaging area; and a data calculation unit that calculates defective area abnormality data by extracting data corresponding to defective areas from the imaging area abnormality data output by the machine learning model for the object image. In such a configuration, inspection areas that were not judged as defective in the primary judgment are excluded from the criteria for determining the state of the object in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas that were not judged as defective in the primary judgment are judged as defective. Moreover, instead of providing multiple machine learning models corresponding to multiple inspection areas, it is sufficient to provide one machine learning model for the imaging area containing multiple inspection areas. Therefore, the user does not need to manage multiple machine learning models, and the user's management burden can be reduced.

[0014] Furthermore, the secondary visual inspection apparatus may be configured to further include a user interface and a display control unit that displays visualized abnormal data, including at least defective area abnormal data, on the user interface. In such a configuration, the user can visually confirm the defective area abnormal data included in the visualized abnormal data.

[0015] Furthermore, the secondary visual inspection device may be configured to further include a mode selection unit that selectively executes an automatic judgment mode in which the quality of the object is determined by a secondary judgment execution unit, and a manual judgment mode in which abnormality data is displayed on the user interface and the quality of the object is determined based on operations performed by the user on the user interface. In such a configuration, selecting the automatic judgment mode reduces the burden on the user in the secondary judgment, and selecting the manual judgment mode allows the user to perform a secondary judgment by visual inspection.

[0016] Furthermore, the mode selection unit may be configured to execute one of the automatic and manual judgment modes selected by the user's operation on the user interface. In such a configuration, the system can execute a mode from among the automatic and manual judgment modes according to the user's request. [Effects of the Invention]

[0017] As described above, according to the present invention, when a machine performs a secondary determination to determine the state of an object that has been determined to be defective by a primary determination that determines the state of the object in each of the multiple inspection areas, it is possible to suppress the occurrence of over-determination. [Brief explanation of the drawing]

[0018] [Figure 1] A block diagram showing an example of the visual inspection system according to the present invention. [Figure 2] A flowchart showing an example of a primary judgment performed by the primary visual inspection device in Figure 1. [Figure 3] A diagram schematically showing the operations executed in the primary determination of FIG. 2. [Figure 4] A block diagram showing a first example of a secondary appearance inspection apparatus according to the present invention. [Figure 5] A flowchart showing the secondary determination executed by the first example of the secondary appearance inspection apparatus of FIG. 4. [Figure 6] A diagram schematically showing the operations executed in the secondary determination of FIG. 5. [Figure 7] A block diagram showing a second example of a secondary appearance inspection apparatus according to the present invention. [Figure 8] A flowchart showing the secondary determination executed by the second example of the secondary appearance inspection apparatus of FIG. 7.

Embodiments for Carrying Out the Invention

[0019] FIG. 1 is a block diagram showing an example of an appearance inspection system according to the present invention. The appearance inspection system 1 in FIG. 1 includes a primary appearance inspection apparatus 2 and a secondary appearance inspection apparatus 4, and inspects the state of solder S that joins components E such as capacitors, resistors, or integrated circuits to a substrate B. The primary appearance inspection apparatus 2 has a light irradiation unit 21 and an imaging camera 22, and while irradiating light from the light irradiation unit 21 to a predetermined imaging region Ac including the solder S that is the inspection object, images the imaging region Ac with the imaging camera 22 to image a solder image Is showing the solder S. Further, the primary appearance inspection apparatus 2 has a communication unit 28 that communicates with the secondary appearance inspection apparatus 4, and a controller 29 that controls the light irradiation unit 21, the imaging camera 22, and the communication unit 28. The controller 29 acquires a primary determination result R that is a result of determining the quality of the solder S based on the solder image Is. Also, the communication unit 28 transmits the solder image Is and the primary determination result R to the secondary appearance inspection apparatus 4.

[0020] As for the specific configuration of the primary visual inspection apparatus 2, for example, the apparatus configuration disclosed in Japanese Patent Publication No. 2010-071844 can be adopted. In the apparatus of that publication, light with different wavelengths (infrared, red, green, and blue) is irradiated onto the solder at different angles, and a camera facing the substrate captures the light reflected by the solder, thereby acquiring an image. The quality of the solder is then determined based on this image. The specific configuration of the primary visual inspection apparatus 2 is not limited to this example, and for example, the apparatus configuration described in WO2018 / 163278 can also be adopted.

[0021] Figure 2 is a flowchart showing an example of a primary judgment performed by the primary visual inspection device in Figure 1, and Figure 3 is a schematic diagram showing the operations performed in the primary judgment in Figure 2. In Figure 3, only component E is shown, and solder S is not shown.

[0022] In the initial assessment shown in Figure 2, multiple different inspection areas A(N) (N=1, 2, 3) are set, as shown in Figure 3. Inspection area A(1) is set to determine whether there is a solder shape defect (unsoldered) where the solder S is indented in front of the electrode of component E; inspection area A(2) is set to determine whether there is a solder shortage defect (slight soldering) where the amount of solder S is insufficient; and inspection area A(3) is set to determine whether there is a metal foil exposure defect (red eye) where the metal foil (copper foil) that should be covered by the solder S is exposed.

[0023] As shown in Figure 2, in step S101, the controller 29 captures a solder image Is by irradiating a predetermined imaging area Ac containing solder S with light from the light irradiation unit 21 and capturing the imaging area Ac with the imaging camera 22. This solder image Is is image data that shows the brightness for each of the multiple pixels included in the imaging area Ac. In step S102, the controller 29 resets the count value N that identifies the inspection area A(N) to zero, and in step S103, the controller 29 increments the count value N by 1.

[0024] The controller 29 then determines whether the condition of inspection area A(N) is good or bad based on the solder image Is that is included in the inspection area A(N) (step S104). The controller 29 also saves the determination result (good / bad) for the inspection area A(N) (step S105). In step S106, it is checked whether the count value N has reached the maximum count value Nx (=3). The maximum count value Nx corresponds to the number of inspection areas A(N) (=3). Steps S104 and S105 are repeated while incrementing the count value N until the count value N reaches the maximum count value Nx. As a result, determination results are obtained for each of the multiple inspection areas A(N) (N=1, 2, 3).

[0025] In step S107, the controller 29 checks whether there is a defect in each of the multiple inspection areas A(N). If there is a defect (if the result is "YES" in step S107), the controller 29 outputs the primary judgment result R, which indicates the defective area An that was judged as defective among the multiple inspection areas A(N), and the solder image Is captured in step S101, to the secondary visual inspection device 4 via the communication unit 28.

[0026] Figure 4 is a block diagram showing a first example of a secondary visual inspection apparatus according to the present invention. The secondary visual inspection apparatus 4 is a computer equipped with a calculation unit 41, a storage unit 42, a communication unit 43, and a UI (User Interface) 44. The calculation unit 41 is a CPU (Central The processor is a Processing Unit (PMS), and the memory unit 42 is a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). The communication unit 43 communicates with the communication unit 28 of the primary visual inspection device 2, and receives, for example, the primary judgment result R and solder image Is output from the communication unit 28 and stores them in the memory unit 42. The UI 44 has input devices such as a mouse and keyboard, and output devices such as a display. Note that the input and output devices of the UI 44 do not need to be configured separately, and they may be integrated using a touch panel display.

[0027] The calculation unit 41 executes a predetermined secondary judgment program to configure the information acquisition unit 411, image acquisition unit 412, data calculation unit 413, automatic judgment execution unit 414, manual judgment control unit 415, and mode selection unit 416. The storage unit 42 is provided with multiple inspection areas A(N) corresponding to each of the multiple inspection areas A(N) Machine learning models Save M(N) (N=1, 2, 3). Machine learning models M(N) is the part of the solder image Is that Machine learning models The relationship between the inspection region image Ia(N) (Figure 3), which is an image within the inspection region A(N) corresponding to M(N), and the abnormality score of that inspection region image Ia(N) has already been learned. When an inspection region image Ia(N) is input, the abnormality score of that inspection region image Ia(N) is output. In other words, Machine learning models M(1) outputs the degree of abnormality in the inspection area A(1) and can be used to determine the presence or absence of solder shape defects (unsoldered areas). Machine learning models M(2) outputs the degree of abnormality in inspection area A(2) and can be used to determine the presence or absence of solder deficiency (small solder). Machine learning models M(3) outputs the degree of abnormality in inspection area A(3) and can be used to determine the presence or absence of metal foil exposure defects (red eye).

[0028] Furthermore, the training method for the machine learning model that outputs the image anomaly score can employ well-known techniques. For example, the feature vector of the intermediate layer of an existing machine learning model obtained when a good product image is input can be saved as a reference. Then, the anomaly score can be calculated by the distance between the feature vector obtained when an image captured by the primary visual inspection device 2 is input to this machine learning model and the reference feature vector. This anomaly score can be calculated for each of the multiple pixels that make up the image.

[0029] Figure 5 is a flowchart showing the secondary determination performed by the first example of the secondary visual inspection apparatus in Figure 4, and Figure 6 is a schematic diagram showing the operations performed in the secondary determination in Figure 5. Note that in Figure 6, only component E is shown, and solder S is not shown.

[0030] As shown in Figure 5, the information acquisition unit 411 acquires the primary judgment result R received by the communication unit 43 from the primary visual inspection device 2 (step S201), and the image acquisition unit 412 acquires the solder image Is received by the communication unit 43 from the primary visual inspection device 2 (step S202). As illustrated in the column for step S202 in Figure 6, this solder image Is shows the solder S captured in the imaging area Ac which includes multiple inspection areas A(N) set in the primary judgment. Note that the execution order of steps S201 and S202 is not limited to this example.

[0031] In step S203, the data calculation unit 413 extracts a defective area image In from the solder image Is, which is the image of the defective area An indicated by the primary judgment result R. In the example shown in the step S203 column of Figure 6, among the multiple inspection areas A(1), A(2), and A(3), inspection area A(2) corresponds to the defective area An, and the defective area image In of the defective area An is extracted from the solder image Is. Furthermore, the data calculation unit 413 extracts images corresponding to each of the multiple inspection areas A(1), A(2), and A(3). Machine learning models Of M(1), M(2), and M(3), the one corresponding to the defective area An (i.e., inspection area A(2)) Machine learning models Select M(2) (step S204). Then, the data calculation unit 413 selects the option selected in step S204. Machine learning models By inputting the defective region image In to M(2), the degree of abnormality of the defective region image In is calculated.

[0032] In step S206, the mode selection unit 416 confirms whether to execute an automatic determination mode in which the automatic determination execution unit 414 automatically performs the secondary determination, or to execute a manual determination mode in which the manual determination control unit 415 performs the secondary determination based on the user's input operation to the UI 44. Specifically, the user can set which of the automatic determination mode and manual determination mode to execute in the mode selection unit 416 by operating the UI 44.

[0033] If the automatic determination mode is set (if "YES" is selected in step S206), the automatic determination execution unit 414 determines the quality of the solder S based on the abnormality level of the defective area image In calculated in step S205 (step S207). Specifically, if the abnormality level of each pixel constituting the defective area image In is below a threshold (if "YES" is selected in step S207), the automatic determination execution unit 414 obtains a good product determination as the result of the secondary determination, indicating that the solder S shown in the solder image Is is in good condition (step S208). On the other hand, if there is a pixel among the pixels constituting the defective area image In whose abnormality level is greater than the threshold (if "NO" is selected in step S207), the automatic determination execution unit 414 obtains a defective product determination as the result of the secondary determination, indicating that the solder S shown in the solder image Is is in poor condition (step S209). The results of the secondary determination obtained in steps S208 and S209 are displayed, for example, on the UI 44 display.

[0034] If manual judgment mode is set (if "NO" is selected in step S206), the manual judgment control unit 415 displays an abnormality map (heatmap) that visualizes the degree of abnormality of the defective area image In on the UI 44 display (step S210). The manual judgment control unit 415 then determines whether the user's judgment entered into the UI 44 is good or not (step S211). If the user's judgment entered into the UI 44 is good (if "YES" is selected in step S211), the manual judgment control unit 415 obtains a good product judgment as a result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is good (step S208). On the other hand, if the user's judgment entered into the UI 44 is bad (if "NO" is selected in step S211), the manual judgment control unit 415 obtains a bad product judgment as a result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is bad (step S209). The results of the secondary determination obtained in steps S208 and S209 are displayed, for example, on the UI44 display.

[0035] In the embodiment described above, a primary judgment result R (defective area information) is obtained that indicates a defective area An (inspection area A(2)) that was determined to be defective in the primary judgment among a plurality of inspection areas A(N) (step S201). Then, an abnormality degree (defective area abnormality data) indicating an abnormality contained in the solder image Is of the solder S (object) in the defective area An indicated by the primary judgment result R, i.e., the abnormality degree of the defective area image In, is calculated (step S205). In steps S206 to S209, the state of the solder S is determined based on this abnormality degree (secondary judgment). In other words, inspection areas A(1) and A(3), which were not determined to be defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and A(3), which were not determined to be defective in the primary judgment, are judged to be defective.

[0036] In particular, the solder image Is (object image), which is an image of the solder S in the imaging area Ac that includes multiple inspection areas A(N) captured in the primary determination, is acquired by the information acquisition unit 411 (step S202). In response to this, multiple inspection areas A(n) are each provided with a plurality Machine learning models M(N) is stored in the memory unit 42. And multiple Machine learning models Among M(N), the defective area image In (inspection area A(2)) corresponds to Machine learning models Using M(2) (corresponding model), the abnormality degree (defective region abnormality data) of the defective region image In is calculated by the data calculation unit 413 (steps S203 to S205). In other words, multiple Machine learning models Each of M(N) is configured to output the degree of abnormality (inspection area abnormality data) contained in the inspection area image Ia(N) captured in the corresponding inspection area A(N). Therefore, the data calculation unit 413 extracts the image contained in the defective area An (inspection area A(2)) from the solder image Is as the defective area image In (step S203), and the image corresponding to the defective area An Machine learning modelsThe abnormality score (inspection area abnormality data) output by M(2) (corresponding model) for the defective area image In is calculated as the abnormality score (defective area abnormality data) for the defective area image In (steps S204, S205). In this configuration, inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are excluded from the criteria for determining the state of solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment in which inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are judged as defective.

[0037] Furthermore, the system includes a UI44 and a manual judgment control unit 415 (display control unit) that displays an abnormality map (visualized abnormality data) on the UI44, which includes at least the degree of abnormality (abnormality data of the abnormal region) in the image In of the defective region. In this configuration, the user can visually confirm the degree of abnormality in the image In of the defective region included in the abnormality map.

[0038] Furthermore, the system is equipped with a mode selection unit 416 that selectively executes an automatic judgment mode (steps S207, S208, S209) in which the quality of the solder S is determined by a secondary judgment performed by an automatic judgment execution unit 414 (secondary judgment execution unit), and a manual judgment mode (steps SS210, S211, S208, S209) in which the quality of the solder S is determined based on the user's operations on the UI 44 while displaying an abnormality map on the UI 44. In this configuration, selecting the automatic judgment mode reduces the burden on the user in the secondary judgment, while selecting the manual judgment mode allows the user to perform a secondary judgment by visual inspection.

[0039] Furthermore, the mode selection unit 416 executes one of the automatic determination mode and manual determination mode selected by the user's operation on the UI 44 (step S206). In this configuration, the system can execute a mode from among the automatic determination mode and manual determination mode according to the user's request.

[0040] Figure 7 is a block diagram showing a second example of the secondary visual inspection apparatus according to the present invention, and Figure 8 is a flowchart showing the secondary determination performed by the second example of the secondary visual inspection apparatus in Figure 7. In the following, the explanation will focus on the differences from the first example, and common points with the first example will be denoted by corresponding reference numerals and explanations will be omitted as appropriate. However, it goes without saying that having a configuration common to the first example will produce the same effect as the first example.

[0041] In the storage unit 42 of the primary visual inspection device 2 shown in Figure 7, Machine learning models Mc is preserved here. Machine learning models Mc is provided to correspond to imaging region Ac which includes multiple examination regions A(N). In other words, Machine learning models Mc has already learned the relationship between the solder image Is captured in the imaging region Ac and the anomaly score of that solder image Is. When a solder image Is is input, Mc outputs the anomaly score of that solder image Is.

[0042] As shown in Figure 8, the information acquisition unit 411 acquires the primary judgment result R received by the communication unit 43 from the primary visual inspection device 2 (step S301), and the image acquisition unit 412 acquires the solder image Is received by the communication unit 43 from the primary visual inspection device 2 (step S302). Note that the execution order of steps S301 and S302 is not limited to this example.

[0043] In step S303, the data calculation unit 413 performs the following: Machine learning models The solder image Is is input to Mc to calculate the degree of abnormality of the solder image Is. Furthermore, the data calculation unit 413 extracts the degree of abnormality in the defective region An indicated by the primary judgment result R from the degree of abnormality calculated in step S303 (step S304).

[0044] In step S306, the mode selection unit 416 checks whether automatic judgment mode or manual judgment mode is set. If automatic judgment mode is set (if "YES" is selected in step S306), the automatic judgment execution unit 414 determines whether the solder S is good or bad based on the degree of abnormality in the defective region An extracted in step S304 (step S307). In other words, if the degree of abnormality of each pixel included in the defective region An is below the threshold (if "YES" is selected in step S307), the automatic judgment execution unit 414 obtains a good product judgment as the result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is good (step S308). On the other hand, if there is a pixel among the pixels included in the defective region An whose degree of abnormality is greater than the threshold (if "NO" is selected in step S307), the automatic judgment execution unit 414 obtains a bad product judgment as the result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is bad (step S309). The results of the secondary determination obtained in steps S308 and S309 are displayed, for example, on the UI44 display.

[0045] If manual judgment mode is set (if "NO" is selected in step S306), steps S310, S311, S308, and S309 are executed in the same manner as steps S210, S211, S208, and S209 in the first example.

[0046] In the embodiment described above, a primary judgment result R (defective area information) is obtained indicating a defective area An (inspection area A(2)) that was determined to be defective in the primary judgment among multiple inspection areas A(N) (step S301). Then, the degree of image abnormality (defective area abnormality data) in the defective area An indicated by the primary judgment result R is calculated from the solder image Is (steps S303, S304). Then, in steps S307 to S309, the state of the solder S is determined based on the degree of image abnormality (defective area abnormality data) in the defective area An (secondary judgment). In other words, inspection areas A(1) and S(3), which were not determined to be defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and S(3), which were not determined to be defective in the primary judgment, are judged to be defective.

[0047] In particular, the image acquisition unit 412 acquires a solder image Is (object image), which is an image of the solder S in the imaging area Ac, which includes multiple inspection areas A(N), captured in the primary determination. In response to this, it outputs the degree of abnormality of the image captured in imaging area Ac (imaging area abnormality data). Machine learning models Mc is stored in memory unit 42. Then, data calculation unit 413, Machine learning models From the abnormality score (imaging area abnormality data) output by Mc for the solder image Is, the abnormality score (defective area abnormality data) in the defective area An is calculated by extracting the abnormality score (data) corresponding to the defective area An (steps S303, S304). In this configuration, inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are excluded from the criteria for determining the state of solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are judged as defective. Moreover, multiple inspection areas A(1), A(2), A(3) are corresponding to multiple Machine learning models Instead of providing M(1), M(2), and M(3), a single imaging region Ac containing multiple examination regions A(1), A(2), and A(3) is used. Machine learning modelsIt is sufficient to set up Mc. Therefore, users do not need to manage multiple machine learning models M(1), M(2), and M(3), which reduces the management burden on users.

[0048] As described above, in this embodiment, the appearance inspection system 1 corresponds to an example of the "appearance inspection system" of the present invention, the primary appearance inspection device 2 corresponds to an example of the "primary appearance inspection device" of the present invention, the secondary appearance inspection device 4 corresponds to an example of the "secondary appearance inspection device" of the present invention, the information acquisition unit 411 corresponds to an example of the "information acquisition unit" of the present invention, the image acquisition unit 412, Machine learning models M(N), Mc, and the data calculation unit 413 work together to function as the "abnormal data calculation unit" of the present invention, the automatic judgment execution unit 414 corresponds to an example of the "secondary judgment execution unit" of the present invention, the inspection area A(N) corresponds to an example of the "inspection area" of the present invention, the defective area An corresponds to an example of the "defective area" of the present invention, the degree of abnormality in the defective area An of the solder image Is corresponds to an example of the "defective area abnormal data" of the present invention, the primary judgment result R corresponds to an example of the "defective area information" of the present invention, and the solder S corresponds to an example of the "object" of the present invention.

[0049] It should be noted that the present invention is not limited to the embodiments described above, and various modifications can be made to those described above without departing from the spirit of the invention. For example, the specific configuration of the multiple inspection areas A(N) in the primary determination is not limited to the examples above. Therefore, two inspection areas A(N) may be set from among inspection areas A(1), A(2), and A(3), or different inspection areas may be set from inspection areas A(1), A(2), and A(3).

[0050] Furthermore, the subjects of the primary and secondary assessments are not limited to the solder S described above.

[0051] Furthermore, the primary visual inspection device 2 and the secondary visual inspection device 4 do not necessarily need to be configured as separate units; they may be configured as an integrated unit.

[0052] Furthermore, the manual judgment mode described above is not mandatory, and it is not necessary to include a manual judgment mode in the secondary judgment. [Explanation of symbols]

[0053] 1…Visual inspection system 2…Primary visual inspection device 4…Secondary visual inspection device 411…Information acquisition department 412...Image acquisition unit (abnormal data calculation unit) 413...Data calculation unit (abnormal data calculation unit) 414...Automatic judgment execution unit (secondary judgment execution unit) A(N)...Inspection area An...Bad area Is... Handa image M(N)... Machine learning models (Anomaly Data Calculation Unit) Mc… Machine learning models (Anomaly Data Calculation Unit) R…Primary judgment result (defective area information) S... Solder (object)

Claims

1. In a primary determination that determines the state of an object in each of several different inspection areas, an information acquisition unit acquires defective area information indicating the defective area that was determined to be defective among the multiple inspection areas. An abnormality data calculation unit calculates abnormality data for a defective region that indicates an abnormality contained in the image of the object in the defective region indicated by the defective region information, A secondary determination execution unit performs a secondary determination to determine the state of the object based on the aforementioned defective area abnormality data. Equipped with, The aforementioned abnormal data calculation unit, An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination; Multiple machine learning models are provided, each corresponding to one of the aforementioned multiple inspection areas, and each outputs inspection area anomaly data indicating anomalies contained in an image that includes only the corresponding inspection area. A data calculation unit that calculates abnormal data for the defective region using a corresponding model among the multiple machine learning models that corresponds to the defective region. It has, The data calculation unit extracts images from the object image that are included in the defective area as defective area images, thereby removing the inspection area that was not determined to be defective in the primary determination from the criteria for determining the state of the object, and using the inspection area abnormality data output by the corresponding model for the defective area image as the defective area abnormality data. The secondary determination execution unit is a secondary visual inspection device that determines the state of the object based on the inspection area abnormality data output by the corresponding model as the defective area abnormality data with respect to the defective area image extracted by the data calculation unit.

2. In a primary determination that determines the state of an object in each of several different inspection areas, an information acquisition unit acquires defective area information indicating the defective area that was determined to be defective among the multiple inspection areas. An abnormality data calculation unit calculates abnormality data for a defective region that indicates an abnormality contained in the image of the object in the defective region indicated by the defective region information, A secondary determination execution unit performs a secondary determination to determine the state of the object based on the aforementioned defective area abnormality data. Equipped with, The aforementioned abnormal data calculation unit, An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination; A machine learning model that outputs imaging region anomaly data, which is an anomaly score obtained for each of the multiple pixels that make up the image captured in the aforementioned imaging region, A data calculation unit extracts data corresponding to the defective area from the imaging area anomaly data output by the machine learning model for the object image, thereby removing the inspection area that was not determined to be defective in the primary judgment from the criteria for determining the state of the object, and using that data as the defective area anomaly data. It has, The secondary determination execution unit is a secondary visual inspection device that determines the state of the object based on the defective area abnormality data, which is the data corresponding to the defective area extracted by the data calculation unit.

3. User interface and A display control unit that displays visualized abnormal data, including at least the abnormal data of the defective area, on the user interface. A secondary visual inspection apparatus according to claim 1 or 2, further comprising:

4. The secondary visual inspection apparatus according to claim 3, further comprising a mode selection unit that selectively executes an automatic determination mode in which the quality of the object is determined by the secondary determination performed by the secondary determination execution unit, and a manual determination mode in which the quality of the object is determined based on operations performed by the user on the user interface while displaying the abnormal data on the user interface.

5. The secondary visual inspection apparatus according to claim 4, wherein the mode selection unit executes one mode selected by user operation on the user interface from among the automatic determination mode and the manual determination mode.

6. A primary visual inspection device that performs a primary determination to determine the state of an object in each of several different inspection areas, and outputs defective area information indicating the defective area among the multiple inspection areas that was determined to be defective in the primary determination, The secondary visual inspection apparatus according to claim 1 or 2 Equipped with, The information acquisition unit is an appearance inspection system that acquires the defective area information output from the primary appearance inspection device.

7. In a primary determination of the state of an object in each of several different inspection areas, the process includes acquiring defective area information indicating the defective area that was determined to be defective among the multiple inspection areas, A step of calculating abnormality data data for a defective area that indicates an abnormality contained in the image of the object in the defective area indicated by the defective area information, using an abnormality data calculation unit. A step in which a secondary determination is performed by a secondary determination execution unit to determine the state of the object based on the aforementioned defective area abnormality data. Equipped with, The aforementioned abnormal data calculation unit, An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination; Multiple machine learning models are provided, each corresponding to one of the aforementioned multiple inspection areas, and each outputs inspection area anomaly data indicating anomalies contained in an image that includes only the corresponding inspection area. A data calculation unit that calculates abnormal data for the defective region using a corresponding model among the multiple machine learning models that corresponds to the defective region. It has, The data calculation unit extracts images from the object image that are included in the defective area as defective area images, thereby removing the inspection area that was not determined to be defective in the primary determination from the criteria for determining the state of the object, and using the inspection area abnormality data output by the corresponding model for the defective area image as the defective area abnormality data. The secondary determination execution unit determines the state of the object based on the inspection area abnormality data output by the corresponding model as the defective area abnormality data with respect to the defective area image extracted by the data calculation unit, as a secondary visual inspection method.

8. In a primary determination of the state of an object in each of several different inspection areas, the process includes acquiring defective area information indicating the defective area that was determined to be defective among the multiple inspection areas, A step of calculating abnormality data data for a defective area that indicates an abnormality contained in the image of the object in the defective area indicated by the defective area information, using an abnormality data calculation unit. A step in which a secondary determination is performed by a secondary determination execution unit to determine the state of the object based on the aforementioned defective area abnormality data. Equipped with, The aforementioned abnormal data calculation unit, An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination; A machine learning model that outputs imaging region anomaly data, which is an anomaly score obtained for each of the multiple pixels that make up the image captured in the aforementioned imaging region, A data calculation unit extracts data corresponding to the defective area from the imaging area anomaly data output by the machine learning model for the object image, thereby removing the inspection area that was not determined to be defective in the primary judgment from the criteria for determining the state of the object, and using that data as the defective area anomaly data. It has, The secondary determination execution unit determines the state of the object based on the defective area abnormality data, which is the data corresponding to the defective area extracted by the data calculation unit, as a secondary visual inspection method.

Images