Determining device, learning device, determining method, learning method, determining program, and learning program

Abstract

To reduce false positives in an inspection system. A determination device includes: a learned image reconstruction section learned in a manner of reconstructing a first image judged as not containing a defect from among images obtained by imaging an inspection target, from a first mask image, the first mask image being an image in which a mask overlaid on an inspection region of the first image is overlaid, the mask being a mask overlaid on the inspection region of the first image and colored according to a kind of material contained in a region corresponding to the inspection target; and a determination section that compares a second reconstructed image with a second image obtained by imaging the inspection target to determine whether the second image contains a defect, the second reconstructed image being an image reconstructed by inputting a second mask image to the learned image reconstruction section, the second mask image being an image in which the mask corresponding thereto is overlaid on an inspection region of the second image.

Description

Technical Field

[0001] The present invention relates to a determination device, a learning device, a determination method, a learning method, a determination program, and a learning program. Background Art

[0002] There is an inspection system in which an inspector visually inspects an inspection image obtained by photographing an inspection object such as a printed circuit board, and determines whether the inspection object is a good product or a defective product by inspecting the image determined to contain defects.

[0003] In this inspection system, for example, whether or not a defect is included in an inspection image is determined by applying image generation AI (Artificial Intelligence) that has been learned to reconstruct a normal image.

[0004] This image generation AI reconstructs a reconstructed image from a masked inspection image, which is a mask superimposed on the inspection image. This image is then compared with the inspection image to determine whether the inspection image is normal. Therefore, this image generation AI enables accurate determination even when new types of defects have occurred.

[0005] <Prior Art Literature>

[0006] <Patent Document>

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-114331 Summary of the Invention

[0008] <Problems to be Solved by the Invention>

[0009] On the other hand, in the case of the image generation AI, it may be difficult to reconstruct a normal image as the reconstructed image. In this case, the difference between the inspection image and the reconstructed image becomes large, so even though the inspection image is a normal image, it may be mistakenly judged to contain defects.

[0010] One aspect of the present invention is to reduce misjudgments in an inspection system.

[0011] <Methods used to solve the problem>

[0012] According to one embodiment, the determining device includes:

[0013] a learned image reconstruction unit that is learned by reconstructing a first image determined to contain no defects from an image obtained by photographing an inspection object based on a first mask image, wherein the first mask image is an image in which a mask is superimposed on an inspection area of ​​the first image, the mask being superimposed on the inspection area of ​​the first image and colored according to a type of material contained in an area corresponding to the inspection object; and

[0014] A determination unit compares a second reconstructed image with a second image obtained by photographing the inspection object to determine whether the second image contains defects, wherein the second reconstructed image is an image reconstructed by inputting a second mask image into the learned image reconstruction unit, and the second mask image is an image of the mask corresponding to the inspection area superimposed on the second image.

[0015] <Effects of the Invention>

[0016] It can reduce misjudgments in the inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A This is a diagram for explaining an example of a learning processing method for a comparative example of image generation AI.

[0018] Figure 1B This is a diagram for explaining an example of a determination processing method of a comparative example when using a learned image generation AI to determine whether a defect is included.

[0019] Figure 2A FIG1 is a first diagram showing an example of misjudgment.

[0020] Figure 2B FIG2 is a second diagram showing an example of misjudgment.

[0021] Figure 3A This is a diagram showing an example of a learning processing method for image generation AI in the inspection system according to the first embodiment.

[0022] Figure 3B This is a first diagram showing an example of a determination processing method when using a learned image generation AI to determine whether a defect is included in the inspection system of the first embodiment.

[0023] Figure 3C FIG. 2 is a second diagram showing an example of a determination processing method when using the learned image generation AI to determine whether a defect is included in the inspection system of the first embodiment.

[0024] Figure 4 This is a diagram showing an example of the system configuration of the inspection system in the learning phase of the first embodiment.

[0025] Figure 5 This is a diagram showing an example of the hardware configuration of a learning device.

[0026] Figure 6 This is a diagram showing a specific example of processing performed by a learning data set generating unit of a learning device.

[0027] Figure 7 This is a diagram showing a specific example of processing performed by the learning unit of the learning device.

[0028] Figure 8 This is a diagram showing an example of the system configuration of the inspection system in the inspection phase of the first embodiment.

[0029] Figure 9 This is a diagram showing an example of the hardware configuration of the determination device.

[0030] Figure 10 This is a diagram showing a specific example of processing performed by the inference unit of the determination device.

[0031] Figure 11 This is a flowchart showing the flow of a learning process performed by the learning device of the inspection system according to the first embodiment.

[0032] Figure 12 This is a flowchart showing the flow of determination processing performed by the determination device of the inspection system according to the first embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In addition, in this specification and the accompanying drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and repeated descriptions are omitted.

[0034] [First embodiment]

[0035] <Description of the Learning Processing Method and the Determination Processing Method of the Comparative Example of Image Generation AI>

[0036] As described later, in the inspection system of the first embodiment, in order to reduce erroneous determinations, a learning processing method and a determination processing method that are different from general learning processing methods and determination processing methods are applied to the image generation AI.

[0037] Therefore, the following will first describe a general learning and determination processing method for image generation AI (referred to as the learning and determination processing method of the comparative example). Next, we will list examples of situations in which misjudgments occur in the learning and determination processing method of the comparative example, and then describe the learning and determination processing method for image generation AI in the inspection system of the first embodiment, which can reduce such misjudgments.

[0038] Figure 1A: is a diagram for explaining an example of a learning processing method for a comparative example of image generation AI. Figure 1A As shown, in the case of the learning processing method of the comparative example, the image generation AI 110 performs learning processing according to the following steps.

[0039] An inspection image captured by the inspection system in which the inspection object is judged to be a good product (referred to as a normal image) is obtained, and a masked normal image is generated by superimposing a mask on the inspection area.

[0040] The error is calculated by inputting the generated masked normal image to the image generation AI 110 and comparing the reconstructed image output from the image generation AI 110 with the normal image.

[0041] The model parameters of the image generation AI 110 are updated so that the calculated error is reduced. This process is performed on a plurality of normal images to perform a learning process on the image generation AI 110, and a learned image generation AI is generated.

[0042] Next, a description will be given of a determination processing method of a comparative example in which AI is generated using the generated learned image to determine whether a defect is included in an inspection image.

[0043] Figure 1B This is a diagram for explaining an example of a comparative example determination method when using a learned image generation AI to determine whether a defect is included. Figure 1B As shown in the upper half of , in the case of the comparative example, the learned image generation AI 120 performs determination processing according to the following steps and determines that no defects are contained.

[0044] Generate a masked inspection image by taking an inspection image captured by the inspection system and overlaying a mask on the inspection area.

[0045] The generated masked inspection image is input to the learned image generation AI 120 , and the learned image generation AI 120 outputs a reconstructed image.

[0046] Calculate the error between the reconstructed image output by the learned image generation AI 120 and the inspection image, and minimize the error. Furthermore, the learned image generation AI 120 learns to reconstruct a normal image from the input masked inspection image. Therefore, the reconstructed image output by the learned image generation AI 120 is close to a normal image.

[0047] Therefore, the calculated error is small, and the inspection image can be determined to be close to a normal image. As a result, it can be determined that the inspection image does not contain a defect.

[0048] On the other hand, Figure 1B As shown in the lower half of , in the case of the comparative example, the learned image generation AI 120 performs determination processing according to the following steps and determines that a defect is contained.

[0049] Generate a masked inspection image by taking an inspection image captured by the inspection system and overlaying a mask on the inspection area.

[0050] The generated masked inspection image is input to the learned image generation AI 120 , and the learned image generation AI 120 outputs a reconstructed image.

[0051] Calculate the error between the reconstructed image output by the learned image generation AI 120 and the inspection image, and obtain the result with the larger error. Furthermore, as described above, the learned image generation AI 120 learns to reconstruct a normal image from the input masked inspection image. Therefore, the reconstructed image output by the learned image generation AI 120 is close to a normal image.

[0052] Therefore, the calculated error is large, and it can be determined that the inspection image is an image far from the normal image. As a result, it can be determined that the inspection image contains a defect.

[0053] And, in Figure 1B In this embodiment, for simplicity of description, whether an inspection image contains a defect is determined based on the calculated error size. However, the determination of whether an inspection image contains a defect is not limited to this. For example, an inspection image and a reconstructed image may be subjected to image processing, and the determination may be made based on the result of the image processing. Furthermore, in this embodiment, for simplicity of description, the determination of whether an inspection image contains a defect is made based on the calculated error size.

[0054] <Explanation of Misjudgment>

[0055] Next, an example will be described in which an inspection image that should be determined by the learned image generation AI 120 to contain no defects is erroneously determined to contain defects.

[0056] (1) Example 1

[0057] An example of misjudgment is when the inspection object is made of multiple materials and the inspection area where the mask overlaps presses the boundary between the materials, making it difficult to reconstruct a normal image. Figure 2A Detailed description. Figure 2A FIG1 is a first diagram showing an example of misjudgment.

[0058] exist Figure 2AIn FIG, the inspection image 201 is an example of an inspection image that does not actually contain a defect but is mistakenly judged to contain a defect by the learned image generation AI 120. The inspection object corresponding to the inspection image 201 has the following structure.

[0059] Made of 2 different materials.

[0060] The boundary portion of the 2 materials overlaps with the inspection area where the mask overlaps.

[0061] In the case of the inspection image 201, when the masked inspection image with the mask overlapped is input to the learned image generation AI 120, a reconstructed image 202 can be reconstructed in the learned image generation AI 120. Figure 2A As shown, when the image 202 is reconstructed, the boundary portion between the two materials cannot be properly reconstructed (see reference numeral 203 ).

[0062] In such cases, the error between the reconstructed image and the inspection image becomes large, so the inspection image is mistakenly judged to contain a defect even though it actually does not contain a defect.

[0063] In contrast, in the inspection system of the first embodiment, in order to appropriately reconstruct the boundary portion of the material and reduce misjudgment, material information is added to the mask as color information (details will be described later).

[0064] (2) Example 2

[0065] Another example of misjudgment is when the inspection object is made of multiple materials and the inspection area where the mask overlaps contains a material that is difficult to reconstruct a normal image, making it difficult to reconstruct a normal image. Figure 2B Detailed description. Figure 2B FIG2 is a second diagram showing an example of misjudgment.

[0066] exist Figure 2B In FIG, inspection image 210 is another example of an inspection image that does not actually contain a defect but is mistakenly judged to contain a defect by the learned image generation AI 120. The inspection object corresponding to inspection image 210 has the following structure.

[0067] Made of 2 different materials.

[0068] One of the two materials (the white area in the inspection image 210 ) has a relatively small surface unevenness, while the other material (the hatched area in the inspection image 210 ) has a fine and irregular surface unevenness.

[0069] Of the two materials, the one with fine and irregular surface irregularities corresponds to the base of the inspection object, and the one with fewer surface irregularities corresponds to the portion of the inspection object other than the base (for example, the circuit portion when the inspection object is a printed circuit board).

[0070] In the case of the inspection image 210, when the masked inspection image with the mask superimposed thereon is input to the learned image generation AI 120, a reconstructed image 220 is reconstructed in the learned image generation AI 120. Figure 2B As shown, in the case of reconstructing the image 220, the white area 222 (the area corresponding to the material with less surface unevenness) in the area 211 overlapping the mask is used to reconstruct the same image as the white area 212 (the area corresponding to the material with less surface unevenness) in the inspection image 210.

[0071] On the other hand, in regions 223 and 224 (regions corresponding to materials with fine and irregular surface irregularities), an image similar to that of hatched regions 213 and 214 (regions corresponding to materials with fine and irregular surface irregularities) in inspection image 210 is not reconstructed. Specifically, in regions 223 and 224 (regions corresponding to materials with fine and irregular surface irregularities), the fine and irregular surface irregularities are not reconstructed, and a flat image is reconstructed. In other words, in the case of inspection image 210, the inspection region includes a basement region, which makes it difficult to reconstruct a normal image.

[0072] In such an example, the error between the reconstructed image 220 and the inspection image 210 may increase, and thus the inspection image 210 may be mistakenly judged to contain a defect.

[0073] In contrast, in the inspection system of the first embodiment, in order to suppress the influence of the background area that makes it difficult to reconstruct a normal image and reduce misjudgment, the background area is removed when calculating the error between the reconstructed image 220 and the inspection image 210 (details will be described later).

[0074] <Learning Processing Method for Image Generation AI in Inspection System of First Embodiment>

[0075] The following describes a learning processing method for image generation AI in the inspection system according to the first embodiment. Figure 3A A diagram showing an example of a learning processing method for image generation AI in the inspection system according to the first embodiment.

[0076] and Figure 1A The difference in the learning processing method of the comparative example shown is that Figure 3AIn the case of , material information based on CAD (Computer Aided Design) data is added as color information to the mask used when generating the masked normal image.

[0077] Specifically, in Figure 3A In the case of , for example, a mask is generated according to the following steps.

[0078] a) A normal image is acquired from among inspection images captured by an inspection system, and CAD data of an inspection object corresponding to the acquired normal image is acquired.

[0079] b) Extracting CAD data of the inspection area from the acquired CAD data and identifying the type of material included in the extracted CAD data.

[0080] c) Generate a mask by assigning a different color to each material of a determined type.

[0081] As described above, the inspection system of the first embodiment uses a mask to which material information based on CAD data is added as color information. This allows for proper reconstruction of the material boundary, even when the inspection area overlapped by the mask presses against the material boundary, thus reducing misjudgments.

[0082] The above steps for mask generation are merely examples; other steps can also be used to generate a mask. For example, steps b) and c) can be swapped to generate a mask. Specifically, a mask can be generated by identifying the types of materials included in the acquired CAD data, assigning a different color to each material of the identified type, and then extracting the CAD data of the inspection area from the CAD data assigned the color.

[0083] Next, a description will be given of a determination processing method when determining whether a defect is included using the learned image generation AI in the inspection system of the first embodiment.

[0084] Figure 3B FIG1 is a first diagram showing an example of a determination processing method when using a learned image generation AI to determine whether a defect is included in the inspection system of the first embodiment. Figure 2A The difference is that in Figure 3B In the case of

[0085] ·Add material information based on CAD data as color information to the mask used when generating the inspection image with mask.

[0086] The inspection image and the reconstructed image are compared, and when calculating the error, the error is calculated after removing the background region from each of the inspection image and the reconstructed image.

[0087] like Figure 3B As shown in FIG, even when the boundary portion of two types of materials overlaps with the inspection area where the mask overlaps, by adding material information based on CAD data as color information to the mask overlapping the inspection area, the boundary portion can be properly reconstructed. As a result, when the inspection image does not contain defects ( Figure 3B In the case where the inspection image contains defects ( Figure 3B ), it is possible to appropriately determine the situation where a defect is included.

[0088] In addition, Figure 3B In the case of the inspection image shown, since the base area is not made of a material with fine, irregular surface irregularities, the effect of calculating the error after removing the base area is not significant. Therefore, a detailed description of the error calculation process after removing the base area is omitted here.

[0089] Figure 3C FIG2 is a second diagram showing an example of a determination processing method for determining whether a defect is included by using a learned image to generate AI in the inspection system of the first embodiment. Figure 2B The difference is that Figure 3C In the case of

[0090] ·Add material information based on CAD data as color information to the mask used when generating the inspection image with mask.

[0091] When comparing the inspection image and the reconstructed image and calculating the error, the error is calculated after removing the background region from each of the inspection image and the reconstructed image.

[0092] In addition, Figure 3C In the inspection image shown, the size of the mask overlapping the boundary between the two materials is small, so the effect of adding the material information based on the CAD data as color information is not significant. Therefore, the detailed description of the process of adding the material information based on the CAD data as color information is omitted here. The detailed description will focus on the process of calculating the error after removing the base area.

[0093] like Figure 3C As shown, the process of calculating the error after removing the background area is performed according to the following steps.

[0094] Eliminate the inspection area in the inspection image where the mask is superimposed, that is, the base area determined as the base based on the CAD data.

[0095] Remove the inspection area overlapped with the mask in the reconstructed image, that is, the base area determined as the base based on the CAD data.

[0096] Compare the inspection image after removing the background region with the reconstructed image after removing the background region, and calculate the error.

[0097] Therefore, even when the inspection area overlapped with the mask contains a background area where it is difficult to reconstruct a normal image, the influence of the background area can be suppressed. As a result, when the inspection image does not contain defects ( Figure 3C In the case shown in the upper half of the image), it is possible to appropriately determine that no defects are included. In addition, in the case where defects are included in the inspection image ( Figure 3C ), it can be appropriately determined that defects are contained.

[0098] As described above, in the inspection system of the first embodiment, even if there is an example where it is difficult to reconstruct a normal image,

[0099] Examples where it is difficult to reconstruct a normal image due to the inspection area overlapped by the mask pressing against the boundary portion of the material, or,

[0100] Even in the case where it is difficult to reconstruct a normal image because the inspection area includes a background area where it is difficult to reconstruct a normal image, misjudgment can be reduced.

[0101] <Checking the system structure (learning phase)>

[0102] Next, the system configuration of the inspection system in the learning phase of the first embodiment to which the image generation AI is applied will be described. Figure 4 This is a diagram showing an example of the system configuration of the inspection system in the learning phase of the first embodiment.

[0103] like Figure 4 As shown, the inspection system 400 in the learning stage includes an automatic optical inspection (AOI) device 410 and a learning device 440 .

[0104] The AOI device 410 performs automated visual inspection of printed circuit boards 430. Using a camera, the AOI device 410 scans the printed circuit boards 430 and inspects various items to identify defect candidates. Inspection items performed by the AOI device 410 include, for example, circuit width, circuit spacing, the presence of missing pads / pads, and circuit shorts.

[0105] Inspection images 420 of each area containing defect candidates detected by the AOI device 410 are sent to a learning device 440 and then to an inspection line. On the inspection line, an inspector 421 or the like visually inspects the inspection images 420 of each area containing defect candidates. Here, it is assumed that the AOI device 410 is configured to perform redundant inspections of inspection images of each area containing defect candidates to avoid mistaking defective products for good products.

[0106] Inspector 421 or the like visually inspects each region of inspection image 420 for defects and ultimately determines whether printed circuit board 430 is a good or defective product. Specifically, if none of the inspection images 420 for regions containing defect candidates contain defects, printed circuit board 430 is determined to be a good product. Alternatively, if any of the inspection images 420 for regions containing defect candidates contain defects, printed circuit board 430 is determined to be a defective product.

[0107] Inspector 421 or the like also notifies learning device 440 of the results of the visual inspection (the results of determining whether defects are included in inspection image 420 of each region). In the example of FIG1 , "Visual inspection result: OK" indicates that the image of the region determined to contain a defect candidate does not contain a defect, while "Visual inspection result: NG" indicates that the image of the region determined to contain a defect candidate does contain a defect.

[0108] The learning device 440 has a learning program installed therein, and the learning device 440 functions as a learning data set generating unit 441 and a learning unit 442 by executing the program.

[0109] The training dataset generator 441 extracts inspection images (normal images) determined to contain no defects by an inspector 421 or the like, based on visual inspection results, from the inspection images 420 transmitted from the AOI device 410 and containing regions of defect candidates. Furthermore, the training dataset generator 441 reads the CAD data of the printed circuit board 430 and extracts regions corresponding to the inspection images (normal images) determined to contain no defects from the read CAD data. Furthermore, based on the CAD data of the extracted regions, the training dataset generator 441 generates a mask in which the inspection regions are colored according to the material type.

[0110] In addition, the learning data set generation unit 441

[0111] ·Extracted inspection images (normal images) of each area,

[0112] Results of visual inspection,

[0113] The area corresponding to the inspection image of each area extracted from the CAD data,

[0114] The generated mask,

[0115] The data are associated with each other and stored in the learning data set storage unit 443 as a learning data set.

[0116] The learning unit 442 reads the inspection images (normal images) of each region included in the training dataset stored in the training dataset storage unit 443. Furthermore, the learning unit 442 generates masked normal images by superimposing the generated masks on the inspection regions of the extracted inspection images (normal images) of each region. Furthermore, the learning unit 442 performs a learning process on the model to reconstruct the inspection images (normal images) of each region based on the generated masked normal images.

[0117] Furthermore, AI is generated using the image for the model that has been learned by the learning unit 442. Hereinafter, this model will be referred to as an "image reconstruction unit."

[0118] <Hardware Structure of Learning Device>

[0119] Next, the hardware configuration of the learning device 440 will be described. Figure 5 FIG is a diagram showing an example of the hardware configuration of a learning device. Figure 5 As shown, the learning device 440 includes a processor 501 , a memory 502 , an auxiliary storage device 503 , an I / F (Interface) device 504 , a communication device 505 , and a drive device 506 . Furthermore, the hardware components of the learning device 440 are interconnected via a bus 507 .

[0120] The processor 501 includes various computing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 501 reads various programs (eg, learning programs) into the memory 502 and executes them.

[0121] The memory 502 includes main storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 501 and the memory 502 form a so-called computer. The processor 501 reads various programs into the memory 502 and executes them. This computer implements the aforementioned functions (for example, the learning dataset generation unit 441 and the learning unit 442).

[0122] The auxiliary storage device 503 stores various programs and various data used when the processor 501 executes the programs. For example, the learning data set storage unit 443 is implemented in the auxiliary storage device 503.

[0123] The I / F device 504 is a connection device that connects an operating device 510, an example of an external device, to the display device 511, and the learning device 440. The I / F device 504 receives operations on the learning device 440 (e.g., input of visual inspection results by the inspector 421, etc., or input of learning instructions by an administrator (not shown) of the learning device 440) through the operating device 510. Furthermore, the I / F device 504 outputs the results of the learning process performed by the learning device 440 and displays them to the administrator of the learning device 440 via the display device 511.

[0124] The communication device 505 is a communication device for communicating with another device (in this embodiment, the AOI device 410 ).

[0125] Drive 506 is a device for loading recording medium 512. Recording medium 512 includes media that record information optically, electrically, or magnetically, such as CD-ROMs, floppy disks, and magneto-optical disks. Recording medium 512 may also include semiconductor memories that record information electrically, such as ROMs and flash memories.

[0126] Furthermore, various programs installed in the auxiliary storage device 503 can be installed by, for example, placing the distributed recording medium 512 in the drive device 506 and having the drive device 506 read the various programs recorded in the recording medium 512. Alternatively, various programs installed in the auxiliary storage device 503 can be installed by downloading them from a network via the communication device 505.

[0127] <Details of Each Part of the Learning Device>

[0128] Next, each component of the learning device 440 (here, the learning data set generating unit 441 and the learning unit 442 ) will be described in detail.

[0129] (1) Specific Example of Processing by the Learning Dataset Generating Unit

[0130] Figure 6 1 is a diagram showing a specific example of the processing performed by the learning data set generating unit of the learning device. Figure 6 As shown, when inspection images 610 , 611 , 620 , and 621 of respective regions including defect candidates are transmitted from the AOI device 410 , for example, the learning data set generator 441 extracts normal images with “visual inspection result: OK”.

[0131] Figure 6The example shows a case where, among inspection images 610, 611, 620, and 621 of each region, inspection images 620 and 621 are abnormal images with a "visual inspection result: NG" rating. Therefore, the learning dataset generator 441 extracts inspection images 610 and 611 of each region (an example of the first image, a normal image with a "visual inspection result: OK" rating) and generates a learning dataset 630.

[0132] like Figure 6 As shown, the learning data set 630 includes “ID”, “inspection image”, “visual inspection result”, “CAD data”, and “mask” as information items.

[0133] "ID" stores an identifier for identifying the inspection image (normal image) of each region. "Inspection Image" stores the inspection image (normal image) of each region. "Visual Inspection Result" stores the results of the visual inspection of the inspection image (normal image) of each region. Since only normal images with "Visual Inspection Result: OK" are stored in the learning dataset 630, only "OK" is stored in "Visual Inspection Result."

[0134] In the “CAD data”, CAD data of a region corresponding to the inspection image (normal image) stored in the “inspection image” is stored, which is extracted from the CAD data of the corresponding inspection object (for example, the printed circuit board 430 ).

[0135] "Mask" stores a mask generated by extracting an inspection area overlaid with the mask from the CAD data stored in "CAD Data" and adding a color corresponding to the material type. Alternatively, "Mask" stores a mask generated by adding a color corresponding to the material type to the CAD data stored in "CAD Data" and extracting an inspection area overlaid with the mask.

[0136] (2) Specific examples of processing performed by the Learning Department

[0137] Figure 7 : is a diagram showing a specific example of the processing performed by the learning unit of the learning device. Figure 7 As shown, the learning unit 442 includes an image input unit 710 , a mask unit 720 , an image reconstruction unit 730 , and a comparison / change unit 740 .

[0138] The image input unit 710 reads the inspection images of each region (eg, inspection images 610 and 611 (normal images)) stored in the “inspection image” of the learning dataset 630 stored in the learning dataset storage unit 443 , and inputs the read images to the mask unit 720 .

[0139] The masking unit 720 reads the mask stored in the "Mask" field of the training dataset 630 stored in the training dataset storage unit 443. Furthermore, the masking unit 720 generates a masked normal image (an example of a first masked image, e.g., masked normal images 721 and 722) by superimposing the read mask on the inspection area of ​​the inspection image (normal image) input by the image input unit 710. The masking unit 720 then inputs the generated masked normal images 721 and 722 to the image reconstruction unit 730.

[0140] The image reconstruction unit 730 reconstructs a reconstructed image (an example of a first reconstructed image, for example, reconstructed images 731 and 732 ) based on the masked normal images 721 and 722 , and outputs the reconstructed images 731 and 732 to the comparison / modification unit 740 .

[0141] The comparison / change unit 740 compares the reconstructed images 731 and 732 reconstructed by the image reconstruction unit 730 with the inspection image (normal image, for example, the inspection images 610 and 611 ) read by the image input unit 710 , and updates the model parameters of the image reconstruction unit 730 so that the two are consistent.

[0142] Thus, the image reconstruction unit 730 performs a learning process of reconstructing the inspection images 610 and 611 (normal images) based on the masked normal images 721 and 722 generated by the masking unit 720 .

[0143] Then, the learned image reconstruction unit obtained by performing the learning process of reconstructing the inspection image (normal image) is used in the inspection stage described later.

[0144] <Inspecting the system structure (Inspection phase)>

[0145] Next, the system configuration in the inspection phase of the inspection system according to the first embodiment will be described. Figure 8 This is a diagram showing an example of the system configuration of the inspection system in the inspection phase of the first embodiment.

[0146] like Figure 8 As shown, the inspection system 800 in the inspection stage includes the AOI device 410 and a determination device 810 .

[0147] The AOI device 410 is the same as the AOI device 410 of the inspection system 400 in the learning phase, and a description thereof is omitted here.

[0148] The determination device 810 has a determination program installed therein, and by executing the program, the determination device 810 functions as an inference unit 811 and an output unit 812 .

[0149] The inference unit 811 includes a learned image reconstruction unit generated during the learning phase. The inference unit 811 performs automatic visual inspection on the inspection object (e.g., printed circuit board 430) to obtain inspection images 420 for each region transmitted from the AOI device 410. Furthermore, the inference unit 811 generates masked inspection images by superimposing a mask on the inspection region of the obtained inspection images 420 for each region. Furthermore, the inference unit 811 reconstructs a reconstructed image by inputting the generated masked inspection image into the learned image reconstruction unit. Furthermore, the inference unit 811 compares the reconstructed image with the inspection image 420 to determine whether the inspection image 420 for each region contains a defect. The inference unit 811 notifies the output unit 812 of the determination result.

[0150] The output unit 812 outputs the judgment result notified by the inference unit 811 to the inspection line. At the inspection line, the inspector 421 performs a visual inspection on the inspection images of each area containing defect candidates. However, during the inspection phase, at the inspection line, with reference to the judgment result output by the output unit 812, the inspection images 420 of each area containing defect candidates that are determined by the judgment device 810 to not contain defects are removed. Furthermore, at the inspection line, the inspection images 820 of each area containing defect candidates that are determined by the judgment device 810 to contain defects are assigned to visual inspection. That is, the output unit 812 outputs the inspection images 820 in such a manner that the inspection images 820 determined to contain defects can be visually inspected.

[0151] As described above, when the AOI device 410 automatically inspects the appearance of the inspection object and detects inspection images 420 of various areas containing defect candidates, the inspection images determined to contain defects are assigned to visual inspection by the determination device 810 in the inspection line. As a result, the inspection system 800 can reduce the number of inspection images assigned to visual inspection and reduce the workload of the inspector 421 performing visual inspections.

[0152] <Hardware Structure of Determination Device>

[0153] Next, the hardware structure of the determination device 810 will be described. Figure 9 FIG. 1 is a diagram showing an example of the hardware structure of the determination device. Figure 9 As shown, the hardware structure of the determination device 810 is substantially the same as the hardware structure of the learning device 440 , and therefore the following mainly describes the differences from the hardware structure of the learning device 440 .

[0154] like Figure 9As shown, the processor 901 reads various programs (e.g., a determination program) into the memory 902 and executes them. By executing the various programs read into the memory 902 by the processor 901, the computer formed by the processor 901 and the memory 902 realizes, for example, the functions described above (the inference unit 811 and the output unit 812).

[0155] <Details of Each Part of the Determination Device>

[0156] Next, each component of the determination device 810 (here, the inference unit 811) will be described in detail. Figure 10 FIG. 1 is a diagram showing a specific example of processing performed by the inference unit of the judgment device. Figure 10 As shown, the inference unit 811 includes an image input unit 1010 , a masking unit 1020 , a learned image reconstruction unit 1030 , a removal unit 1040 , and a determination unit 1050 .

[0157] The image input unit 1010 obtains an inspection image of each area (an example of a second image) from the input image data set 1000 including the inspection image of each area sent from the AOI device 410, and inputs the image to the mask unit 1020. It is assumed that the input image data set 1000 has already been generated when the image input unit 1010 obtains the inspection image of each area.

[0158] like Figure 10 As shown, the input image data set 1000 includes "ID", "inspection image", "CAD data", and "mask" as information items.

[0159] In "ID", identifiers for identifying inspection images of each area are stored. In "Inspection Image", inspection images of each area are stored. In "CAD Data", CAD data of an area corresponding to the inspection image stored in "Inspection Image" is stored, which is extracted from the CAD data of the corresponding inspection object (for example, the printed circuit board 430). In "Mask", a mask generated by extracting an inspection area of ​​an overlapping mask from the CAD data stored in "CAD Data" and adding a color corresponding to the type of material is stored. Alternatively, in "Mask", a mask generated by adding a color corresponding to the type of material to the CAD data stored in "CAD Data" and extracting an inspection area of ​​an overlapping mask is stored.

[0160] The masking unit 1020 reads the mask stored in the "Mask" field of the input image dataset 1000. Furthermore, the masking unit 1020 generates a masked inspection image (an example of a second masked image, e.g., masked inspection images 1021 and 1022) by superimposing the read mask on the inspection region of the inspection image input by the image input unit 1010. Furthermore, the masking unit 1020 inputs the generated masked inspection images 1021 and 1022 to the learned image reconstruction unit 1030.

[0161] The learned image reconstruction unit 1030 is a learned model generated by performing a learning process on the image reconstruction unit 730 in the learning phase. The learned image reconstruction unit 1030 reconstructs a reconstructed image (an example of a second reconstructed image, for example, reconstructed images 1031 and 1032 ) based on the masked inspection image.

[0162] The removal unit 1040 reads the type of material from the "CAD data" of the input image dataset 1000 and determines the material corresponding to the base ( Figure 10 In the example of material B and material D, the removal unit 1040 removes the base region, which is the region of the identified material, from the inspection image read by the image input unit 1010 and notifies the determination unit 1050 of the inspection image after the removal. Furthermore, the removal unit 1040 removes the base region, which is the region of the identified material, from the reconstructed images 1031 and 1032 reconstructed by the learned image reconstruction unit 1030 and notifies the determination unit 1050 of the reconstructed image after the removal.

[0163] The determination unit 1050 compares the inspection image after removal notified by the removal unit 1040 with the reconstructed image after removal, and determines whether the inspection image includes a defect.

[0164] Specifically, the determination unit 1050 calculates the mean square error (MSE) of the pixel values ​​of each pixel in the inspection image after removal and the reconstructed image after removal. The determination unit 1050 then determines whether the calculated MSE is below a predetermined threshold (Th). If the calculated MSE is determined to be below the predetermined threshold, the inspection image read by the image input unit 1010 is determined to contain no defects. Conversely, if the calculated MSE exceeds the predetermined threshold, the inspection image read by the image input unit 1010 is determined to contain defects.

[0165] As described above, inference unit 811 reconstructs a reconstructed image based on a masked inspection image superimposed with a mask containing color information corresponding to the material information. Consequently, inference unit 811 can appropriately reconstruct the material boundary, even in an example where reconstruction of a normal image is difficult due to the inspection area of ​​the superimposed mask being pressed against the material boundary (inspection image ID = 101). Consequently, inference unit 811 can reduce misjudgments.

[0166] Furthermore, after removing the background region, the inference unit 811 compares the reconstructed image with the inspection image to determine whether a defect is present. Therefore, even in an example where the inspection region of the overlap mask contains a background region that makes normal image reconstruction difficult (inspection image ID = 102), the influence of the background region can be suppressed. Consequently, the inference unit 811 can reduce false positives.

[0167] <Flow of learning process>

[0168] Next, the flow of the learning process performed by the learning device 440 of the inspection system 400 will be described. Figure 11 This is a flowchart showing the flow of a learning process performed by the learning device of the inspection system according to the first embodiment.

[0169] In step S1101 , the learning data set generator 441 of the learning device 440 acquires inspection images of each region including defect candidates via the AOI device 410 .

[0170] In step S1102, the learning dataset generator 441 of the learning device 440 extracts normal images with a "Visual Inspection Result: OK" rating from the acquired inspection images of each region. Furthermore, the learning dataset generator 441 acquires CAD data, extracts regions corresponding to the normal images with a "Visual Inspection Result: OK" rating, and generates masks.

[0171] In step S1103 , the learning data set generation unit 441 of the learning device 440 generates a learning data set.

[0172] In step S1104 , the learning unit 442 of the learning device 440 generates a masked normal image by superimposing the mask included in the learning dataset on the inspection region of the inspection image (normal image) of each region included in the learning dataset.

[0173] In step S1105 , the learning unit 442 of the learning device 440 causes the image reconstruction unit 730 to learn to reconstruct the inspection image (normal image) based on the generated masked normal image.

[0174] In step S1106, the learning unit 442 of the learning device 440 determines whether to end the learning process. If it is determined in step S1106 that the learning process is to be continued ("No" in step S1106), the process returns to step S1101.

[0175] On the other hand, if it is determined in step S1106 that the learning process is to be completed (if "YES" in step S1106), the process proceeds to step S1107.

[0176] In step S1107 , the learning unit 442 of the learning device 440 outputs the learned image reconstruction unit 1030 and ends the learning process.

[0177] <Flow of Judgment Processing>

[0178] Next, the flow of the determination process performed by the determination device 810 of the inspection system 800 will be described. Figure 12 This is a flowchart showing the flow of determination processing performed by the determination device of the inspection system according to the first embodiment.

[0179] In step S1201 , the inference unit 811 of the determination device 810 acquires an inspection image of each region including a defect candidate from the AOI device 410 .

[0180] In step S1202, the inference unit 811 of the determination device 810 extracts the area of ​​the CAD data corresponding to the acquired inspection image of each area and generates a mask.

[0181] In step S1203, the inference unit 811 of the determination device 810 generates a masked inspection image by superimposing the generated mask on the inspection region of the acquired inspection image of each region.

[0182] In step S1204 , the inference unit 811 of the determination device 810 reconstructs a reconstructed image by inputting the generated masked inspection image to the learned image reconstruction unit 1030 .

[0183] In step S1205, the inference unit 811 of the determination device 810 generates a post-removal inspection image obtained by removing the base region from the inspection image acquired in step S1201. Furthermore, the inference unit 811 of the determination device 810 generates a post-removal reconstructed image obtained by removing the base region from the reconstructed image reconstructed in step S1204.

[0184] In step S1206, the inference unit 811 of the determination device 810 compares the inspection image after removal with the reconstructed image after removal to determine whether the inspection image obtained in step S1201 contains a defect. The inference unit 811 of the determination device 810 then outputs the determination result.

[0185] In step S1207, the inference unit 811 of the determination device 810 determines whether to terminate the determination process. If it is determined in step S1207 that the determination process is to be continued ("No" in step S1207), the process returns to step S1201.

[0186] On the other hand, if it is determined in step S1207 that the determination process is to be terminated (in the case of YES in step S1207 ), the determination process is terminated.

[0187] Summary

[0188] As has been clearly stated in the above description, the inspection system 400 of the first embodiment,

[0189] Generate a learning dataset that includes a normal image determined to contain no defects in an inspection image obtained by photographing an inspection object, and a mask superimposed on the inspection area of ​​the normal image, wherein the mask is colored according to the type of material contained in the area corresponding to the inspection object.

[0190] A learned image reconstruction unit is generated by performing learning to reconstruct a normal image included in the training data set by superimposing a masked normal image on an inspection region of the normal image included in the training data set.

[0191] In addition, the inspection system 800 of the first embodiment,

[0192] A masked inspection image is generated by superimposing a mask colored according to the type of material contained in the region corresponding to the inspection object on the inspection region of the inspection image obtained by capturing the inspection object.

[0193] The masked inspection image is input to the learning image reconstruction unit to reconstruct the reconstruction image.

[0194] By comparing a post-removal inspection image obtained by removing the background region from an inspection image obtained by photographing the inspection object with a post-removal reconstructed image obtained by removing the background region, it is determined whether the inspection image obtained by photographing the inspection object contains a defect.

[0195] Therefore, according to the first embodiment, even when it is difficult to reconstruct a normal image, it is possible to reduce erroneous determinations in the inspection system.

[0196] [Second embodiment]

[0197] While the first embodiment does not mention the details of the colors added when generating the mask, the colors added when generating the mask can vary depending on the type of material, and any color can be added. For example, in the first embodiment, the image captured by the AOI device 410 is a monochrome image, and in the first embodiment, a color (white, black, gray) close to the color of the image captured by the AOI device 410 is added to the mask. However, the color added when generating the mask does not necessarily have to be close to the color of the image captured by the AOI device 410, and any color can be added.

[0198] In the first embodiment, the use of a mask with a color corresponding to the material type enables appropriate reconstruction of the material boundary. However, in addition to applying the color corresponding to the material, the mask may alternatively include a mask with a boundary line added to the material boundary. Furthermore, a mask with a color corresponding to the material type contains more information than a mask with a boundary line added to the material boundary, thus enabling more appropriate reconstruction of the material boundary.

[0199] Furthermore, the first embodiment does not describe in detail the method for extracting CAD data for the area corresponding to the inspection image transmitted by the AOI device 410. However, when extracting CAD data for the corresponding area, for example, the CAD data may be subjected to positional or size correction based on the inspection image. This is because, depending on the imaging conditions of the AOI device 410, the position and size of the inspection image transmitted by the AOI device 410 may be offset relative to the CAD data.

[0200] Furthermore, while the first embodiment does not mention the position of the inspection area where the mask overlaps, it is assumed that the inspection area is adjusted to be located in the center of the inspection image in the AOI device 410. Furthermore, the first embodiment does not mention the size of the overlapped mask, but it is assumed that the size of the overlapped mask is adjusted based on the size of the inspection area of ​​the inspection image transmitted from the AOI device 410. However, there are no restrictions on the position or size of the inspection area where the mask overlaps, and masks of any position and size can be overlapped.

[0201] In the first embodiment, details of the process of determining whether a defect is included in an inspection image by performing image processing using the inspection image and the reconstructed image are not mentioned. However, examples of such process include the following.

[0202] Calculate the absolute difference between each pixel of the inspection image and the reconstructed image to generate a difference image.

[0203] Binarize the difference image and extract the area where the absolute value of the difference is greater than a threshold.

[0204] Perform contour extraction on the binarized difference image, and extract the contour of the area where the absolute value of the difference is greater than a threshold.

[0205] If the shape and size of the extracted contour meet the prescribed conditions, it is determined that a defect is included. If the shape and size of the extracted contour do not meet the prescribed conditions, it is determined that no defect is included.

[0206] As described above, by determining whether a defect is included in an inspection image based on the result of image processing, the determination accuracy can be improved compared to the case of determining based on the size of an error.

[0207] The present invention is not limited to the configurations shown in the above embodiments and the combinations of other elements described herein. These points can be modified within the scope of the present invention and can be appropriately determined according to the application form.

[0208] This application claims priority based on Japanese Patent Application No. 2023-054110 filed with the Japan Patent Office on March 29, 2023, and the entire contents of that Japanese patent application are cited herein.

[0209] Explanation of symbols

[0210] 400 Inspection System

[0211] 410AOI device

[0212] 440 learning device

[0213] 441 Learning Dataset Generation Department

[0214] 442 Learning Department

[0215] 630 learning dataset

[0216] 710 Image input unit

[0217] 720 Mask Department

[0218] 730 Image Reconstruction Department

[0219] 740 Comparison / Change Department

[0220] 810 judgment device

[0221] 811 Reasoning Department

[0222] 812 output unit

[0223] 1000 input image dataset

[0224] 1010 Image input unit

[0225] 1020 Mask Department

[0226] 1030 Completed the study of image reconstruction department

[0227] 1040 Removal Department

[0228] 1050 Judgment Department

Claims

1. A determination device, comprising: a learned image reconstruction unit that is learned by reconstructing a first image determined to contain no defects from an image obtained by photographing an inspection object based on a first mask image, wherein the first mask image is an image in which a mask is superimposed on an inspection area of ​​the first image, the mask being superimposed on the inspection area of ​​the first image and colored according to a type of material contained in an area corresponding to the inspection object; and A determination unit compares a second reconstructed image with a second image obtained by photographing the inspection object to determine whether the second image contains defects, wherein the second reconstructed image is an image reconstructed by inputting a second mask image into the learned image reconstruction unit, and the second mask image is an image of the mask corresponding to the inspection area superimposed on the second image.

2. The determination device according to claim 1, wherein: The determination device further includes a removal unit that generates the second image obtained by removing the region containing the specific type of material from the second image, and the second reconstructed image obtained by removing the region containing the specific type of material from the second reconstructed image. The determination unit determines whether the second image includes a defect by comparing the second image after the removal with the second reconstructed image after the removal.

3. The determination device according to claim 2, wherein: The determination unit, If a value calculated based on the error between the pixel values ​​of each pixel of the second image after the removal and the second reconstructed image after the removal satisfies a predetermined condition, it is determined that the second image does not contain a defect. If a value calculated based on an error between pixel values ​​of each pixel of the second image after removal and the second reconstructed image after removal does not satisfy a predetermined condition, it is determined that the second image includes a defect.

4. The determination device according to claim 1, wherein: The mask superimposed on the first image is CAD data obtained by extracting an area corresponding to the inspection area of ​​the first image from the CAD data of the inspection object, and is CAD data colored by determining the type of material contained in the area corresponding to the inspection object.

5. The determination device according to claim 4, wherein: When a region corresponding to the inspection region of the first image is extracted from the CAD data of the inspection object, the position and size of the CAD data of the inspection object are corrected based on the first image.

6. The determination device according to claim 1, wherein: The mask superimposed on the second image is CAD data obtained by extracting an area corresponding to the inspection area of ​​the second image from the CAD data of the inspection object, and is CAD data colored by determining the type of material contained in the area corresponding to the inspection object.

7. The determination device according to claim 6, wherein: When a region corresponding to the inspection region of the second image is extracted from the CAD data of the inspection object, the position and size of the CAD data of the inspection object are corrected based on the second image.

8. A learning device comprising: a mask unit for generating a first mask image by superimposing a mask, colored according to a type of material contained in an area corresponding to the inspection object, on an inspection area of ​​a first image obtained by photographing the inspection object, the mask being superimposed on the inspection area of ​​the first image determined to contain no defects; and The image reconstruction unit outputs a first reconstructed image when the first mask image is input. The image reconstruction unit performs learning so that the first reconstructed image becomes close to the first image.

9. A determination method, for causing a computer of a determination device to execute the following determination steps: The determination device stores a learned image reconstruction unit, which is learned by reconstructing a first image determined to contain no defects from an image obtained by photographing an inspection object based on a first mask image, wherein the first mask image is an image in which a mask is superimposed on an inspection area of ​​the first image, the mask being superimposed on the inspection area of ​​the first image and colored according to a type of material contained in an area corresponding to the inspection object; In the judgment process, the second reconstructed image is compared with the second image obtained by photographing the inspection object to determine whether the second image contains defects. The second reconstructed image is an image reconstructed by inputting a second mask image into the learned image reconstruction unit. The second mask image is an image of the mask corresponding to the inspection area superimposed on the second image.

10. A learning method, comprising causing a computer of a learning device to execute the following steps: a masking step of generating a first mask image by superimposing a mask colored according to the type of material contained in the region corresponding to the inspection object, which is superimposed on the inspection region of the first image obtained by photographing the inspection object and is determined not to contain defects; and an image reconstruction step in which, upon inputting the first mask image, the image reconstruction unit outputs a first reconstructed image; In the image reconstruction step, the image reconstruction unit is subjected to a learning process so that the first reconstructed image approaches the first image.

11. A determination program for causing a computer of a determination device to execute the following determination process: The determination device stores a learned image reconstruction unit, which is learned by reconstructing a first image determined to contain no defects among images obtained by photographing the inspection object based on a first mask image, wherein the first mask image is an image in which a mask is superimposed on an inspection area of ​​the first image, and the mask is a mask superimposed on the inspection area of ​​the first image and colored according to the type of material contained in the area corresponding to the inspection object. In the judgment process, the second reconstructed image is compared with the second image obtained by photographing the inspection object to determine whether the second image contains defects. The second reconstructed image is an image reconstructed by inputting a second mask image into the learned image reconstruction unit. The second mask image is an image of the mask corresponding to the inspection area superimposed on the second image.

12. A learning program for causing a computer of a learning device to execute the following steps: a masking step of generating a first mask image by superimposing a mask colored according to the type of material contained in the region corresponding to the inspection object, which is superimposed on the inspection region of the first image obtained by photographing the inspection object and is determined not to contain defects; and an image reconstruction step in which, upon inputting the first mask image, the image reconstruction unit outputs a first reconstructed image; In the image reconstruction step, the image reconstruction unit is subjected to a learning process so that the first reconstructed image approaches the first image.

Images