Judgment device, learning device, judgment method, learning method, judgment procedure and learning procedure

Abstract

To reduce misjudgments in the inspection system, the determination device includes: a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and the image reconstruction unit is learned in a manner that makes the first reconstructed image approximate the first image, wherein the first mask image is generated by overlaying a mask on the inspection area of ​​a first image that is determined not to contain defects in an image obtained from photographing the object to be inspected; a synthesis unit that, when multiple second reconstructed images are reconstructed by inputting multiple second mask images to the learned image reconstruction unit, synthesizes the multiple second reconstructed images to generate a second synthesized image, wherein the multiple second mask images are generated by sequentially overlaying multiple masks on the inspection area of ​​a second image obtained from photographing the object to be inspected; and a determination unit that compares the second synthesized image with the second image to determine whether the second image contains defects.

Description

Technical Field

[0001] This invention relates to a determination device, a learning device, a determination method, a learning method, a determination procedure, and a learning procedure. Background Technology

[0002] There is an existing inspection system that uses images obtained from photographing printed circuit boards and other inspection objects to visually inspect images that are determined to contain defects in order to determine whether the inspection object is a good product or a defective product.

[0003] In this inspection system, for example, image generation AI (Artificial Intelligence) learned from reconstructed normal images is used to determine whether an image contains defects.

[0004] In this image generation AI, a reconstructed image is generated from a masked inspection image with an overlaid mask on the inspection image. The reconstructed image is then compared with the original inspection image to determine whether it is a normal image. Therefore, this image generation AI can appropriately determine defects even when new types of defects are generated.

[0005] <Prior art documents>

[0006] <Patent Documents>

[0007] Patent Document 1: Japanese Patent Application Publication No. 2022-114331 Summary of the Invention

[0008] <Problem to be solved by this invention>

[0009] On the other hand, in the case of the object being inspected as described above, manufacturing deviations may occur within the range of good products. In this case, the gap between the inspected image and the reconstructed image becomes larger, so in the image generation AI, even if the inspected image is a normal image, it may be misjudged as containing defects.

[0010] One aspect of the present invention aims to reduce misjudgments in inspection systems.

[0011] <Methods for solving problems>

[0012] According to one method, the determination device includes: a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and an image reconstruction unit that is learned in a manner that makes the first reconstructed image close to the first image, wherein the first image is an image determined to be free of defects from an image obtained by photographing an object to be inspected, and the first mask image is generated by overlaying a mask on the inspection area of ​​the first image.

[0013] The compositing unit, after reconstructing multiple second reconstructed images by inputting multiple second mask images into the learned image reconstruction unit, synthesizes the multiple second reconstructed images to generate a second composite image. The multiple second mask images are generated by sequentially overlaying multiple masks on the inspection area of ​​a second image obtained from photographing an inspected object.

[0014] The determination unit compares the second synthesized image with the second image to determine whether the second image contains defects.

[0015] <The Effects of the Invention>

[0016] It can reduce false positives in the inspection system. Attached Figure Description

[0017] Figure 1A This is a diagram illustrating one example of a learning process for comparative examples of image generation AI.

[0018] Figure 1B This is a diagram illustrating an example of a comparison case determination process used when using AI generated from learned images to determine whether a defect is present.

[0019] Figure 2 This is a diagram illustrating an example of a misjudgment.

[0020] Figure 3A This is a diagram illustrating an example of the learning processing method for image generation AI in the inspection system of the first embodiment.

[0021] Figure 3B This diagram illustrates an example of a determination processing method in the inspection system of the first embodiment that uses learned image-generated AI to determine whether a defect is present.

[0022] Figure 4 This is a diagram illustrating an example of the system structure of the inspection system during the learning phase of the first embodiment.

[0023] Figure 5 This is a diagram illustrating an example of the hardware structure of a learning device.

[0024] Figure 6 This is a diagram illustrating a specific example of the processing performed by the learning dataset generation unit of the learning device.

[0025] Figure 7 This is a diagram illustrating a specific example of the processing performed by the learning unit of the learning device.

[0026] Figure 8 This is a diagram illustrating an example of the system structure of the inspection system in the inspection stage of the first embodiment.

[0027] Figure 9 This is a diagram illustrating an example of the hardware structure of a determination device.

[0028] Figure 10 This diagram illustrates a specific example of the processing performed by the reasoning unit of the decision-making device.

[0029] Figure 11 This is a flowchart illustrating the learning process performed by the learning device of the inspection system in the first embodiment.

[0030] Figure 12 This is a flowchart illustrating the determination process performed by the determination device of the inspection system in the first embodiment.

[0031] Figure 13 This is a diagram used to illustrate the overview of the verification process.

[0032] Figure 14 This is a diagram representing an example of the test results.

[0033] Figure 15 This is a diagram representing the first mask and other examples of multiple second masks. Detailed Implementation

[0034] Hereinafter, each embodiment will be described with reference to the accompanying drawings. Furthermore, in this specification and the drawings, structural elements having substantially the same functional structure are labeled with the same symbols, and repeated descriptions are omitted.

[0035] [First Implementation]

[0036] <Explanation of the learning and decision-making methods for comparative examples in image generation AI>

[0037] As will be described later, in the inspection system of the first embodiment, in order to reduce misjudgments, the image generation AI is applied with a learning processing method and a judgment processing method that are different from the general learning processing method and judgment processing method.

[0038] Therefore, the following will first explain the general learning processing method and the judgment processing method for image generation AI (referred to as the learning processing method and judgment processing method for the comparative example). Next, we will list the cases in which misjudgments occur in the learning processing method and the judgment processing method for the comparative example, and explain the learning processing method and the judgment processing method for image generation AI in the inspection system of the first embodiment that can reduce such misjudgments.

[0039] Figure 1A This is a diagram illustrating one example of a learning process for comparative examples in image generation AI. For example... Figure 1AAs shown, in the case of the learning processing method of the comparative example, AI110 is generated from the image and learning processing is performed according to the following steps.

[0040] • Obtain the inspection image (referred to as the normal image) from the inspection images captured by the inspection system that indicates the inspected object is a good product, and generate a masked normal image by overlaying a mask on the inspection area.

[0041] • The generated masked normal image is input into the image generation AI110, and the reconstructed image output from the image generation AI110 is compared with the normal image to calculate the error.

[0042] • Update the model parameters of the image generation AI110 to reduce the calculated error. This process is then applied to multiple normal images to train the image generation AI110, resulting in a fully trained image generation AI.

[0043] Next, we will explain the comparison processing method for using the generated trained images to generate AI to determine whether an inspection image contains defects.

[0044] Figure 1B This diagram illustrates an example of a comparative example processing method used to determine whether a defect exists when generating AI from a learned image. For example... Figure 1B As shown in the upper part, in the case of the comparative example, AI120, which is generated from the learned image, is used to perform the following judgment process to determine that it does not contain defects.

[0045] • A masked inspection image is generated by acquiring an inspection image captured in the inspection system and overlaying a mask on the inspection area.

[0046] • The generated masked inspection image is input into the learned image to generate AI120, and the learned image to generate AI120 outputs the reconstructed image.

[0047] • The error between the reconstructed image output by the learned image generation AI120 and the inspection image is calculated to obtain a result with a small error. Furthermore, the learned image generation AI120 is the result of learning to reconstruct a normal image from the input masked inspection image. Therefore, the reconstructed image output by the learned image generation AI120 is an image that is close to the normal image.

[0048] Therefore, the calculated error is small, and the inspected image can be considered to be close to a normal image. As a result, it can be determined that the inspected image does not contain defects.

[0049] On the other hand, such as Figure 1BAs shown in the lower half, in the case of the comparative example, AI120, which is generated using the learned image, performs the following judgment process to determine that it contains defects.

[0050] • A masked inspection image is generated by acquiring an inspection image captured in the inspection system and overlaying a mask on the inspection area.

[0051] • The generated masked inspection image is input into the learned image to generate AI120, and the learned image to generate AI120 outputs the reconstructed image.

[0052] • The error between the reconstructed image output by the learned image generation AI120 and the inspection image is calculated, resulting in a larger error. Furthermore, as mentioned above, the learned image generation AI120 is the result of learning to reconstruct a normal image from the input masked inspection image; therefore, the reconstructed image output by the learned image generation AI120 is an image close to the normal image.

[0053] Therefore, a large calculated error indicates that the inspected image is far removed from a normal image. Consequently, the inspected image can be determined to contain defects.

[0054] Furthermore, in Figure 1B In this embodiment, for the sake of simplicity, it is determined whether the inspection image contains defects based on the magnitude of the calculated error. However, the determination of whether an inspection image contains defects is not limited to this. For example, it may be configured to perform image processing using the inspection image and the reconstructed image, and make the determination based on the result of the image processing. Furthermore, in this embodiment, for the sake of simplicity, the case of determining whether an inspection image contains defects based on the magnitude of the calculated error will be described.

[0055] <Explanation of Misjudgment>

[0056] Next, an example will be given regarding an inspection image that should have been determined by AI120 to be free of defects from the learned image but was incorrectly determined to contain defects. Figure 2 This is a diagram illustrating an example of a misjudgment.

[0057] exist Figure 2 In the example, inspection image 210 is an example of an inspection image that does not contain defects, that is, an example of an inspection image that was misjudged as containing defects by the learned image generation AI 120. The inspection object corresponding to inspection image 210 has the following characteristics.

[0058] • In the inspection area, the shape of a part of the object being inspected changes (see symbol 211).

[0059] • The shape variation of a portion of the inspected item is due to manufacturing deviation and falls within the acceptable range.

[0060] When examining image 210, after the overlap mask is input into the learned image generation AI120, it will output, for example, from the learned image generation AI120. Figure 2 The reconstructed image 220 is shown. (As shown) Figure 2 As shown, the reconstructed image 220 output by the image generation AI120 after learning is a typical normal image, and the reconstruction fails to include changes in the shape of a part of the object being examined (refer to symbol 221).

[0061] Therefore, when comparing the reconstructed image 220 with the inspection image 210, the error between the two will increase, which may lead to a misjudgment that the inspection image 210 contains defects.

[0062] In order to reduce misjudgments in this situation, the inspection system of the first embodiment has the following structure.

[0063] • Generate a first mask by regularly arranging multiple mask pieces smaller than the overlapping mask on the inspection area with specified intervals (intervals that are integer multiples of the mask piece size) in the vertical, horizontal, and diagonal directions.

[0064] Generate multiple second masks with multiple mask patches regularly arranged at specified intervals in the vertical, horizontal, and diagonal directions, respectively, to fill the first mask. Generate a number of second masks corresponding to the size of the intervals.

[0065] • Multiple reconstructed images are reconstructed by inputting an image (a masked inspection image) formed by sequentially overlapping a first mask and multiple second masks into the learned image generation AI.

[0066] • A composite image is generated by combining multiple reconstructed images.

[0067] By adopting the above structure, the inspection system according to the first embodiment can also reconstruct the shape changes of a portion of the inspection object caused by manufacturing deviations within the good product range, thereby reducing misjudgments.

[0068] <Learning Processing Method for Image Generation AI in the Inspection System of the First Embodiment>

[0069] The learning processing method for image generation AI in the inspection system of the first embodiment will be described. Figure 3A This diagram illustrates an example of the learning processing method for image generation AI in the inspection system of the first embodiment.

[0070] and Figure 1A The difference in the learning processing methods of the comparative examples shown is that, Figure 3AIn the case of generating a masked normal image of the input image to AI310, the mask includes...

[0071] • A first mask is formed by regularly arranging multiple mask pieces smaller than the overlapping mask on the inspection area with specified intervals (intervals that are integer multiples of the mask piece size) in the vertical, horizontal, and diagonal directions.

[0072] Using this learning processing method, the inspection system of the first embodiment can generate a learned image generation AI that can reconstruct changes in the shape of a portion of the inspected object caused by manufacturing deviations within the good product range.

[0073] Next, we will discuss the determination process in the inspection system of the first embodiment, which uses learned image generation AI to determine whether a defect is present.

[0074] Figure 3B This diagram illustrates an example of a determination processing method in the inspection system of the first embodiment, where a learned image-generated AI is used to determine whether a defect is present. Figure 1B The difference is that, in Figure 3B In the case of, including

[0075] • A first mask is formed by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with specified intervals (intervals that are integer multiples of the mask piece size) in the vertical, horizontal, and diagonal directions.

[0076] • A second mask consisting of three types, in which multiple mask pieces are regularly arranged at specified intervals in the vertical direction, the horizontal direction, and the diagonal direction, respectively, to fill the first mask.

[0077] and Figure 1B The difference in the determination and processing methods of the comparative examples shown is that, in Figure 3B In this case,

[0078] • By inputting four masked inspection images, formed by sequentially overlapping a first mask and three types of second masks, into the AI320 image generation system after learning, four reconstructed images are generated.

[0079] • A composite image is generated by combining the four reconstructed images.

[0080] Using this determination and processing method, the inspection system of the first embodiment can reconstruct a portion of the shape change of the inspected object caused by manufacturing deviations within the good product range, thereby reducing the error between the inspected image and the synthesized image. As a result, it can be determined that the inspected image does not contain defects (avoiding misjudgment that the inspected image contains defects), thus reducing misjudgments.

[0081] <Inspecting the System Architecture (Learning Phase)>

[0082] Next, the system structure of the inspection system in the learning phase of the first embodiment of the image generation AI will be described. Figure 4 This is a diagram illustrating an example of the system structure of the inspection system during the learning phase of the first embodiment.

[0083] like Figure 4 As shown, the inspection system 400 in the learning phase includes an automated optical inspection (AOI) device 410 and a learning device 440.

[0084] AOI device 410 performs automated visual inspection of printed circuit board 430. AOI device 410 scans printed circuit board 430 using a camera and performs various inspection items to identify potential defects. Inspection items performed by AOI device 410 include, for example, circuit width, circuit spacing, absence of missing pads / pads, and short circuits.

[0085] Inspection images 420 containing defect candidates identified by the AOI device 410 are sent to the learning device 440 and then to the inspection line. On the inspection line, inspectors 421 and others visually inspect the inspection images 420 containing defect candidates. Here, to avoid mistaking defective products for good ones, it is assumed that the AOI device 410 is configured to perform excessive inspections on the inspection images containing defect candidates.

[0086] Inspector 421 and others visually inspect the inspection images 420 of each area for defects and ultimately determine whether the printed circuit board 430 is a good or defective product. Specifically, if none of the inspection images 420 containing defect candidates contain defects, the printed circuit board 430 is determined to be a good product. Conversely, if any one of the inspection images 420 containing defect candidates contains a defect, the printed circuit board 430 is determined to be a defective product.

[0087] Furthermore, inspector 421 and others notify the learning device 440 of the results of the visual inspection (the results of determining whether the inspection images 420 of each region contain defects). In the example of Figure 1, "Visual inspection result: OK" means that the image of the region judged to contain a defect candidate does not contain a defect, and "Visual inspection result: NG" means that the image of the region judged to contain a defect candidate contains a defect.

[0088] The learning device 440 is equipped with a learning program, and by executing the program, the learning device 440 functions as a learning dataset generation unit 441 and a learning unit 442.

[0089] The learning dataset generation unit 441 extracts inspection images (normal images) from the inspection images 420 containing defect candidates sent by the AOI device 410, which are determined by the visual inspection results of the inspector 421 and others to be free of defects, and generates a learning dataset. Furthermore, the learning dataset generation unit 441 stores the generated learning dataset in the learning dataset storage unit 443.

[0090] The learning unit 442 reads the inspection images (normal images) of each region contained in the learning dataset stored in the learning dataset storage unit 443. Furthermore, the learning unit 442 overlays a first mask onto the inspection regions of the extracted inspection images (normal images) to generate a masked normal image. In addition, when the generated masked normal image is input, the learning unit 442 performs learning processing on the model used to output the reconstructed image to make the reconstructed image approximate the normal image.

[0091] In addition, image generation AI is used for the model that is learned and processed by the learning unit 442. Hereinafter, this model will be referred to as the "image reconstruction unit".

[0092] <Hardware Structure of the Learning Device>

[0093] Next, the hardware structure of the learning device 440 will be explained. Figure 5 This is a diagram illustrating an example of the hardware structure of a learning device. For example... Figure 5 As shown, the learning device 440 includes a processor 501, a memory 502, an auxiliary storage device 503, an I / F (Interface) device 504, a communication device 505, and a driver device 506. Furthermore, the hardware of the learning device 440 is interconnected via a bus 507.

[0094] The processor 501 includes various computing devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 501 loads various programs (e.g., learning programs) into the memory 502 and executes them.

[0095] The memory 502 includes main storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 501 and the memory 502 form a so-called computer, which implements, for example, the functions described above (learning dataset generation unit 441, learning unit 442) by loading various programs into the memory 502 and executing them.

[0096] The auxiliary storage device 503 stores various programs and various data used by the processor 501 when executing these programs. For example, a learning dataset storage unit 443 is implemented in the auxiliary storage device 503.

[0097] The I / F device 504 is a connection device that connects the operating device 510, the display device 511, and the learning device 440, which are examples of external devices. The I / F device 504 receives operations from the operating device 510 concerning the learning device 440 (e.g., inputting the results of a visual inspection performed by an inspector 421, or inputting instructions for learning processing by the administrator of the learning device 440 (not shown)). Furthermore, the I / F device 504 outputs the results of the learning processing performed by the learning device 440 and displays them to the administrator of the learning device 440 via the display device 511.

[0098] The communication device 505 is a communication device used to communicate with other devices (AOI device 410 in this embodiment).

[0099] The drive unit 506 is a device for mounting the recording medium 512. Here, the recording medium 512 includes media such as CD-ROMs, floppy disks, and magneto-optical disks that record information optically, electrically, or magnetically. Alternatively, the recording medium 512 may also include semiconductor memories such as ROMs and flash memory that record information electrically.

[0100] Furthermore, various programs installed in the auxiliary storage device 503 can be installed, for example, by placing the distributed recording medium 512 on the drive device 506 and having the drive device 506 read the various programs recorded on the recording medium 512. Alternatively, the various programs installed in the auxiliary storage device 503 can also be installed by downloading them from a network via the communication device 505.

[0101] <Details of the various parts of the learning device>

[0102] Next, the various parts of the learning device 440 will be described in detail (here, the learning dataset generation unit 441 and the learning unit 442).

[0103] (1) Specific examples of the processing performed by the learning dataset generation unit

[0104] Figure 6 This is a diagram illustrating a specific example of the processing performed by the learning dataset generation unit of the learning device. For example... Figure 6 As shown, when inspection images 610-630 containing defect candidates for each region are sent from the AOI device 410, for example, the learning dataset generation unit 441 extracts normal images that show "Visual inspection result: OK".

[0105] Figure 6 The example illustrates a case where, among the inspection images 610-630 in each region, inspection image 630 is an abnormal image with a "visual inspection result: NG". Therefore, in the learning dataset generation unit 441, inspection images 610 and 620 (normal images with a "visual inspection result: OK") in each region are extracted to generate the learning dataset 640.

[0106] like Figure 6 As shown, the learning dataset 640 includes items as information such as "ID", "inspection image", and "visual inspection result".

[0107] The "ID" field stores identifiers used to identify the inspection images (normal images) for each region. The "Inspection Image" field stores the inspection images (normal images) for each region. The "Visual Inspection Result" field stores the visual inspection results for the inspection images (normal images) for each region. Furthermore, since only normal images with "Visual Inspection Result: OK" are stored in the learning dataset 640, only "OK" is stored in the "Visual Inspection Result" field.

[0108] (2) Specific examples of processing carried out by the Learning Department

[0109] Figure 7 This is a diagram illustrating a specific example of the processing performed by the learning unit of the learning device. For example... Figure 7 As shown, the learning unit 442 includes an image input unit 710, a mask unit 720, an image reconstruction unit 730, and a comparison / modification unit 750.

[0110] The image input unit 710 reads the inspection images (an example of the first image, such as inspection image 610 (normal image)) of each region stored in the "inspection image" of the learning dataset 640 stored in the learning dataset storage unit 443, and inputs them into the mask unit 720.

[0111] The masking unit 720 generates a masked normal image (an example of the first masked image, such as a masked normal image 760) by overlaying a pre-generated first mask onto the inspection area of ​​the inspection image input by the image input unit 710. The masking unit 720 then inputs the generated masked normal image 760 into the image reconstruction unit 730.

[0112] The image reconstruction unit 730 reconstructs the masked normal image 760 and outputs a reconstructed image (an example of the first reconstructed image, such as reconstructed image 762).

[0113] The comparison / modification unit 750 compares the reconstructed image 762 output by the image reconstruction unit 730 with the inspection image (normal image, for example, inspection image 610) read by the image input unit 710, and updates the model parameters of the image reconstruction unit 730 to make the two consistent.

[0114] Therefore, when a reconstructed image 762 is reconstructed from a masked normal image 760, the image reconstruction unit 730 performs learning processing to make the reconstructed image 762 approximate the inspection image (normal image).

[0115] Furthermore, the image reconstruction unit, which has undergone learning processing to make the reconstructed image approximate the inspection image (normal image), is used in the inspection stage described later.

[0116] As described above, according to the learning unit 442, it is possible to generate a learned image generation AI that can reconstruct the shape changes of a portion of the inspected object caused by manufacturing deviations within the good product range.

[0117] <Inspect the system architecture (inspection phase)>

[0118] Next, the system structure in the inspection phase of the inspection system of the first embodiment will be described. Figure 8 This is a diagram illustrating an example of the system structure of the inspection system in the inspection stage of the first embodiment.

[0119] like Figure 8 As shown, the inspection system 800 in the inspection phase includes an AOI device 410 and a judgment device 810.

[0120] The AOI device 410 is the same as the AOI device 410 of the inspection system 400 in the learning phase, and its description is omitted here.

[0121] The determination device 810 is equipped with a determination program. By executing the program, the determination device 810 functions as the reasoning unit 811 and the output unit 812.

[0122] The inference unit 811 includes a learning-completed image reconstruction unit generated during the learning phase. The inference unit 811 acquires inspection images 420 of each region sent from the AOI device 410 by performing an automatic visual inspection of the inspection object (e.g., printed circuit board 430). Furthermore, the inference unit 811 generates multiple masked inspection images (an example of multiple second masked images) by sequentially overlaying a first mask and multiple second masks onto the inspection areas of the acquired inspection images 420 of each region (an example of a second image). The inference unit 811 then reconstructs multiple reconstructed images (an example of multiple second reconstructed images) by sequentially inputting the generated multiple masked inspection images into the learning-completed image reconstruction unit. The inference unit 811 then synthesizes the multiple reconstructed images to generate a composite image (an example of a second composite image). Finally, the inference unit 811 determines whether each region of the inspection image 420 contains defects by comparing the composite image with the inspection image 420. The inference unit 811 then notifies the output unit 812 of the determination result.

[0123] The output unit 812 outputs the judgment result notified by the inference unit 811 to the inspection line. At the inspection line, the inspector 421 performs visual inspection of the inspection images containing defect candidates for each area. However, during the inspection phase, at the inspection line, based on the judgment result output by the output unit 812, inspection images 420 containing defect candidates that are determined by the judgment device 810 to not contain defects are removed. Furthermore, at the inspection line, inspection images 820 containing defect candidates that are determined by the judgment device 810 to contain defects are assigned to visual inspection. That is, the output unit 812 outputs the inspection image 820 in a manner that allows for visual inspection of the inspection image 820 determined to contain defects.

[0124] As described above, when the AOI device 410 performs automatic visual inspection on the object to be inspected and identifies inspection images 420 containing candidate defects, the determination device 810 assigns inspection images determined to contain defects to visual inspection along the inspection line. As a result, the number of inspection images assigned to visual inspection can be reduced by the inspection system 800, and the workload of visual inspection performed by the inspector 421 can be reduced.

[0125] <Hardware Structure of the Judgment Device>

[0126] Next, the hardware structure of the determination device 810 will be explained. Figure 9 This is a diagram illustrating an example of the hardware structure of a determination device. For example... Figure 9 As shown, the hardware structure of the determination device 810 is roughly the same as that of the learning device 440. Therefore, the following mainly describes the differences in hardware structure between the two devices.

[0127] like Figure 9 As shown, the processor 901 reads various programs (e.g., decision programs, etc.) into the memory 902 and executes them. By executing the various programs read into the memory 902 by the processor 901, the computer formed by the processor 901 and the memory 902 can, for example, implement the functions described above (inference unit 811, output unit 812).

[0128] <Details of the various parts of the determination device>

[0129] Next, the various parts of the determination device 810 (here, the reasoning unit 811) will be described in detail. Figure 10 This diagram illustrates a specific example of the processing performed by the reasoning unit of the decision-making device. For example... Figure 10 As shown, the inference unit 811 includes an image input unit 1010, a masking unit 1020, a learned image reconstruction unit 1030, a synthesis unit 1040, and a determination unit 1050.

[0130] The image input unit 1010 acquires the inspection images 420 of each area sent by the AOI device 410 and inputs them into the mask unit 1020.

[0131] The masking unit 1020 generates four masked inspection images (e.g., four masked inspection images 1060) by sequentially overlaying a first mask and three types of second masks onto the inspection area of ​​the inspection image input by the image input unit 1010. Furthermore, the masking unit 1020 inputs the four masked inspection images 1060 generated to the learned image reconstruction unit 1030.

[0132] The learned image reconstruction unit 1030 generates a learned model by performing learning processing on the image reconstruction unit 730 during the learning phase. The learned image reconstruction unit 1030 reconstructs four reconstructed images (e.g., four reconstructed images 1061) based on four masked inspection images 1060.

[0133] The compositing unit 1040 combines the four reconstructed images 1061 reconstructed by the image reconstruction unit 1030 after learning to generate a composite image (e.g., composite image 1062). Furthermore, the compositing unit 1040 notifies the determination unit 1050 of the generated composite image 1062.

[0134] The determination unit 1050 compares the composite image 1062 notified by the synthesis unit 1040 with the inspection image input by the image input unit 1010 to determine whether the inspection image contains defects.

[0135] Specifically, the determination unit 1050 calculates the mean square error (MSE) of the pixel values ​​of each pixel in both the composite image and the inspection image. Then, the determination unit 1050 determines whether the calculated MSE is below a predetermined threshold (Th). If the calculated MSE is below the predetermined threshold, it is determined that the inspection image read by the image input unit 1010 does not contain defects. Conversely, if the calculated MSE exceeds the predetermined threshold, it is determined that the inspection image input by the image input unit 1010 contains defects.

[0136] As described above, according to the inference unit 811, it is possible to reconstruct a portion of the shape change of the inspected object caused by manufacturing deviations within the good product range, thereby reducing the error between the inspection image and the synthesized image. As a result, according to the inference unit 811, there will no longer be cases where an inspection image is mistakenly judged to contain a defect even when there is no defect, thereby reducing misjudgments.

[0137] <Learning Process>

[0138] Next, the learning process of the learning device 440 of the inspection system 400 will be explained. Figure 11 This is a flowchart illustrating the learning process performed by the learning device of the inspection system of the first embodiment.

[0139] In step S1101, the learning dataset generation unit 441 of the learning device 440 obtains inspection images 420 containing defect candidates for each region through the AOI device 410.

[0140] In step S1102, the learning dataset generation unit 441 of the learning device 440 extracts a normal image with the message "Visual inspection result: OK" from the inspection images 420 of each region.

[0141] In step S1103, the learning dataset generation unit 441 of the learning device 440 generates a learning dataset.

[0142] In step S1104, the learning unit 442 of the learning device 440 generates a masked normal image by overlapping a first mask with the inspection regions of the inspection images (normal images) of each region contained in the learning dataset.

[0143] In step S1105, the learning unit 442 of the learning device 440 reconstructs the image from the generated masked normal image.

[0144] In step S1106, the learning unit 442 of the learning device 440 updates the parameters of the image reconstruction unit by making the reconstructed image close to the inspection image (normal image), thereby enabling the image reconstruction unit to perform learning processing.

[0145] In step S1107, the learning unit 442 of the learning device 440 determines whether to end the learning process. If it is determined in step S1107 that the learning process should continue (if it is "no" in step S1107), the process returns to step S1103.

[0146] In contrast, if the learning process is determined to end in step S1107 (if "yes" is indicated in step S1107), then proceed to step S1108.

[0147] In step S1108, the learning unit 442 of the learning device 440 outputs the image reconstruction unit after learning is completed, and the learning process ends.

[0148] <Decision Processing Flow>

[0149] Next, the determination process of the determination device 810 of the inspection system 800 will be explained. Figure 12 This is a flowchart illustrating the process of determination processing performed by the determination device of the inspection system of the first embodiment.

[0150] In step S1201, the inference unit 811 of the determination device 810 obtains inspection images 420 containing defect candidates for each region from the AOI device 410.

[0151] In step S1202, the inference unit 811 of the determination device 810 generates multiple masked inspection images by overlaying a first mask and multiple second masks on the inspection areas of the inspection images 420 of each region.

[0152] In step S1203, the inference unit 811 of the determination device 810 reconstructs multiple reconstructed images by inputting the generated multiple masked inspection images into the learned image reconstruction unit.

[0153] In step S1204, the inference unit 811 of the determination device 810 synthesizes the multiple reconstructed images to generate a composite image.

[0154] In step S1205, the inference unit 811 of the determination device 810 compares the inspection image obtained in step S1201 with the composite image generated in step S1204 to calculate the MSE, thereby determining whether the inspection image contains defects. Furthermore, the inference unit 811 of the determination device 810 outputs a determination result.

[0155] In step S1206, the inference part 811 of the determination device 810 determines whether to end the determination process. If it is determined in step S1206 that the determination process should continue (if it is "no" in step S1206), the process returns to step S1201.

[0156] In contrast, if the determination process is terminated in step S1206 (if "yes" is indicated in step S1206), the determination process is terminated.

[0157] <Verification by the Reasoning Department>

[0158] Next, the generation accuracy will be verified when the inference unit 811 of the determination device 810 is used to reconstruct the reconstructed image and generate the composite image. The verification of the generation accuracy of the composite image will be performed in the following order.

[0159] 1) Prepare the comparison device.

[0160] 2) By overlaying different first masks on the same normal image, different learning datasets are generated, and learning is performed using each learning dataset to generate...

[0161] • The reasoning unit (the image reconstruction unit after learning) of the comparison device, and,

[0162] • The reasoning unit 811 of the judgment device 810 (the image reconstruction unit 1030 after learning).

[0163] 3) By inputting the same verification image into the inference unit of the comparison device and the inference unit 811 of the determination device 810 respectively, the reconstructed image is reconstructed and a composite image is generated.

[0164] 4) Calculate the error between the input verification image and the composite image generated in the inference unit of the comparison device, and the error between the input verification image and the composite image generated in the inference unit 811 of the determination device 810.

[0165] 5) By repeatedly performing steps 3) and 4) on multiple verification images and calculating the average error, the generation accuracy of the synthesized image is verified.

[0166] Figure 13 This is a diagram used to illustrate an overview of the verification process. Among them, Figure 13 (a) represents a summary of the processing performed by the inference unit of the comparison device. As shown by symbol 1310, the inference unit of the comparison device generates a masked verification image using four rectangular masks (including one rectangular mask piece, one first mask, and three second masks) that overlap on each of the four regions that divide the inspection region 1311 into four parts.

[0167] Specifically, as shown by symbol 1312, the inference unit of the comparison device generates four masked verification images by overlapping four rectangular masks onto the verification image 1301.

[0168] Next, the inference unit of the comparator reconstructs the four masked verification images to generate a composite image 1313. Then, the inference unit of the comparator calculates the error between the verification image 1301 and the composite image 1313.

[0169] on the other hand, Figure 13 (b) represents a summary of the processing performed by the inference unit 811 of the determination device 810. As shown by reference numeral 1320, the inference unit 811 of the determination device 810 generates a masked verification image using four overlapping grid-shaped masks (containing multiple mask pieces arranged in a grid pattern, one first mask and three second masks) on the inspection area 1321.

[0170] Specifically, as shown by symbol 1322, the inference unit 811 of the determination device 810 generates four masked verification images by overlaying four grid-shaped masks onto the verification image 1301.

[0171] Next, the inference unit 811 of the determination device 810 reconstructs the four masked verification images and generates a composite image 1323. Then, the inference unit 811 of the determination device 810 calculates the error between the verification image 1301 and the composite image 1323.

[0172] Figure 14 This is a diagram representing an example of the verification results. In Figure 14 In the symbol 1410, the average error of the pixel value of each pixel between multiple verification images and multiple synthetic images generated by the inference unit of the comparison device based on the multiple verification images is represented.

[0173] In addition, Figure 14 In the figure, reference numeral 1420 represents the average error of the pixel value of each pixel between the multiple verification images and the multiple synthetic images generated by the inference unit 811 of the determination device 810 based on the multiple verification images.

[0174] The comparison of the average error (=3.69) shown by symbol 1410 with the average error (2.61) shown by symbol 1420 shows that the average error shown by symbol 1420 is reduced by 29.3% compared to the average error shown by symbol 1410. This indicates that the inference unit 811 of the determination device 810 generates the synthesized image with higher accuracy than the inference unit of the comparison device.

[0175] In addition, by improving the synthesis accuracy of the synthesized image generated by the inference unit 811 of the determination device 810, the risk of misjudging a normal image as an abnormal image can be reduced.

[0176] Summary

[0177] As can be seen from the above description, in the inspection system 400 of the first embodiment,

[0178] • A first mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with specified intervals in the vertical, horizontal, and diagonal directions.

[0179] Generate multiple second masks, each corresponding to a specified interval size, at regular intervals in the vertical, horizontal, and diagonal directions of the first mask.

[0180] • A masked normal image is generated by overlaying a first mask onto the inspection area of ​​a normal image that is determined to be free of defects in an image obtained from photographing the object to be inspected.

[0181] • The image reconstruction is performed by inputting the generated masked normal image, and the image reconstruction unit is subjected to learning processing to make the reconstructed image close to the inspection image (normal image), thereby generating the learned image reconstruction unit.

[0182] Therefore, the inspection system 400 according to the first embodiment can generate a learned image generation AI that can reconstruct the shape changes of a portion of the inspection object caused by manufacturing deviations within the good product range.

[0183] Furthermore, in the inspection system 800 of the first embodiment,

[0184] • By sequentially overlaying a first mask and multiple second masks on the inspection area of ​​the inspection image obtained from photographing the object to be inspected, multiple masked inspection images are generated.

[0185] • When multiple reconstructed images are reconstructed by inputting multiple masked inspection images into the learned image reconstruction unit, a composite image is generated by synthesizing the multiple reconstructed images.

[0186] • Compare the inspected image with the synthesized image to determine whether the inspected image contains defects.

[0187] Therefore, the inspection system 800 according to the first embodiment can reconstruct the shape changes of a portion of the inspected object caused by manufacturing deviations within the good product range, and can reduce the error between the inspection image and the synthesized image. As a result, the inspection system 800 according to the first embodiment will not misjudge cases that do not contain defects as inspection images containing defects, and can reduce misjudgments.

[0188] [Second Implementation]

[0189] In the first embodiment described above, a first mask is generated by regularly arranging multiple mask pieces, each smaller than the mask overlapping on the inspection area, with spacing equal to the size of the mask pieces in the vertical, horizontal, and diagonal directions. Furthermore, in the first embodiment, three types of second masks are generated by regularly arranging mask pieces at positions that fill the horizontal, vertical, and diagonal spacings equal to the size of the mask pieces.

[0190] However, the methods for generating the first mask and multiple second masks are not limited to those described above. Figure 15 This is another example of a first mask and multiple second masks. Among them, Figure 15 (a) is used for comparison to show the first mask and three second masks shown in the first embodiment.

[0191] on the other hand, Figure 15 (b) indicates generation

[0192] A first mask is formed by regularly arranging multiple mask pieces smaller than the overlapping mask on the inspection area, with spacing equal to the size of the mask pieces in the vertical direction, and spacing twice the size of the mask pieces in the horizontal and diagonal directions.

[0193] • A second mask consisting of regularly arranged mask pieces at intervals in the vertical direction equal to the size of the filler mask piece; and five other types of second masks consisting of regularly arranged mask pieces at intervals in the horizontal direction and diagonal direction, which are twice the size of the filler mask piece.

[0194] also, Figure 15 (c) represents generation

[0195] • A first mask is formed by regularly arranging multiple mask pieces smaller than the overlapping mask on the inspection area, with spacing equal to twice the size of the mask pieces in the vertical and diagonal directions, and spacing equal to the size of the mask pieces in the horizontal direction.

[0196] • One type of second mask is formed by regularly arranging mask pieces at horizontal intervals equal to the size of the fill mask pieces, and five types of second mask shapes are formed by regularly arranging mask pieces at vertical and diagonal intervals that are twice the size of the fill mask pieces.

[0197] in addition, Figure 15 (d) indicates generation

[0198] A first mask is formed by regularly arranging multiple mask pieces smaller than the overlapping mask on the inspection area, with spacing equal to the size of the mask pieces in both the vertical and horizontal directions.

[0199] • A second mask is formed by regularly arranging mask pieces at positions with horizontal spacing equal to the size of the filler mask pieces.

[0200] and, Figure 15 (e) indicates generation

[0201] • The size is smaller than the mask overlapping the inspection area, and the size is smaller than... Figure 15 As shown in (a) to (d), a first mask is formed by regularly arranging multiple large mask pieces at intervals equal to the size of the mask pieces in the vertical, horizontal, and diagonal directions.

[0202] • Three types of second masks are formed by regularly arranging mask pieces at positions that fill the horizontal, vertical, and diagonal intervals that are equal in size to the mask pieces.

[0203] As mentioned above, there are various methods for generating the first mask and multiple second masks, which can be generated based on considerations such as the size of the obtained inspection image and the size of the defects contained in the inspection image.

[0204] [Third Implementation]

[0205] In the learning device 440 of the first embodiment described above, the case where the masking unit 720 generates a masked normal image using a first mask was explained. However, the method for generating a masked normal image is not limited to this; for example, one of a plurality of second masks may also be used to generate a masked normal image.

[0206] Furthermore, in the learning apparatus 440 of the first embodiment described above, the case where the masking unit 720 performs learning processing on the image reconstruction unit 730 by generating a masked normal image using a single mask is explained. However, the method for learning processing on the image reconstruction unit 730 is not limited to this; for example, learning processing can be performed by generating a masked normal image using multiple masks (e.g., a first mask and a second mask, or multiple second masks).

[0207] Furthermore, the learning device 440 of the first embodiment described above employs a structure without a synthesis unit. However, the structure of the learning device 440 is not limited to this; similar to the determination device 810, a synthesis unit may also be provided. In this case, the learning device 440 inputs multiple masked normal images generated by sequentially overlapping a first mask and multiple second masks into the image reconstruction unit 730, and the synthesis unit synthesizes the multiple reconstructed images reconstructed by the image reconstruction unit 730 to generate a synthesized image. Thus, in the learning device 440, the image reconstruction unit 730 performs learning processing to make the synthesized image approximate the inspection image (normal image).

[0208] Furthermore, although the position of the inspection area where the first mask and multiple second masks overlap is not mentioned in the first embodiment, it is envisioned that the inspection area be adjusted in the AOI device 410 so that it is located in the center of the inspection image. Also, although the size of the overlapping masks is not mentioned in the first embodiment, it is envisioned that the size of the overlapping masks be adjusted according to the size of the inspection area of ​​the inspection image sent by the AOI device 410. However, there are no restrictions on the position and size of the inspection area of ​​the overlapping masks; masks of any position and size can be overlapped.

[0209] Furthermore, in the first embodiment above, the details of the process for determining whether an inspection image contains defects by performing image processing using the inspection image and the reconstructed image are not mentioned, but the following process can be cited as an example of such process.

[0210] • Calculate the absolute difference between each pixel of the inspected image and the reconstructed image to generate a difference image.

[0211] • By binarizing the difference image, regions with an absolute difference value above a threshold are extracted.

[0212] • Perform contour extraction on the binarized difference image and extract the contours of regions where the absolute difference value is above the threshold.

[0213] • If the shape and size of the extracted contour meet the specified conditions, it is determined to contain a defect; if the shape and size of the extracted contour do not meet the specified conditions, it is determined not to contain a defect.

[0214] As mentioned above, determining whether an image contains defects based on the results of image processing can improve the accuracy of the determination compared to determining the defects based on the magnitude of the error.

[0215] Furthermore, the present invention is not limited to the structures described herein, such as the structures shown in the above embodiments and combinations of other elements. Changes can be made to these points without departing from the spirit of the invention, and can be appropriately determined according to its application.

[0216] This application claims priority to Japanese Patent Application No. 2023-054111, filed with the Japan Patent Office on March 29, 2023, and the entire contents of that Japanese Patent Application are incorporated herein by reference.

[0217] Symbol Explanation

[0218] 400 Inspection System

[0219] 410 AOI device

[0220] 440 Learning Device

[0221] 441 Learning Dataset Generation Department

[0222] 442 Study Department

[0223] 640 Learning Dataset

[0224] 710 Image Input Unit

[0225] 720 mask section

[0226] 730 Image Reconstruction Department

[0227] 750 Comparison / Change Department

[0228] 810 Detection Device

[0229] 811 Reasoning Department

[0230] 812 Output Section

[0231] 1010 Image Input Unit

[0232] 1020 mask section

[0233] 1030 completed the Image Reconstruction Department training

[0234] 1040 Synthesis Department

[0235] 1050 Judgment Department

Claims

1. A determining device, comprising: The image reconstruction unit, after learning, outputs a first reconstructed image when a first mask image is input, and is an image reconstruction unit that learns to make the first reconstructed image close to the first image. The first image is an image that is determined to be free of defects from an image obtained by photographing an object to be inspected. The first mask image is generated by overlaying a mask on the inspection area of ​​the first image. The compositing unit, when reconstructing multiple second reconstructed images by inputting multiple second mask images into the learned image reconstruction unit, synthesizes the multiple second reconstructed images to generate a second composite image. The multiple second mask images are generated by sequentially overlapping multiple masks on the inspection area of ​​the second image obtained by photographing the inspection object. and The determination unit compares the second synthesized image with the second image to determine whether the second image contains defects.

2. The determining device according to claim 1, wherein, The plurality of masks includes a first mask and a plurality of second masks. The first mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with specified intervals in the vertical, horizontal, and diagonal directions. By regularly arranging multiple mask pieces at positions that fill the first mask at specified intervals in the vertical direction, the horizontal direction, and the diagonal direction, a second mask of a number corresponding to the size of the intervals is generated.

3. The determining device according to claim 1, wherein, The plurality of masks includes a first mask and a plurality of second masks. The first mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with specified intervals in the vertical and horizontal directions. By regularly arranging multiple mask pieces at specified intervals in the vertical direction and the horizontal direction to fill the first mask, a number of second masks corresponding to the size of the intervals are generated.

4. The determining device according to claim 2 or 3, wherein, The specified interval is an integer multiple of the size of the mask.

5. The determining device according to claim 1, wherein, The determination unit. If the value calculated based on the error between the pixel values ​​of each pixel in the second image and the second composite image meets the specified conditions, it is determined that the second image does not contain defects. If the value calculated based on the error between the pixel values ​​of each pixel in the second image and the second composite image does not meet the specified conditions, the second image is determined to contain defects.

6. A learning device comprising: The masking section generates a first mask image by overlaying a mask on the inspection area of ​​a normal image that is determined not to contain defects in an image obtained by photographing the object to be inspected. and The image reconstruction unit, when a first mask image is input, outputs a first reconstructed image. The mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with a specified interval in at least the vertical and horizontal directions. The image reconstruction unit learns in a way that makes the first reconstructed image approximate the normal image.

7. The learning device according to claim 6, wherein, The mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with specified intervals in the vertical, horizontal, and diagonal directions.

8. A determination method for causing a computer of a determination device to perform the following steps, The determination device stores a learned image reconstruction unit. This learned image reconstruction unit outputs a first reconstructed image when a first mask image is input. The image reconstruction unit is obtained by learning to make the first reconstructed image approximate a first image. The first image is an image determined to be free of defects from images of the object being inspected. The first mask image is generated by overlaying a mask onto the inspection area of ​​the first image. The process includes: Synthesis process: When multiple second reconstructed images are reconstructed by inputting multiple second mask images into the learned image reconstruction unit, the multiple second reconstructed images are synthesized to generate a second synthesized image. The multiple second mask images are generated by sequentially overlapping multiple masks on the inspection area of ​​the second image obtained by photographing the inspection object. and The determination process involves comparing the second synthesized image with the second image to determine whether the second image contains defects.

9. A learning method for causing a computer in a learning device to perform: The masking process generates a first mask image by overlaying a mask onto the inspection area of ​​a normal image determined to be free of defects in an image obtained from photographing the object to be inspected; and In the image reconstruction process, when a first mask image is input, the image reconstruction unit outputs a first reconstructed image. The mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with a specified interval in at least the vertical and horizontal directions. In the image reconstruction process, the image reconstruction unit learns in a way that makes the first reconstructed image approximate the normal image.

10. A computer program product comprising a determination program for causing a computer of a determination device to perform the following steps, The determination device stores a learned image reconstruction unit. The learned image reconstruction unit outputs a first reconstructed image when a first mask image is input. It is an image reconstruction unit that is learned in a way that makes the first reconstructed image close to the first image. The first image is an image that is determined to be free of defects from the images obtained by taking pictures of the object to be inspected. The first mask image is generated by overlaying a mask on the inspection area of ​​the first image. In the compositing process, when multiple second reconstructed images are reconstructed by inputting multiple second mask images into the learned image reconstruction unit, the multiple second reconstructed images are synthesized to generate a second composite image. The multiple second mask images are generated by sequentially overlapping multiple masks on the inspection area of ​​the second image obtained by photographing the inspection object. and The determination process involves comparing the second synthesized image with the second image to determine whether the second image contains defects.

11. A computer program product comprising a learning program for causing a computer of a learning device to perform the following steps: The masking process generates a first mask image by overlaying a mask onto the inspection area of ​​a normal image determined to be free of defects in an image obtained from photographing the object to be inspected; and In the image reconstruction process, when a first mask image is input, the image reconstruction unit outputs a first reconstructed image. The mask is generated by regularly arranging multiple mask pieces, each smaller than the overlapping mask on the inspection area, with a specified interval in at least the vertical and horizontal directions. In the image reconstruction process, the image reconstruction unit learns in a way that makes the first reconstructed image approximate the normal image.

Images