Determination device, learning device, determination method, learning method, determination program, and learning program

Abstract

And misjudgment in an inspection system is reduced. The determination device includes: a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and learns the first reconstructed image so that the first reconstructed image approaches the first image; the first mask image is generated by overlapping a mask on an inspection area of a first image which is judged not to contain defects in an image obtained by shooting an inspection object; and a synthesis unit that, when a plurality of second reconstructed images are reconstructed by inputting a plurality of second mask images into the learned image reconstruction unit, synthesizes the plurality of second reconstructed images to generate a second synthesized image, the plurality of second mask images are generated by sequentially overlapping a plurality of masks on an inspection area of a second image obtained by shooting the inspection object; and a determination unit that compares the second composite image with the second image to determine whether or not the second image includes a defect.

Description

Technical Field

[0001] The present invention relates to a determination device, a learning device, a determination method, a learning method, a determination program, and a learning program. Background Art

[0002] There is already an inspection system in which, for an image determined to contain a defect in an inspection image obtained by photographing an object to be inspected such as a printed circuit board, an inspector visually inspects to determine whether the object to be inspected is a non-defective product or a defective product.

[0003] In this inspection system, for example, by applying an image generation AI (Artificial Intelligence) that has learned to reconstruct a normal image, it is determined whether the inspection image contains a defect.

[0004] In this image generation AI, based on a masked inspection image in which a mask is overlapped on the inspection image, a reconstructed image is reconstructed, and it is determined whether the inspection image is a normal image by comparing it with the inspection image. Therefore, according to this image generation AI, even when a new type of defect occurs, an appropriate determination can be made.

[0005] <Prior Art Documents>

[0006] <Patent Documents>

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-114331 Summary of the Invention

[0008] <Problems to be Solved by the Invention>

[0009] On the other hand, in the case of the object to be inspected as described above, there may be a case where a manufacturing deviation occurs within the range of non-defective products. In this case, the difference between the inspection image and the reconstructed image becomes larger. Therefore, in the image generation AI, even if the inspection image is a normal image, it may be misjudged as containing a defect.

[0010] One aspect of the present invention aims to reduce misjudgments in an inspection system.

[0011] <Means for Solving the Problems>

[0012] According to one mode, a determination device includes: a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and that is an image reconstruction unit learned in such a way that the first reconstructed image approaches the first image, the first image being an image obtained by photographing an object to be inspected and determined not to contain a defect, and the first mask image being generated by overlapping a mask on an inspection area of the first image;

[0013] A synthesizing unit that synthesizes a plurality of second reconstructed images to generate a second synthesized image when the plurality of second reconstructed images are reconstructed by inputting a plurality of second mask images into the learned image reconstruction unit, where the plurality of second mask images are generated by sequentially overlapping a plurality of masks on an inspection area of a second image obtained by photographing an inspection object; and

[0014] A determination unit that compares the second synthesized image with the second image to determine whether the second image contains a defect.

[0015] <Effects of the Invention>

[0016] False judgments in the inspection system can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A FIG. is an example of a learning processing method for a comparative example of an image generation AI.

[0018] Figure 1B FIG. is an example of a determination processing method for a comparative example when determining whether a defect is included using a learned image generation AI.

[0019] Figure 2 FIG. is an example showing a false judgment.

[0020] Figure 3A FIG. is an example of a learning processing method for an image generation AI in the inspection system according to the first embodiment.

[0021] Figure 3B FIG. is an example of a determination processing method when determining whether a defect is included using a learned image generation AI in the inspection system according to the first embodiment.

[0022] Figure 4 FIG. is an example of the system configuration of the inspection system in the learning stage according to the first embodiment.

[0023] Figure 5 FIG. is an example of the hardware configuration of the learning device.

[0024] Figure 6 FIG. is an example of a specific example of the processing performed by the learning dataset generation unit of the learning device.

[0025] Figure 7 FIG. is an example of a specific example of the processing performed by the learning unit of the learning device.

[0026] Figure 8 FIG. is an example of the system configuration of the inspection system in the inspection stage according to the first embodiment.

[0027] Figure 9 It is a diagram showing an example of the hardware structure of the determination device.

[0028] Figure 10 It is a diagram showing a specific example of the processing performed by the inference unit of the determination device.

[0029] Figure 11 It is a flowchart showing the process flow of the learning process performed by the learning device of the inspection system according to the first embodiment.

[0030] Figure 12 It is a flowchart showing the process flow of the determination process performed by the determination device of the inspection system according to the first embodiment.

[0031] Figure 13 It is a diagram for explaining the outline of the verification process.

[0032] Figure 14 It is a diagram showing an example of the inspection result.

[0033] Figure 15 It is a diagram showing another example of the first mask and the plurality of second masks. Detailed Embodiments

[0034] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In addition, in this specification and the drawings, structural elements having substantially the same functional structure are denoted by the same reference numerals, and repeated descriptions are omitted.

[0035] [First Embodiment]

[0036] [Explanation of the Learning Processing Method and Determination Processing Method for the Comparative Example of the Image Generation AI]

[0037] As described later, in the inspection system according to the first embodiment, in order to reduce misjudgment, a learning processing method and a determination processing method different from the general learning processing method and determination processing method are applied to the image generation AI.

[0038] Therefore, first, the general learning processing method and determination processing method for the image generation AI (referred to as the learning processing method and determination processing method of the comparative example) will be described below. Next, cases where misjudgment occurs in the learning processing method and determination processing method of the comparative example will be listed, and the learning processing method and determination processing method for the image generation AI in the inspection system according to the first embodiment that can reduce the misjudgment will be described.

[0039] Figure 1A It is a diagram showing an example of the learning processing method of the comparative example for the image generation AI. As Figure 1AAs shown, in the case of the learning processing method of the comparative example, the following steps are performed for the image generation AI 110 for learning processing.

[0040] · Obtain inspection images in the inspection system that are determined to be non-defective for the inspection object (referred to as normal images), and generate masked normal images by overlapping a mask on the inspection area.

[0041] · By inputting the generated masked normal images into the image generation AI 110 and comparing the reconstructed images output from the image generation AI 110 with the normal images, calculate the error.

[0042] · Update the model parameters of the image generation AI 110 to reduce the calculated error. By performing this process on multiple normal images, the learning processing of the image generation AI 110 is performed to generate a learned image generation AI.

[0043] Next, a description will be given of the determination processing method of the comparative example when determining whether a defect is included in the inspection image using the generated learned image generation AI.

[0044] Figure 1B is a diagram showing an example of the determination processing method of the comparative example when determining whether a defect is included using the learned image generation AI. As Figure 1B shown in the upper part of, in the case of the comparative example, the following steps are performed for determination processing using the learned image generation AI 120, and it is determined that no defect is included.

[0045] · By obtaining the inspection image taken in the inspection system and overlapping a mask on the inspection area, generate a masked inspection image.

[0046] · By inputting the generated masked inspection image into the learned image generation AI 120, the learned image generation AI 120 outputs a reconstructed image.

[0047] · Calculate the error between the reconstructed image output by the learned image generation AI 120 and the inspection image, and obtain a result with a small error. And since the learned image generation AI 120 is the result of learning to reconstruct a normal image for the input masked inspection image, the reconstructed image output by the learned image generation AI 120 becomes an image close to the normal image.

[0048] · Therefore, the calculated error is small, and it can be determined that the inspection image is an image close to the normal image. As a result, it can be determined that the inspection image does not include a defect.

[0049] On the other hand, as Figure 1BAs shown in the lower part of [], in the case of the comparative example, the learned image generation AI 120 is used to perform a determination process according to the following steps to determine that a defect is included.

[0050] · By obtaining the inspection image taken in the inspection system and overlapping a mask on the inspection area, a masked inspection image is generated.

[0051] · By inputting the generated masked inspection image into the learned image generation AI 120, the learned image generation AI 120 outputs a reconstructed image.

[0052] · Calculate the error between the reconstructed image output by the learned image generation AI 120 and the inspection image to obtain a result with a large error. And, as described above, the learned image generation AI 120 is the result of learning to reconstruct a normal image from the input masked inspection image. Therefore, the reconstructed image output by the learned image generation AI 120 becomes an image close to the normal image.

[0053] · Therefore, if the calculated error is large, it can be determined that the inspection image is an image far from the normal image. As a result, it can be determined that the inspection image contains a defect.

[0054] And, in Figure 1B For simplicity of explanation, it is determined whether the inspection image contains a defect based on the magnitude of the calculated error. However, the determination of whether the inspection image contains a defect is not limited to this. For example, it can also be configured to perform image processing on the inspection image and the reconstructed image and make a determination based on the result of this image processing. In addition, in the present embodiment, for simplicity of explanation, the case of determining whether the inspection image contains a defect based on the magnitude of the calculated error is described.

[0055] <Explanation of misjudgment>

[0056] Next, an example in which an inspection image that should be determined by the learned image generation AI 120 not to contain a defect is erroneously determined to contain a defect will be described. Figure 2 is a diagram showing an example of misjudgment.

[0057] In Figure 2 , the inspection image 210 is an example of an inspection image that does not contain a defect, that is, an example of an inspection image that is erroneously determined by the learned image generation AI 120 to contain a defect. The inspection object corresponding to the inspection image 210 has the following characteristics.

[0058] · In the inspection area, the shape of a part of the inspection object changes (reference numeral 211).

[0059] · The shape change of a part of the inspection object is caused by manufacturing deviation and is within the range of good products.

[0060] When checking the image 210, when the overlapping mask is input into the learned image generation AI 120, for example, the reconstructed image 220 as shown will be output from the learned image generation AI 120. Figure 2 As shown. Figure 2 As shown, the reconstructed image 220 output by the learned image generation AI 120 is a typical normal image, and the reconstruction fails to include the shape change of a part of the inspection object (reference sign 221).

[0061] Therefore, when comparing the reconstructed image 220 with the inspection image 210, the error between the two will increase, resulting in a misjudgment that the inspection image 210 contains defects.

[0062] In view of this situation, in order to reduce misjudgment, the inspection system of the first embodiment has the following structure.

[0063] · Generate a first mask in which a plurality of mask pieces smaller than the mask overlapping the inspection area in size are regularly arranged at a prescribed interval (an interval that is an integer multiple of the mask piece size) in the vertical, horizontal, and diagonal directions.

[0064] · Generate a plurality of second masks in which a plurality of mask pieces are regularly arranged at positions filling the prescribed intervals in the vertical direction, the prescribed intervals in the horizontal direction, and the prescribed intervals in the diagonal direction of the first mask. Generate a number of second masks corresponding to the size of the intervals.

[0065] · Reconstruct a plurality of reconstructed images by inputting the image (masked inspection image) obtained by sequentially overlapping the first mask and the plurality of second masks into the learned image generation AI.

[0066] · Generate a composite image by synthesizing the plurality of reconstructed images.

[0067] By adopting the above structure, according to the inspection system of the first embodiment, regarding the shape change of a part of the inspection object caused by manufacturing deviations within the acceptable range, reconstruction can also be performed, and misjudgment can be reduced.

[0068] <Learning processing method for the image generation AI in the inspection system of the first embodiment>

[0069] The learning processing method for the image generation AI in the inspection system of the first embodiment will be described. Figure 3A is a diagram showing an example of the learning processing method for the image generation AI in the inspection system of the first embodiment.

[0070] Differences from Figure 1A the learning processing method of the comparative example shown in Figure 3AIn the case of, as a mask used when generating a masked normal image input to the image generation AI 310, it includes

[0071] · A first mask formed by regularly arranging a plurality of mask pieces smaller than the mask overlapping the size ratio inspection region at regular intervals (intervals that are an integer multiple of the mask piece size) in the vertical, horizontal, and diagonal directions.

[0072] With this learning processing method, through the inspection system of the first embodiment, it is possible to generate a learned image generation AI that can reconstruct a change in a part of the shape of the inspection object caused by manufacturing deviations within the acceptable range.

[0073] Next, regarding the determination processing method in the case of using the learned image generation AI in the inspection system of the first embodiment to determine whether a defect is included.

[0074] Figure 3B It is a diagram showing an example of the determination processing method in the case of using the learned image generation AI in the inspection system of the first embodiment to determine whether a defect is included. Different from Figure 1B In the case of Figure 3B it includes

[0075] · A first mask formed by regularly arranging a plurality of mask pieces smaller than the mask overlapping the size ratio inspection region at regular intervals (intervals that are an integer multiple of the mask piece size) in the vertical, horizontal, and diagonal directions,

[0076] · Three types of second masks formed by regularly arranging a plurality of mask pieces at positions filling the regular intervals in the vertical direction, the regular intervals in the horizontal direction, and the regular intervals in the diagonal direction of the first mask.

[0077] Different from Figure 1B the determination processing method of the comparative example shown in, in the case of Figure 3B it is

[0078] · By inputting four masked inspection images formed by sequentially overlapping the first mask and the three types of second masks into the learned image generation AI 320, four reconstructed images are reconstructed,

[0079] · By synthesizing the four reconstructed images, a synthesized image is generated.

[0080] With this determination processing method, through the inspection system of the first embodiment, it is possible to reconstruct a change in a part of the shape of the inspection object caused by manufacturing deviations within the acceptable range, and the error between the inspection image and the synthesized image can be reduced. As a result, it can be determined that the inspection image does not contain a defect (it will not be misjudged that the inspection image contains a defect), and misjudgment can be reduced.

[0081] <System structure of the inspection system (learning phase)>

[0082] Next, the system structure of the inspection system in the learning phase of the first embodiment of applying the image generation AI will be described. Figure 4 It is a diagram showing an example of the system structure of the inspection system in the learning phase of the first embodiment.

[0083] As Figure 4 shown, the inspection system 400 in the learning phase includes an Automated Optical Inspection (AOI) device 410 and a learning device 440.

[0084] The AOI device 410 performs an automatic appearance inspection of the printed circuit board 430. The AOI device 410 scans the printed circuit board 430 using a camera and inspects various inspection items to detect defect candidates. The inspection items inspected by the AOI device 410 include, for example, circuit width, circuit pitch, no missing pads / pads, circuit short circuits, etc.

[0085] The inspection image 420 of each area including the defect candidates detected by the AOI device 410 is sent to the learning device 440 and also sent to the inspection line. On the inspection line, an inspector 421, etc. visually inspects the inspection image 420 of each area including the defect candidates. Here, in order to avoid misjudging defective products as non-defective products, it is assumed that the AOI device 410 is set to perform an over-inspection on the inspection image of each area including the defect candidates.

[0086] The inspector 421, etc. visually inspects whether the inspection image 420 of each area contains a defect and finally determines whether the printed circuit board 430 is a non-defective product or a defective product. Specifically, when none of the inspection images 420 of the areas including the defect candidates contain a defect, it is determined that the printed circuit board 430 is a non-defective product. In addition, when any one of the inspection images 420 of the areas including the defect candidates contains a defect, it is determined that the printed circuit board 430 is a defective product.

[0087] In addition, the inspector 421, etc. notifies the learning device 440 of the result of the visual inspection (the result of judging whether the inspection image 420 of each area contains a defect). In the example of FIG. 1, "Visual inspection result: OK" indicates that the image of the area including the defect candidate is judged not to contain a defect, and "Visual inspection result: NG" indicates that the image of the area including the defect candidate is judged to contain a defect.

[0088] A learning program is installed in the learning device 440, and the learning device 440 functions as a learning dataset generation unit 441 and a learning unit 442 by executing this program.

[0089] The learning dataset generation unit 441 extracts inspection images (normal images) determined not to contain defects by visual inspection by the inspector 421 etc. from among the inspection images 420 of each region containing defect candidates sent from the AOI device 410, and generates a learning dataset. In addition, the learning dataset generation unit 441 stores the generated learning dataset in the learning dataset storage unit 443.

[0090] The learning unit 442 reads the inspection images (normal images) of each region included in the learning dataset stored in the learning dataset storage unit 443. In addition, the learning unit 442 overlaps a first mask on the inspection regions of the extracted inspection images (normal images) to generate a masked normal image. Further, when the generated masked normal image is input, the learning unit 442 performs a learning process on the model for outputting a reconstructed image to make the reconstructed image close to the normal image.

[0091] In addition, an image generation AI is used for the model processed by the learning unit 442. Hereinafter, this model will be referred to as the "image reconstruction unit".

[0092] <Hardware Structure of the Learning Device>

[0093] Next, the hardware structure of the learning device 440 will be described. Figure 5 is a diagram showing an example of the hardware structure of the learning device. As Figure 5 shown, the learning device 440 includes a processor 501, a memory 502, an auxiliary storage device 503, an I / F (Interface) device 504, a communication device 505, and a drive device 506. And, the hardware of the learning device 440 is interconnected via a bus 507.

[0094] The processor 501 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 501 reads various programs (for example, a learning program etc.) onto the memory 502 and executes them.

[0095] The memory 502 includes main storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 501 and the memory 502 form a so-called computer, and by the processor 501 reading various programs onto the memory 502 and executing them, this computer implements the above-described various functions (the learning dataset generation unit 441, the learning unit 442) for example.

[0096] The auxiliary storage device 503 stores various programs and various data used when the processor 501 executes the various programs. For example, a learning dataset storage unit 443 is implemented in the auxiliary storage device 503.

[0097] The I / F device 504 is a connection device that connects an operation device 510, a display device 511, and a learning device 440, which are examples of external devices. The I / F device 504 receives operations on the learning device 440 (for example, operations for inputting the results of visual inspections performed by an inspector 421, etc., or operations for inputting instructions for learning processes by a manager (not shown) of the learning device 440, etc.) through the operation device 510. In addition, the I / F device 504 outputs the results of learning processes performed by the learning device 440, etc., and displays them to the administrator of the learning device 440 through the display device 511.

[0098] The communication device 505 is a communication device for communicating with other devices (in this embodiment, an AOI device 410).

[0099] The drive device 506 is a device for mounting a recording medium 512. Here, the recording medium 512 includes media that record information optically, electrically, or magnetically, such as CD-ROMs, floppy disks, magneto-optical disks, etc. In addition, the recording medium 512 may also include semiconductor memories that record information electrically, such as ROMs, flash memories, etc.

[0100] In addition, various programs installed in the auxiliary storage device 503 are installed, for example, by setting the issued recording medium 512 in the drive device 506 and having the drive device 506 read the various programs recorded in the recording medium 512. Or, various programs installed in the auxiliary storage device 503 can also be installed by downloading them from a network by the communication device 505.

[0101] <Details of each part of the learning device>

[0102] Next, each part of the learning device 440 (here, the learning dataset generation unit 441 and the learning unit 442) will be described in detail.

[0103] (1) Specific example of the process performed by the learning dataset generation unit

[0104] Figure 6 is a diagram showing a specific example of the process performed by the learning dataset generation unit of the learning device. As Figure 6 shown, when inspection images 610 to 630 including defect candidates are sent from the AOI device 410, for example, in the learning dataset generation unit 441, normal images with "visual inspection result: OK" are extracted.

[0105] Figure 6 The example of Figure 6 shows a case where, among the inspection images 610 to 630 in each region, the inspection image 630 is an abnormal image with a visual inspection result of "NG". Therefore, in the learning dataset generation unit 441, the inspection images 610 and 620 (normal images with a visual inspection result of "OK") in each region are extracted to generate the learning dataset 640.

[0106] As Figure 6 shown, in the learning dataset 640, as items of information, it includes "ID", "inspection image", and "visual inspection result".

[0107] In "ID", an identifier for identifying the inspection image (normal image) in each region is stored. In "inspection image", the inspection image (normal image) in each region is stored. In "visual inspection result", the result of the visual inspection of the inspection image (normal image) in each region is stored. And since only the normal images with a visual inspection result of "OK" are stored in the learning dataset 640, only "OK" is stored in "visual inspection result".

[0108] (2) Specific example of the processing performed by the learning unit

[0109] Figure 7 is a diagram showing a specific example of the processing performed by the learning unit of the learning device. As Figure 7 shown, the learning unit 442 includes an image input unit 710, a masking unit 720, an image reconstruction unit 730, and a comparison / modification unit 750.

[0110] The image input unit 710 reads the inspection images (an example of the first image. For example, the inspection image 610 (normal image)) in each region stored in the "inspection image" of the learning dataset 640 stored in the learning dataset storage unit 443, and inputs them to the masking unit 720.

[0111] The masking unit 720 generates a masked normal image (an example of the first masked image. For example, the masked normal image 760) by overlapping a pre-generated first mask on the inspection region of the inspection image input by the image input unit 710. Furthermore, the masking unit 720 inputs the generated masked normal image 760 to the image reconstruction unit 730.

[0112] The image reconstruction unit 730 reconstructs the masked normal image 760 and outputs a reconstructed image (an example of the first reconstructed image. For example, the reconstructed image 762).

[0113] The comparison / change unit 750 compares the reconstructed image 762 output by the image reconstruction unit 730 with the inspection image (normal image, for example, the inspection image 610) read by the image input unit 710, and updates the model parameters of the image reconstruction unit 730 to make the two consistent.

[0114] Thus, in the case where the reconstructed image 762 is reconstructed from the masked normal image 760, the image reconstruction unit 730 performs learning processing to make the reconstructed image 762 close to the inspection image (normal image).

[0115] Moreover, the learned image reconstruction unit that has performed learning processing to make the reconstructed image close to the inspection image (normal image) is used in the inspection stage described later.

[0116] As described above, according to the learning unit 442, it is possible to generate a learned image generation AI that can reconstruct a change in a part of the shape of the inspection object caused by manufacturing deviations within the acceptable range.

[0117] <System structure of the inspection system (inspection stage)>

[0118] Next, the system structure in the inspection stage of the inspection system according to the first embodiment will be described. Figure 8 It is a diagram showing an example of the system structure of the inspection system in the inspection stage of the first embodiment.

[0119] As Figure 8 shown, the inspection system 800 in the inspection stage includes the AOI device 410 and the determination device 810.

[0120] Among them, the AOI device 410 is the same as the AOI device 410 of the inspection system 400 in the learning stage, and its description is omitted here.

[0121] The determination program is installed in the determination device 810, and by executing this program, the determination device 810 functions as the inference unit 811 and the output unit 812.

[0122] The inference unit 811 includes a learned image reconstruction unit generated during the learning phase. The inference unit 811 obtains inspection images 420 of each area transmitted from the AOI device 410 by automatically inspecting an object to be inspected (e.g., the printed circuit board 430). In addition, the inference unit 811 generates a plurality of masked inspection images (an example of a plurality of second masked images) by sequentially overlapping a first mask and a plurality of second masks on the inspection areas of the obtained inspection images 420 (an example of the second image) of each area. In addition, the inference unit 811 reconstructs a plurality of reconstructed images (an example of a plurality of second reconstructed images) by sequentially inputting the generated plurality of masked inspection images into the learned image reconstruction unit. In addition, the inference unit 811 synthesizes the plurality of reconstructed images to generate a synthesized image (an example of the second synthesized image). In addition, the inference unit 811 determines whether a defect is included in the inspection image 420 of each area by comparing the synthesized image with the inspection image 420. In addition, the inference unit 811 notifies the determination result to the output unit 812.

[0123] The output unit 812 outputs the determination result notified by the inference unit 811 to the inspection line. At the inspection line, the inspector 421 visually inspects the inspection images of each area including defect candidates. However, at the inspection stage, at the inspection line, referring to the determination result output by the output unit 812, the inspection images determined by the determination device 810 not to include defects are removed from the inspection images 420 of each area including defect candidates. And at the inspection line, the inspection images 820 determined by the determination device 810 to include defects among the inspection images 420 of each area including defect candidates are assigned to visual inspection. That is, in the output unit 812, the inspection image 820 is output in such a way that the inspection image 820 determined to include a defect can be visually inspected.

[0124] As described above, when the AOI device 410 automatically inspects the object to be inspected and detects the inspection images 420 of each area including defect candidates, at the inspection line, the determination device 810 assigns the inspection images determined to include defects to visual inspection. As a result, by the inspection system 800, the number of inspection images assigned to visual inspection can be reduced, and the work load of visual inspection by the inspector 421 can be reduced.

[0125] <Hardware Structure of the Determination Device>

[0126] Next, the hardware structure of the determination device 810 will be described. Figure 9 is a diagram showing an example of the hardware structure of the determination device. As Figure 9 shown, the hardware structure of the determination device 810 is substantially the same as the hardware structure of the learning device 440, so the differences from the hardware structure of the learning device 440 will be mainly described below.

[0127] As Figure 9 shown, the processor 901 reads various programs (e.g., determination programs, etc.) onto the memory 902 and executes them. By executing the various programs read onto the memory 902 by the processor 901, a computer formed by the processor 901 and the memory 902 implements, for example, the various functions (inference unit 811, output unit 812) described above.

[0128] <Details of each part of the determination device>

[0129] Next, each part of the determination device 810 (here, the inference unit 811) will be described in detail. Figure 10 is a diagram showing a specific example of processing performed by the inference unit of the determination device. As Figure 10 shown, the inference unit 811 includes an image input unit 1010, a masking unit 1020, a learned image reconstruction unit 1030, a synthesis unit 1040, and a determination unit 1050.

[0130] The image input unit 1010 acquires the inspection images 420 of each region transmitted by the AOI device 410 and inputs them to the masking unit 1020.

[0131] The masking unit 1020 generates four masked inspection images (e.g., four masked inspection images 1060) by sequentially overlapping a first mask and three types of second masks on the inspection regions of the inspection images input by the image input unit 1010. In addition, the masking unit 1020 inputs the four generated masked inspection images 1060 to the learned image reconstruction unit 1030.

[0132] The learned image reconstruction unit 1030 is a learned model generated by performing learning processing on the image reconstruction unit 730 in the learning stage. The learned image reconstruction unit 1030 reconstructs four reconstructed images (e.g., four reconstructed images 1061) based on the four masked inspection images 1060.

[0133] The synthesis unit 1040 synthesizes the four reconstructed images 1061 reconstructed by the learned image reconstruction unit 1030 to generate a synthesized image (e.g., synthesized image 1062). In addition, the synthesis unit 1040 notifies the generated synthesized image 1062 to the determination unit 1050.

[0134] The determination unit 1050 compares the synthesized image 1062 notified by the synthesis unit 1040 with the inspection image input by the image input unit 1010 to determine whether the inspection image contains a defect.

[0135] Specifically, the determination unit 1050 calculates the mean square error (MSE) of the pixel values of each pixel of the composite image and the inspection image. Then, the determination unit 1050 determines whether the calculated MSE is below a specified threshold (Th). If it is determined that the calculated MSE is below the specified threshold, it is determined that the inspection image read by the image input unit 1010 does not contain defects. In contrast, if it is determined that the calculated MSE exceeds the specified threshold, it is determined that the inspection image input by the image input unit 1010 contains defects.

[0136] As described above, according to the inference unit 811, it is possible to reconstruct the change in the shape of a part of the inspection object caused by manufacturing deviations within the acceptable range, so the error between the inspection image and the composite image can be reduced. As a result, according to the inference unit 811, the situation where the inspection image is misjudged as containing defects even when there are no defects will no longer occur, and thus misjudgments can be reduced.

[0137] <Flow of learning process>

[0138] Next, the flow of the learning process of the learning device 440 of the inspection system 400 will be described. Figure 11 It is a flowchart showing the flow of the learning process performed by the learning device of the inspection system according to the first embodiment.

[0139] In step S1101, the learning dataset generation unit 441 of the learning device 440 obtains inspection images 420 of each region containing defect candidates through the AOI device 410.

[0140] In step S1102, the learning dataset generation unit 441 of the learning device 440 extracts normal images with "visual inspection result: OK" from the obtained inspection images 420 of each region.

[0141] In step S1103, the learning dataset generation unit 441 of the learning device 440 generates a learning dataset.

[0142] In step S1104, the learning unit 442 of the learning device 440 generates a masked normal image by overlapping the first mask on the inspection region of the inspection images (normal images) contained in the learning dataset.

[0143] In step S1105, the learning unit 442 of the learning device 440 reconstructs the reconstructed image from the generated masked normal image.

[0144] In step S1106, the learning unit 442 of the learning device 440 updates the parameters of the image reconstruction unit in such a way that the reconstructed image approaches the inspection image (normal image), and can perform learning processing on the image reconstruction unit.

[0145] In step S1107, the learning unit 442 of the learning device 440 determines whether to end the learning processing. When it is determined in step S1107 to continue the learning processing (when it is "No" in step S1107), the process returns to step S1103.

[0146] In contrast, when it is determined in step S1107 to end the learning processing (when it is "Yes" in step S1107), the process proceeds to step S1108.

[0147] In step S1108, the learning unit 442 of the learning device 440 outputs the learned image reconstruction unit and ends the learning processing.

[0148] <Flow of determination processing>

[0149] Next, the flow of the determination processing of the determination device 810 of the inspection system 800 will be described. Figure 12 It is a flowchart showing the flow of the determination processing performed by the determination device of the inspection system according to the first embodiment.

[0150] In step S1201, the inference unit 811 of the determination device 810 acquires the inspection image 420 of each region including defect candidates from the AOI device 410.

[0151] In step S1202, the inference unit 811 of the determination device 810 generates a plurality of masked inspection images by overlapping the first mask and a plurality of second masks on the inspection regions of the acquired inspection images 420 of each region.

[0152] In step S1203, the inference unit 811 of the determination device 810 reconstructs a plurality of reconstructed images by inputting the generated plurality of masked inspection images into the learned image reconstruction unit.

[0153] In step S1204, the inference unit 811 of the determination device 810 synthesizes the plurality of reconstructed images that have been reconstructed to generate a synthesized image.

[0154] In step S1205, the inference unit 811 of the determination device 810 calculates the MSE by comparing the inspection image acquired in step S1201 with the synthesized image generated in step S1204, thereby determining whether the inspection image contains a defect. In addition, the inference unit 811 of the determination device 810 outputs the determination result.

[0155] In step S1206, the inference section 811 of the determination device 810 determines whether to end the determination process. If it is determined in step S1206 to continue the determination process (when it is "No" in step S1206), the process returns to step S1201.

[0156] In contrast, if it is determined in step S1206 to end the determination process (when it is "Yes" in step S1206), the determination process ends.

[0157] <Verification of the inference section>

[0158] Next, the generation accuracy in the case of reconstructing a reconstructed image and generating a composite image using the inference section 811 of the determination device 810 is verified. The verification of the generation accuracy of the composite image is performed in the following order.

[0159] 1) Prepare a comparison device.

[0160] 2) By overlapping different first masks on the same normal image, different learning data sets are generated, and each learning data set is used for learning to generate

[0161] · The inference section (the learned image reconstruction section) of the comparison device, and

[0162] · The inference section 811 (the learned image reconstruction section 1030) of the determination device 810.

[0163] 3) By inputting the same verification image into the inference section of the comparison device and the inference section 811 of the determination device 810 respectively, the reconstructed image is reconstructed and a composite image is generated.

[0164] 4) Calculate the error between the input verification image and the composite image generated in the inference section of the comparison device, and the error between the input verification image and the composite image generated in the inference section 811 of the determination device 810.

[0165] 5) By repeatedly performing the above 3) and 4) for multiple verification images and averaging the errors, the generation accuracy of the composite image is verified.

[0166] Figure 13 is a diagram for explaining the outline of the verification process. Among them, Figure 13 (a) shows the outline of the process performed by the inference section of the comparison device. As indicated by reference numeral 1310, the inference section of the comparison device uses four rectangular masks (one first mask and three second masks each including one rectangular mask piece) overlapping on each of the four divided regions of the inspection region 1311 to generate a masked verification image.

[0167] Specifically, as shown by reference numeral 1312, the inference unit of the comparison device generates four masked verification images by respectively overlapping four rectangular masks on the verification image 1301.

[0168] Next, the inference unit of the comparison device reconstructs the reconstructed images for the four generated masked verification images respectively, and generates a composite image 1313. Furthermore, the inference unit of the comparison device calculates the error between the verification image 1301 and the composite image 1313.

[0169] On the other hand, Figure 13 The (b) of represents an outline of the process performed by the inference unit 811 of the determination device 810. As shown by reference numeral 1320, the inference unit 811 of the determination device 810 uses four grid-shaped masks (including a plurality of mask pieces arranged in a grid, one first mask and three second masks) overlapping on the inspection area 1321 to generate masked verification images.

[0170] Specifically, as shown by reference numeral 1322, the inference unit 811 of the determination device 810 generates four masked verification images by respectively overlapping four grid-shaped masks on the verification image 1301.

[0171] Next, the inference unit 811 of the determination device 810 reconstructs the reconstructed images for the four generated masked verification images respectively, and generates a composite image 1323. Furthermore, the inference unit 811 of the determination device 810 calculates the error between the verification image 1301 and the composite image 1323.

[0172] Figure 14 is a diagram showing an example of the verification result. In Figure 14 the reference numeral 1410 represents the average value of the pixel values of each pixel between a plurality of verification images and a plurality of composite images generated by the inference unit of the comparison device based on the plurality of verification images.

[0173] In addition, in Figure 14 the reference numeral 1420 represents the average value of the pixel values of each pixel between a plurality of verification images and a plurality of composite images generated by the inference unit 811 of the determination device 810 based on the plurality of verification images.

[0174] As a result of comparing the average value of the errors ( = 3.69) indicated by reference numeral 1410 with the average value of the errors (2.61) indicated by reference numeral 1420, the average value of the errors indicated by reference numeral 1420 is reduced by 29.3% with respect to the average value of the errors indicated by reference numeral 1410. This indicates that the inference unit 811 of the determination device 810 has a higher generation accuracy of the composite image compared to the inference unit of the comparison device.

[0175] In addition, by improving the synthesis accuracy of the synthesized image generated by the inference unit 811 of the determination device 810, the risk of misjudgment of determining a normal image as an abnormal image can be reduced.

[0176] <Summary>

[0177] As can be seen from the above description, in the inspection system 400 of the first embodiment,

[0178] · By regularly arranging a plurality of mask pieces smaller than the mask overlapping on the size ratio inspection area at a prescribed interval in the vertical direction, horizontal direction, and diagonal direction, a first mask is generated.

[0179] · Generate a plurality of second masks corresponding to the number of intervals of the interval size, which are regularly arranged at positions corresponding to the prescribed intervals in the vertical direction, prescribed intervals in the horizontal direction, and prescribed intervals in the diagonal direction of the first mask filled.

[0180] · By overlapping the first mask on the inspection area of the normal image determined not to contain defects in the image obtained by photographing the inspection object, a masked normal image is generated.

[0181] · By inputting the generated masked normal image to perform reconstruction of the reconstructed image, and performing learning processing on the image reconstruction unit to make the reconstructed reconstructed image close to the inspection image (normal image), a learned image reconstruction unit is generated.

[0182] Therefore, according to the inspection system 400 of the first embodiment, a learned image generation AI capable of reconstructing a change in the shape of a part of the inspection object caused by manufacturing deviations within the acceptable range can be generated.

[0183] In addition, in the inspection system 800 of the first embodiment,

[0184] · By sequentially overlapping the first mask and a plurality of second masks on the inspection area of the inspection image obtained by photographing the inspection object, a plurality of masked inspection images are generated.

[0185] · When reconstructing a plurality of reconstructed images by inputting the generated plurality of masked inspection images into the learned image reconstruction unit, a synthesized image is generated by synthesizing the plurality of reconstructed images.

[0186] · Compare the inspection image with the synthesized image to determine whether the inspection image contains defects.

[0187] Thus, according to the inspection system 800 of the first embodiment, it is possible to reconstruct the change in the shape of a part of the object to be inspected caused by manufacturing deviations within the acceptable range, and it is possible to reduce the error between the inspection image and the synthesized image. As a result, according to the inspection system 800 of the first embodiment, a situation without defects will not be misjudged as the inspection image containing defects, and misjudgment can be reduced.

[0188] [Second Embodiment]

[0189] In the above first embodiment, the first mask is generated by regularly arranging a plurality of mask pieces smaller than the mask overlapping the inspection area in the vertical direction, horizontal direction, and diagonal direction at intervals equal to the size of the mask piece. In addition, in the first embodiment, three types of second masks are generated by regularly arranging mask pieces at positions filling the horizontal interval, vertical interval, and diagonal interval equal to the size of the mask piece.

[0190] However, the generation methods of the first mask and the plurality of second masks are not limited to the above. Figure 15 It is a diagram showing another example of the first mask and the plurality of second masks. Among them, Figure 15 (a) shows, for comparison, the first mask and three types of second masks shown in the first embodiment.

[0191] On the other hand, Figure 15 (b) shows the generation of

[0192] · A first mask formed by regularly arranging a plurality of mask pieces smaller than the mask overlapping the inspection area at intervals equal to the size of the mask piece in the vertical direction and at intervals twice the size of the mask piece in the horizontal and diagonal directions,

[0193] · One type of second mask formed by regularly arranging mask pieces at positions filling the vertical interval equal to the size of the mask piece, and five types of second masks formed by regularly arranging mask pieces at positions filling the horizontal and diagonal intervals twice the size of the mask piece.

[0194] In addition, Figure 15 (c) shows the generation of

[0195] · A first mask formed by regularly arranging a plurality of mask pieces smaller than the mask overlapping the inspection area at intervals twice the size of the mask piece in the vertical and diagonal directions and at intervals equal to the size of the mask piece in the horizontal direction,

[0196] · One type of second mask formed by regularly arranging mask sheets at positions filling the horizontal intervals equal to the size of the mask sheet, and five types of second masks formed by regularly arranging mask sheets at positions filling the vertical intervals and diagonal intervals twice the size of the mask sheet.

[0197] In addition, Figure 15 (d) indicates generation

[0198] · A first mask formed by regularly arranging a plurality of mask sheets smaller in size than the mask overlapping on the inspection area, with an interval equal to the size of the mask sheet in the vertical direction and an interval equal to the size of the mask sheet in the horizontal direction.

[0199] · One type of second mask formed by regularly arranging mask sheets at positions filling the horizontal intervals equal to the size of the mask sheet.

[0200] And, Figure 15 (e) indicates generation

[0201] · A plurality of mask sheets smaller in size than the mask overlapping on the inspection area and larger in size than the mask sheets shown in (a) to (d), regularly arranged with an interval equal to the size of the mask sheet in the vertical direction, horizontal direction, and diagonal direction. Figure 15 (a) to (d)

[0202] · Three types of second masks formed by regularly arranging mask sheets at positions filling the horizontal intervals, vertical intervals, and diagonal intervals equal to the size of the mask sheet.

[0203] As described above, there are various methods for generating the first mask and the plurality of second masks, which can be generated based on considerations such as the size of the acquired inspection image and the size of the defects included in the inspection image.

[0204] [Third Embodiment]

[0205] In the learning device 440 of the first embodiment described above, it has been described that the masking unit 720 generates a masked normal image using the first mask. However, the method for generating the masked normal image is not limited to this. For example, a masked normal image can also be generated using one of the plurality of second masks.

[0206] In addition, in the learning device 440 of the above-described first embodiment, the case where the masking unit 720 performs learning processing on the image reconstruction unit 730 by using one type of mask to generate a masked normal image has been described. However, the method of learning processing on the image reconstruction unit 730 is not limited to this. For example, learning processing is performed by using a plurality of masks (for example, a first mask and a second mask, or a plurality of second masks) to generate a masked normal image.

[0207] In addition, in the learning device 440 of the above-described first embodiment, a structure in which a synthesis unit is not provided is adopted. However, the structure of the learning device 440 is not limited to this, and similar to the determination device 810, a synthesis unit may also be provided. In this case, the learning device 440 inputs a plurality of masked normal images generated by sequentially overlapping the first mask and the plurality of second masks into the image reconstruction unit 730, and the synthesis unit synthesizes the plurality of reconstructed images reconstructed by the image reconstruction unit 730 to generate a synthesized image. Thus, in the learning device 440, learning processing is performed on the image reconstruction unit 730 to make the synthesized image close to the inspection image (normal image).

[0208] In addition, although the position of the inspection area where the first mask and the plurality of second masks overlap is not mentioned in the first embodiment, it is assumed that in the AOI device 410, the inspection area is adjusted to be located at the center of the inspection image. In addition, in the above-described first embodiment, although the size of the overlapping masks is not mentioned, it is assumed that the size of the overlapping masks is adjusted according to the size of the inspection area of the inspection image transmitted by the AOI device 410. However, there are no restrictions on the position and size of the inspection area of the overlapping masks, and masks of any position and size can be overlapped.

[0209] In addition, in the above-described first embodiment, details of the process of determining whether the inspection image contains a defect by performing image processing using the inspection image and the reconstructed image are not mentioned. However, as this process, for example, the following processes can be cited.

[0210] · Calculate the absolute value of the difference between each pixel of the inspection image and the reconstructed image to generate a difference image.

[0211] · By performing binarization processing on the difference image, extract the area where the absolute value of the difference is greater than or equal to the threshold value.

[0212] · Perform contour extraction processing on the binarized difference image, and extract the contour of the area where the absolute value of the difference is greater than or equal to the threshold value.

[0213] · Determine that a defect is included when the shape and size of the extracted contour satisfy the specified conditions, and determine that no defect is included when the shape and size of the extracted contour do not satisfy the specified conditions.

[0214] As described above, by determining whether a defect is included in the inspection image based on the result of image processing, the determination accuracy can be improved compared to the case of determining based on the error magnitude.

[0215] In addition, the present invention is not limited to the structures shown in the above embodiments, combinations with other elements, etc., the structures described herein. Regarding these points, changes can be made within the scope not departing from the gist of the present invention and can be appropriately determined according to its application form.

[0216] This application claims priority based on Japanese Patent Application No. 2023-054111 filed with the Japan Patent Office on March 29, 2023, and the entire contents of this Japanese patent application are incorporated herein by reference.

[0217] Reference Signs

[0218] 400 Inspection system

[0219] 410 AOI device

[0220] 440 Learning device

[0221] 441 Learning dataset generation unit

[0222] 442 Learning unit

[0223] 640 Learning dataset

[0224] 710 Image input unit

[0225] 720 Masking unit

[0226] 730 Image reconstruction unit

[0227] 750 Comparison / change unit

[0228] 810 Determination device

[0229] 811 Inference unit

[0230] 812 Output unit

[0231] 1010 Image input unit

[0232] 1020 Masking unit

[0233] 1030 Learned image reconstruction unit

[0234] 1040 Synthesis unit

[0235] 1050 Determination unit

Claims

1. A determination device, comprising: A learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and is an image reconstruction unit learned in such a way that the first reconstructed image is close to a first image, where the first image is an image obtained by photographing an inspection object and determined not to contain defects, and the first mask image is generated by overlapping a mask on an inspection area of the first image; A synthesis unit that synthesizes a plurality of second reconstructed images to generate a second synthesized image when the plurality of second reconstructed images are reconstructed by inputting a plurality of second mask images into the learned image reconstruction unit, where the plurality of second mask images are generated by sequentially overlapping a plurality of masks on an inspection area of a second image obtained by photographing an inspection object; And A determination unit that compares the second synthesized image with the second image to determine whether the second image contains defects.

2. The determination device according to claim 1, wherein The plurality of masks include a first mask and a plurality of second masks, The first mask is generated by regularly arranging a plurality of mask pieces smaller in size than the mask overlapping on the inspection area at a predetermined interval in the vertical direction, horizontal direction, and diagonal direction; The plurality of second masks corresponding to the size of the interval are generated by regularly arranging a plurality of mask pieces at positions filling the predetermined intervals in the vertical direction, horizontal direction, and diagonal direction of the first mask.

3. The determination device according to claim 1, wherein The plurality of masks include a first mask and a plurality of second masks, The first mask is generated by regularly arranging a plurality of mask pieces smaller in size than the mask overlapping on the inspection area at a predetermined interval in the vertical direction and horizontal direction; The plurality of second masks corresponding to the size of the interval are generated by regularly arranging a plurality of mask pieces at positions filling the predetermined intervals in the vertical direction and horizontal direction of the first mask.

4. The determination device according to claim 2 or 3, wherein The predetermined interval is an integer multiple of the size of the mask piece.

5. The determination device according to claim 1, wherein The determination unit Determines that the second image does not contain defects when the value calculated based on the error between the pixel values of each pixel of the second image and the second synthesized image satisfies a predetermined condition; Determines that the second image contains defects when the value calculated based on the error between the pixel values of each pixel of the second image and the second synthesized image does not satisfy the predetermined condition.

6. A learning device, comprising: A mask unit that generates a first mask image by overlapping a mask on an inspection area of a normal image obtained by photographing an inspection object and determined not to contain defects; And An image reconstruction unit that outputs a first reconstructed image when a first mask image is input, The mask is generated by regularly arranging a plurality of mask pieces smaller in size than the mask overlapping on the inspection area at least at a predetermined interval in the vertical direction and horizontal direction The image reconstruction unit learns in such a way that the first reconstructed image approaches the normal image.

7. The learning device according to claim 6, wherein the mask is generated by regularly arranging a plurality of mask pieces smaller than the mask overlapping on the size ratio inspection area at a predetermined interval in the vertical direction, horizontal direction, and diagonal direction.

8. A determination method for causing a computer of a determination device to execute the following steps The determination device stores a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and is an image reconstruction unit learned in such a way that the first reconstructed image approaches a first image. The first image is an image obtained by photographing an inspection object and determined not to contain a defect. The first mask image is generated by overlapping a mask on an inspection area of the first image. The steps include: A synthesis step: when a plurality of second reconstructed images are reconstructed by inputting a plurality of second mask images into the learned image reconstruction unit, synthesizing the plurality of second reconstructed images to generate a second synthesized image. The plurality of second mask images are generated by sequentially overlapping a plurality of masks on an inspection area of a second image obtained by photographing an inspection object. and A determination step of comparing the second synthesized image with the second image to determine whether the second image contains a defect.

9. A learning method for causing a computer of a learning device to execute: A masking step of generating a first mask image by overlapping a mask on an inspection area of a normal image determined not to contain a defect in an image obtained by photographing an inspection object; and An image reconstruction step of outputting a first reconstructed image by the image reconstruction unit when a first mask image is input. The mask is generated by regularly arranging a plurality of mask pieces smaller than the mask overlapping on the size ratio inspection area at a predetermined interval at least in the vertical direction and horizontal direction. In the image reconstruction step, the image reconstruction unit learns in such a way that the first reconstructed image approaches the normal image.

10. A determination program for causing a computer of a determination device to execute the following steps The determination device stores a learned image reconstruction unit that outputs a first reconstructed image when a first mask image is input, and is an image reconstruction unit learned in such a way that the first reconstructed image approaches a first image. The first image is an image obtained by photographing an inspection object and determined not to contain a defect. The first mask image is generated by overlapping a mask on an inspection area of the first image. A synthesis step of synthesizing a plurality of second reconstructed images to generate a second synthesized image when a plurality of second reconstructed images are reconstructed by inputting a plurality of second mask images into the learned image reconstruction unit. The plurality of second mask images are generated by sequentially overlapping a plurality of masks on an inspection area of a second image obtained by photographing an inspection object. and A determination step of comparing the second composite image with the second image to determine whether the second image contains a defect.

11. A learning program for causing a computer of a learning device to execute the following steps: A masking step of generating a first masked image by overlapping a mask on an inspection area of a normal image determined not to contain a defect in an image obtained by photographing an object to be inspected; and An image reconstruction step of outputting a first reconstructed image by an image reconstruction unit when the first masked image is input, The mask is generated by arranging a plurality of mask pieces smaller than the mask overlapping the inspection area in size at regular intervals in at least the vertical and horizontal directions in a regular manner. In the image reconstruction step, the image reconstruction unit learns in such a way that the first reconstructed image approaches the normal image.

Images