Inspection system, inspection method and inspection procedure

Abstract

An inspection system, an inspection method, and an inspection program are provided, which are used to reduce the workload of inspectors in the inspection system. The inspection system includes: an image reproduction unit that is trained to reproduce a first image before masking from a first mask image, the first mask image being generated by masking a portion of the first image that is determined to contain no defects among the images obtained by photographing an inspection object; and a determination unit that determines whether a defect is contained in the second image based on a reproduced image reproduced by inputting a second mask image to the image reproduction unit and a second image, the second mask image being generated by masking a portion of the second image obtained by photographing a new inspection object.

Description

Technical Field

[0001] The invention relates to an inspection system, an inspection method and an inspection program. Background Art

[0002] Conventionally, there is known an inspection system that takes an image of an inspection object such as a printed circuit board and compares the image with a reference image to inspect whether the inspection object is a conforming product or a defective product.

[0003] In this inspection system, for example, defect candidates are detected excessively so as not to treat defective products as good products, and finally an inspector visually inspects images of the detected defect candidates, thereby preventing missed detection of defects.

[0004] <Prior Art Literature>

[0005] <Patent Documents>

[0006] Patent Document 1: Japanese Patent Application Publication No. 2013-098267

[0007] Patent Document 2: Japanese Patent Application Publication No. 2020-052474 Summary of the invention

[0008] <Problems to be Solved by the Invention>

[0009] However, in the case of the above inspection system, since manufacturing deviations within the range of qualified products are also detected as defect candidates, there are many images of defect candidates to be visually inspected by inspectors, which increases the workload of inspectors.

[0010] According to one aspect, the present invention aims to reduce the workload of inspectors in an inspection system.

[0011] <Methods used to solve the problem>

[0012] According to one embodiment, an inspection system is provided, including: an image reproduction unit, which is trained in a manner of reproducing a first image before masking from a first masked image, wherein the first masked image is generated by masking a portion of the first image obtained by photographing an inspection object and is judged to contain no defects; and a determination unit, which determines whether defects are contained in the second image based on a reproduced image reproduced by inputting a second masked image into the image reproduction unit and a second image, wherein the second masked image is generated by masking a portion of the second image obtained by photographing a new inspection object.

[0013] <Effects of the Invention>

[0014] It can reduce the workload of inspectors in the inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 : is a diagram showing an example of the system configuration of the inspection system in the training phase.

[0016] Figure 2 It is a diagram showing an example of the hardware configuration of the training device.

[0017] Figure 3 2 is a diagram showing a specific example of processing performed by a training data set generating unit of a training device.

[0018] Figure 4 FIG. 1 is a first diagram showing a specific example of processing performed by a training unit of a training device.

[0019] Figure 5 : is a diagram showing an example of the system configuration of the inspection system in the inspection phase.

[0020] Figure 6 This is a diagram showing an example of the hardware configuration of the estimation device.

[0021] Figure 7 FIG. 1 is a first diagram showing a specific example of processing performed by an estimating unit of an estimating device.

[0022] Figure 8 This is a first flowchart showing the flow of training processing in the inspection system.

[0023] Fig. 9 This is a first flowchart showing the flow of inspection processing in the inspection system.

[0024] Fig.10 FIG. 2 is a second diagram showing a specific example of processing performed by the training unit of the training device.

[0025] Fig.11 FIG. 2 is a second diagram showing a specific example of processing performed by the estimation unit of the estimation device.

[0026] Fig.12 This is a second flowchart showing the flow of the training process in the inspection system.

[0027] Fig.13 This is a second flow chart showing the flow of the inspection process in the inspection system.

[0028] Explanation of symbols

[0029] 100: Check system

[0030] 110: AOI device

[0031] 120: Training device

[0032] 121: Training data set generation department

[0033] 122: Training Department

[0034] 310: Training Dataset

[0035] 410: Covering

[0036] 420: Image reproduction unit

[0037] 500: Check system

[0038] 510: Inference Device

[0039] 511: Inference Department

[0040] 512: Output

[0041] 710: Covering

[0042] 720: Training completed image reproduction unit

[0043] 730: Judgment Department DETAILED DESCRIPTION

[0044] Hereinafter, each embodiment will be described in detail with reference to the drawings. It should be noted that in this specification and the drawings, components having substantially the same functional configuration may be denoted by the same reference numerals to omit repeated descriptions.

[0045] [First embodiment]

[0046] <System Configuration of Inspection System in Training Phase>

[0047] First, the system configuration in the training phase of the inspection system according to the first embodiment will be described. Figure 1 : is a diagram showing an example of the system configuration of the inspection system in the training phase.

[0048] like Figure 1 As shown, the inspection system 100 in the training stage includes an automatic optical inspection (AOI: Automated Optical Inspection) device 110 and a training device 120 .

[0049] The AOI device 110 performs an automatic appearance inspection on the printed substrate 130. The AOI device 110 scans the printed substrate 130 using a camera and inspects various inspection items to inspect candidate defects. Inspection items inspected by the AOI device 110 include, for example, circuit width, circuit spacing, missing pads / no pads, circuit shorts, etc.

[0050] The images 140 of each region including the defect candidates detected by the AOI device 110 are sent to the training device 120 and sent to the inspection line. On the inspection line, inspectors such as the inspector 111 visually inspect the images 140 of each region including the defect candidates. It should be noted that it is assumed that the AOI device 110 is set to excessively detect images of each region including the defect candidates in a manner that does not treat a defective product as a qualified product.

[0051] The inspector 111 visually checks whether the image 140 of each region contains a defect, and finally determines whether the printed substrate 130 is a qualified product or a defective product. Specifically, when no defect is contained in the image 140 of each region containing a defect candidate, the printed substrate 130 is determined to be a qualified product. In addition, when any one of the images 140 of each region containing a defect candidate contains a defect, the printed substrate 130 is determined to be a defective product.

[0052] It should be noted that the inspector 111 notifies the training device 120 of the result of the visual inspection (the result of determining whether the image 140 of each region contains a defect). Figure 1 In the example of , “Visual inspection result: OK” indicates that the image including the defect candidate region is judged to contain no defect, and “Visual inspection result: NG” indicates that the image including the defect candidate region is judged to contain a defect.

[0053] A training program is installed in the training device 120 , and the training device 120 functions as a training data set generating unit 121 and a training unit 122 by executing the program.

[0054] The training data set generation unit 121 extracts the visual inspection results of the inspector 111 and the images determined to contain no defects from the images 140 of each region containing defect candidates sent from the AOI device 110. In addition, the training data set generation unit 121 associates the extracted images of each region with the visual inspection results and stores them in the training data set storage unit 123 as training data sets.

[0055] The training unit 122 reads out the images of each region included in the training data set stored in the training data set storage unit 123. In addition, the training unit 122 generates a masked image by masking a part of the image of each region, and performs training processing on the model in a manner of reproducing the image before masking from the generated masked image.

[0056] It should be noted that, for the model trained by the training unit 122, for example, a model for completing blank areas of an image is used. In this model, when an image with a blank area is input, the blank area is completed based on the surrounding area of ​​the blank area to reproduce the original image before the blank area is left. Hereinafter, this model is referred to as an "image reproduction unit".

[0057] <Hardware composition of training device>

[0058] Next, the hardware configuration of the training device 120 will be described. Figure 2 FIG. 1 is a diagram showing an example of the hardware configuration of a training device. Figure 2 As shown, the training device 120 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a driver device 206. It should be noted that the hardware components of the training device 120 are connected to each other via a bus 207.

[0059] The processor 201 includes various computing devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 201 reads various programs (such as training programs) into the memory 202 and executes them.

[0060] The memory 202 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 201 and the memory 202 constitute a so-called computer, and the processor 201 executes various programs read from the memory 202, so that the computer realizes the above-mentioned functions (training data set generation unit 121, training unit 122).

[0061] The auxiliary storage device 203 stores various programs or various data used when the processor 201 executes the various programs. For example, the training data set storage unit 123 is implemented in the auxiliary storage device 203.

[0062] The I / F device 204 is a connection device that connects the operation device 210 and the display device 211, which are examples of external devices, to the training device 120. The I / F device 204 receives an operation on the training device 120 (for example, an operation to input the result of the visual inspection by the inspector 111, or an operation to input an instruction for training processing by the administrator (not shown) of the training device 120, etc.) via the operation device 210. In addition, the I / F device 204 outputs the result of the training processing of the training device 120, etc., and displays it to the administrator of the training device 120 via the display device 211.

[0063] The communication device 205 is a communication device for communicating with another device (the AOI device 110 in this embodiment).

[0064] The drive device 206 is a device for setting the recording medium 212. The recording medium 212 referred to here includes a medium that records information optically, electrically or magnetically, such as a CD-ROM, a floppy disk, a magneto-optical disk, etc. In addition, the recording medium 212 may include a semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0065] It should be noted that the various programs installed in the auxiliary storage device 203 are installed by, for example, setting the distributed recording medium 212 in the drive device 206 and reading the various programs recorded in the recording medium 212 through the drive device 206. Alternatively, the various programs installed in the auxiliary storage device 203 can be installed by downloading from the network via the communication device 205.

[0066] <Details of each part of the training device>

[0067] Next, the details of each part of the training device 120 (here, the training data set generating unit 121 and the training unit 122) will be described.

[0068] (1) Specific Example of Processing Performed by the Training Dataset Generating Unit

[0069] Figure 3 FIG. 1 is a diagram showing a specific example of the processing performed by the training data set generating unit of the training device. Figure 3 As shown, when images 301 to 306 of respective regions including defect candidates are transmitted from the AOI device 110 , for example, an image of “visual inspection result: OK” is extracted in the training data set generating unit 121 .

[0070] Figure 3The example shows that images 301, 302, 304, and 305 among images 301 to 306 of each region are “visual inspection result: NG” images. Therefore, in the training data set generation unit 121, images 303 and 306 (“visual inspection result: OK” images) of each region are extracted to generate a training data set 310.

[0071] like Figure 3 As shown, the training data set 310 has “ID”, “image”, and “visual inspection result” as information items.

[0072] The identifier for identifying the image of each region is stored in "ID". The image of each region is stored in "Image". The result of the visual inspection of the image of each region is stored in "Visual Inspection Result". It should be noted that since only images of "Visual Inspection Result: OK" are stored in the training data set 310, only "OK" is stored in "Visual Inspection Result".

[0073] In this way, the training data set generation unit 121 uses

[0074] Images of each region including candidate defects detected by the AOI device 110, and

[0075] Images judged by the inspector 111 to contain no defects (images with “visual inspection result: OK”)

[0076] To generate a training data set 310.

[0077] (2) Specific examples of processing performed by the training unit

[0078] Figure 4 FIG. 1 is a first diagram showing a specific example of processing performed by a training unit of a training device. Figure 4 As shown, the training unit 122 includes a masking unit 410 , an image reproduction unit 420 , and a comparison / changing unit 430 .

[0079] The masking unit 410 reads out an image of each area stored in the "image" of the training data set 310 stored in the training data set storage unit 123 (an example of a first image. For example, image 440). In addition, the masking unit 410 generates a masked image (an example of a first masked image. For example, masked images 440_1 to 440_n) by masking a portion of the read image 440. In addition, the masking unit 410 inputs the generated masked images 440_1 to 440_n into the image reproduction unit 420. The masking position when the masking unit 410 masks the read image 440 is random. For example, the masking unit 410 masks at approximately 9 to 16 positions.

[0080] The image reproduction unit 420 reproduces the pre-mask images 441_1 to 441_n based on the mask images 440_1 to 440_n, and outputs the reproduced images to the comparison / modification unit 430.

[0081] The comparing / changing unit 430 compares the images 441_1 to 441_n reproduced by the image reproducing unit 420 with the image 440 before masking read by the masking unit 410 , and updates the model parameters of the image reproducing unit 420 so that the two images coincide with each other.

[0082] Thus, the image reproduction unit 420 is trained by reproducing the image 440 before masking from the mask images 440_1 to 440_n generated by the masking unit 410. In this way, by setting the configuration using the image reproduction unit 420, unsupervised training (learning) processing can be performed by the training device 120. In addition, training processing can be performed using only images of each area that does not contain defects (images with "visual inspection result: OK"). Thus, compared with the case where images of each area containing various types of defects (images with "visual inspection result: NG") are collected for training processing, the training cost can be reduced.

[0083] It should be noted that the trained image reproduction unit that has been trained to reproduce the image before masking will be used in the inspection phase described later.

[0084] <System Configuration of Inspection System in Inspection Phase>

[0085] Next, the system configuration in the inspection phase of the inspection system according to the first embodiment will be described. Figure 5 : is a diagram showing an example of the system configuration of the inspection system in the inspection phase.

[0086] like Figure 5 As shown, the inspection system 500 in the inspection stage has the AOI device 110 and the inference device 510 .

[0087] Since the AOI device 110 is the same as the AOI device 110 of the inspection system 100 in the training phase, the description thereof is omitted here.

[0088] The inference device 510 has an inspection program installed therein, and when the program is executed, the inference device 510 functions as an inference unit 511 and an output unit 512 .

[0089] The inference unit 511 includes a training image reproducing unit generated in the training phase. The inference unit 511 acquires the image 540 of each region sent from the AOI device 110 by performing an automatic appearance inspection on the printed substrate 530 as the inspection object. In addition, the inference unit 511 generates a masked image by masking a part of the acquired image 540 of each region, and then inputs the masked image to the training image reproducing unit. In addition, the inference unit 511 compares the image reproduced by the training image reproducing unit with the image 540 before masking, thereby determining whether a defect is included in the image 540 of each region. In addition, the inference unit 511 notifies the output unit 512 of the determination result.

[0090] The output unit 512 outputs the judgment result notified by the inference unit 511 to the inspection line. On the inspection line, the inspector 111 performs a visual inspection on the images of each area containing the defect candidate. However, in the detection stage, on the inspection line, with reference to the judgment result output by the output unit 512, the images 540 of each area containing the defect candidate that are judged by the inference device 510 as not containing defects are excluded. Then, on the inspection line, the images 550 of each area containing the defect candidate that are judged by the inference device 510 as containing defects are assigned to the visual inspection. In other words, the image 550 is output on the output unit 512, so that the image 550 judged as containing defects is visually inspected.

[0091] In this way, when the AOI device 110 performs automatic appearance inspection on the printed circuit board 530 and detects images of each area including defect candidates, the images determined by the inference device 510 as containing defects are assigned to visual inspection on the inspection line. Thus, the inspection system 500 can reduce the number of images assigned to visual inspection, thereby reducing the workload of the inspector 111 for visual inspection.

[0092] <Hardware Configuration of Estimation Device>

[0093] Next, the hardware configuration of the estimation device 510 will be described. Figure 6 FIG. 1 is a diagram showing an example of the hardware configuration of the inference device. Figure 6 As shown, since the hardware configuration of the inference device 510 is basically the same as the hardware configuration of the training device 120, the description here focuses on the differences between the hardware configuration of the inference device 510 and the training device 120.

[0094] The processor 601 reads various programs (such as inspection programs, etc.) to the memory 602 and executes them. The processor 601 executes the various programs read to the memory 602, so that the computer composed of the processor 601 and the memory 602 realizes the above-mentioned functions (inference unit 511, output unit 512).

[0095] <Details of each part of the inference device>

[0096] Next, the details of each part of the estimation device 510 (here, the estimation unit 511) are described. Figure 7 FIG. 1 is a first diagram showing a specific example of processing performed by the inference unit of the inference device. Figure 7 As shown, the inference unit 511 includes a masking unit 710 , a trained image reproduction unit 720 , and a determination unit 730 .

[0097] The masking unit 710 obtains an image of each area including defect candidates sent from the AOI device 110 (an example of a second image, such as image 741), and generates a masked image (an example of a second masked image, such as masked images 741_1 to 741_n) by masking a portion of the obtained image. In addition, the masking unit 710 inputs the generated masked images 741_1 to 741_n into the trained image reproduction unit 720.

[0098] The trained image reproduction unit 720 is a trained model generated by training the image reproduction unit 420 in the training phase. The trained image reproduction unit 720 reproduces an image before masking (an example of a second reproduced image, such as images 751-1 to 751_n) based on the mask images 741-1 to 741_n.

[0099] The determination unit 730 compares the images 751_1 to 751_n reproduced by the trained image reproduction unit 720 with the image 741 before masking obtained by the masking unit 710 to determine whether a defect is included.

[0100] Specifically, the determination unit 730 first calculates the mean square error (MSE) of the pixel values ​​of each pixel between the reproduced image 751_1 and the image 741 before masking. Next, the determination unit 730 calculates the mean square error (MSE) of the pixel values ​​of each pixel between the reproduced image 751_2 and the image 741 before masking. Hereinafter, similarly, the mean square error (MSE) of the pixel values ​​of each pixel between the reproduced image 751_3 and the image 741 before masking to the mean square error (MSE) of the pixel values ​​of each pixel between the reproduced image 751_n and the image 741 before masking are calculated.

[0101] Next, the determination unit 730 determines whether each calculated MSE is less than a predetermined threshold value (Th). When it is determined that each calculated MSE is less than a predetermined threshold value, it is determined that there is no defect in the image 741 ("determination result: OK"). On the other hand, when it is determined that any one of the calculated MSEs exceeds the predetermined threshold value, it is determined that there is a defect in the image 741 ("determination result: NG").

[0102] In this way, the reproduced image is compared with the image before masking to determine whether a defect is included in the inference unit 511. Thus, for example, compared with the case where the image before masking is compared with the reference image to determine whether a defect is included, a determination result corresponding to manufacturing deviation within the range of qualified products can be output.

[0103] In other words, the inspection system 500 can appropriately exclude images that are likely to be "visual inspection results: OK" ("judgment result: OK") from the allocation of visual inspections for images of each region containing defect candidates detected in a manner to prevent missed defects. Thus, the inspection system 500 can reduce the number of images allocated for visual inspections, thereby reducing the workload of inspectors.

[0104] <Flow of training process>

[0105] Next, the flow of the training process in the inspection system 100 will be described. Figure 8 This is a first flowchart showing the flow of training processing in the inspection system.

[0106] In step S801 , the training data set generator 121 of the training device 120 acquires images of each region including defect candidates from the AOI device 110 .

[0107] In step S802 , the training data set generation unit 121 of the training device 120 extracts images of “visual inspection result: OK” from the acquired images of each region, and generates a training data set.

[0108] In step S803 , the training unit 122 of the training device 120 generates a mask image by masking a part of the image of each region included in the training data set, and trains the image reproduction unit to reproduce the image before masking from the generated mask image.

[0109] In step S804, the training unit 122 of the training device 120 determines whether to end the training process. If it is determined in step S804 that the training process is to be continued (if "No" in step S804), the process returns to step S801.

[0110] On the other hand, if it is determined in step S804 that the training process is to be ended (if YES in step S804 ), the process proceeds to step S805 .

[0111] In step S805, the training unit 122 of the training device 120 outputs the trained image reproduction unit and ends the training process.

[0112] <Inspection Process>

[0113] Next, the flow of the inspection process in the inspection system 500 will be described. Fig. 9 This is a first flowchart showing the flow of inspection processing in the inspection system.

[0114] In step S901 , the estimation unit 511 of the estimation device 510 acquires images of each region including defect candidates from the AOI device 110 .

[0115] In step S902 , the estimation unit 511 of the estimation device 510 generates a mask image by masking a part of the acquired image of each region, and reproduces the image before masking by inputting the generated mask image to the trained image reproduction unit.

[0116] In step S903 , the estimation unit 511 of the estimation device 510 compares the reproduced image with the image before masking and calculates the MSE, thereby determining whether the image before masking includes a defect.

[0117] Specifically, in the inference unit 511 of the inference device 510, for example, the acquired image is divided into 9 parts, 3 parts in the vertical direction and 3 parts in the horizontal direction. Next, in the inference unit 511 of the inference device 510, the process of calculating the MSE for the masked image generated by masking any of the segmented areas is performed 9 times while sequentially changing the masked segmented areas. Then, in the case where the MSE for any of the masked images exceeds the threshold value, the inference unit 511 of the inference device 510 determines that the acquired image contains a defect ("determination result: NG").

[0118] In step S904, the output unit 512 of the inference device 510 reports the determination result. Thus, the images of each region including defect candidates detected by the AOI device 110 through the automatic appearance inspection are classified into images determined not to contain defects (images with "determination result: OK") and images determined to contain defects (images with "determination result: NG"). Therefore, only images determined to contain defects (images with "determination result: NG") can be assigned to visual inspection.

[0119] In step S905, the inference unit 511 of the inference device 510 determines whether to end the inspection process. If it is determined in step S905 that the inspection process is to be continued (if "No" in step S905), the process returns to step S901.

[0120] On the other hand, if it is determined in step S905 that the checking process is to be ended (if YES in step S905 ), the checking process is ended.

[0121] <Conclusion>

[0122] As is clear from the above description, the inspection system according to the first embodiment performs the following processing. A mask image is generated by masking a portion of an image obtained by photographing a printed circuit board that is determined not to contain a defect. In addition, a training process is performed on the image reproduction unit in such a manner that an image before masking is reproduced from the generated mask image.

[0123] A masked image is generated by masking a portion of an image obtained by photographing a new printed substrate. Furthermore, the generated masked image is input to a trained image reproduction unit to reproduce the image before masking, and the reproduced image is compared with the image before masking to determine whether the image before masking contains a defect.

[0124] In this way, by comparing the reproduced image with the image before masking to determine whether a defect is included, it is possible to output a determination result corresponding to manufacturing variations within the range of acceptable products.

[0125] In other words, according to the first embodiment, images that are likely to be "visual inspection results: OK" ("judgment result: OK") can be appropriately excluded from the allocation of visual inspection for images of each area containing defect candidates detected in a manner to prevent missed detection of defects. Thus, according to the first embodiment, the number of images allocated for visual inspection can be reduced, thereby reducing the workload of inspectors.

[0126] [Second embodiment]

[0127] In the first embodiment described above, a case where a mask image generated by masking a part of an image of each region is input to the image reproduction unit is described as an example. In contrast, in the second embodiment, a case where CAD (Computer-aided design) data is superimposed on a mask image generated by masking a part of an image of each region and then input to the image reproduction unit is described. The second embodiment is described below, centering on the differences from the first embodiment.

[0128] <Specific Example of Processing Performed by the Training Unit>

[0129] First, a specific example of the processing performed by the training unit 122 among the components of the training device 120 according to the second embodiment will be described. Fig.10 FIG. 2 is a second diagram showing a specific example of processing performed by the training unit of the training device.

[0130] like Fig.10 As shown, in the second embodiment, it is assumed that when the training unit 122 performs processing, the training data set 1010 is stored in the training data set storage unit 123.

[0131] The training data set 1010 also includes “CAD data” as an information item in addition to the information items of the training data set 310. In the “CAD data”, CAD data of regions corresponding to the images of the regions stored in the “image” are stored. Fig.10 The example of exemplifies a case where CAD data 1011 of a region corresponding to an image 1012 is stored.

[0132] In addition, if Fig.10 As shown, the training unit 122 includes a masking unit 410 , a superimposing unit 1020 , an image reproducing unit 1030 , and a comparing / changing unit 1040 .

[0133] The covering portion 410 is Figure 4 The same as the covering portion 410 described in Fig.10 The example of FIG. 1 shows a case where a mask image 1031 is generated by reading an image 1012 stored in “image” of the training data set 1010 and masking a part of the image 1012.

[0134] The superimposing unit 1020 superimposes a corresponding area of ​​the CAD data on the masked area in the mask image generated by the masking unit 410 to generate a superimposed image. Fig.11 The example of FIG. 1 shows a case where a corresponding area of ​​the CAD data 1011 is superimposed on a masked area in the mask image 1141 to generate a superimposed image 1032 .

[0135] The image reproduction unit 1030 reproduces the image before masking based on the superimposed image generated by the superimposition unit 1020, and outputs it to the comparison / change unit 1040. Fig.10 The example of FIG. 1 shows a case where an image 1033 before masking is reproduced based on the superimposed image 1032 generated by the superimposing unit 1020 and outputted to the comparing / changing unit 1040 .

[0136] The comparing / changing unit 1040 compares the image 1033 reproduced by the image reproducing unit 1030 with the image 1012 before masking read by the masking unit 410 , and updates the model parameters of the image reproducing unit 1030 so that the two images coincide with each other.

[0137] Thus, the image reproduction unit 1030 is trained to reproduce the image 1012 before masking from the superimposed image 1032 generated by the superimposing unit 1020. The trained image reproduction unit trained to reproduce the image before masking is used in the inspection stage described later.

[0138] <Specific Example of Processing by Each Part of the Estimation Device>

[0139] Next, a specific example of the processing performed by the estimation unit 511 of the estimation device 510 will be described. Fig.11 FIG. 2 is a second diagram showing a specific example of processing performed by the inference unit of the inference device. Fig.11 As shown, the inference unit 511 includes a masking unit 710 , a superimposition unit 1110 , a trained image reproduction unit 1120 , and a determination unit 1130 .

[0140] The covering portion 710 and Figure 7 The same as the covering portion 710, Fig.11 The example of FIG. 1 shows a case where a mask image 1141 is generated by masking a part of the image 1101 sent from the AOI device 110. It should be noted that the image 1101 is an image including a defect such as a short circuit.

[0141] The superimposing unit 1110 superimposes a corresponding area of ​​the CAD data on the masked area in the mask image generated by the masking unit 710 to generate a superimposed image. Fig.11 The example of FIG. 1 shows a case where a corresponding area of ​​the CAD data 1102 is superimposed on a masked area in the mask image 1141 to generate a superimposed image 1142 .

[0142] The trained image reproduction unit 1120 is a trained model generated by training the image reproduction unit 1030 in the training phase. The trained image reproduction unit 1120 reproduces the image before masking based on the superimposed image notified by the superimposition unit 1110. Fig.11 The example of exemplifies a case where an image 1143 before masking is reproduced based on a superimposed image 1142 notified from the superimposing unit 1110 .

[0143] The determination unit 1130 compares the image 1143 reproduced by the trained image reproduction unit 1120 with the image 1101 before masking obtained by the masking unit 710 to determine whether a defect is included. Fig.11 In the example, as a result of comparing the image 1143 reproduced by the trained image reproduction unit 1120 with the image 1101 before masking obtained by the masking unit 710, the MSE exceeds a predetermined threshold and is determined to contain defects ("determination result: NG").

[0144] Thus, in the case of an image 1101 including a defect such as a short circuit, it is difficult to operate in a manner of reproducing an image of "visual inspection result: OK" by only using the masked image described in the first embodiment. This is because, in the case of a defect such as a short circuit, the periphery of the masked area does not contain information necessary to complete the masked area.

[0145] In this regard, as described above, by superimposing the corresponding area of ​​the CAD data on the masked area, it is possible to operate in a manner of reproducing an image of "visual inspection result: OK". As a result, the error between the reproduced image and the image before masking becomes larger, so that the situation including defects such as circuit short circuits can be accurately determined.

[0146] <Flow of training process>

[0147] Next, the flow of the training process in the inspection system 100 according to the second embodiment will be described. Fig.12 is a second flowchart showing the flow of training processing in the inspection system. Figure 8 The difference from the first flowchart shown is steps S1201 to S1203.

[0148] In step S1201 , the training data set generating unit 121 of the training device 120 acquires CAD data corresponding to the image of each region acquired in step S801 .

[0149] In step S1202, the training data set generation unit 121 of the training device 120 extracts images of "visual inspection result: OK" from the acquired images of each region. In addition, the training data set generation unit 121 of the training device 120 associates the extracted images of each region with the CAD data of the corresponding region to generate a training data set.

[0150] In step S1203 , the training unit 122 of the training device 120 generates a mask image by masking a part of the image of each region included in the training data set, and generates a superimposed image by superimposing the corresponding region of the CAD data.

[0151] In the following, since steps S803 to S805 have been used Figure 8 The description has been given, so the description is omitted here.

[0152] <Inspection Process>

[0153] Next, the flow of the inspection process in the inspection system 500 according to the second embodiment will be described. Fig.13 is a second flow chart showing the flow of inspection processing in the inspection system. Fig. 9 The difference from the first flowchart shown is steps S1301 to S1302.

[0154] In step S1301 , the estimation unit 511 of the estimation device 510 acquires CAD data of the region corresponding to the image of each region acquired in step S901 .

[0155] In step S1302 , the estimation unit 511 of the estimation device 510 generates a mask image by masking a part of the acquired image of each region, and then generates a superimposed image by superimposing the corresponding region of the CAD data.

[0156] In the following, since steps S902 to S905 have been used Fig. 9 The description has been given, so the description is omitted here.

[0157] <Conclusion>

[0158] As is clear from the above description, the inspection system according to the second embodiment performs the following processing. A mask image is generated by masking a portion of an image determined to contain no defects among images obtained by photographing a printed circuit board, and then a corresponding area of ​​CAD data is superimposed to generate a superimposed image. In addition, a training process is performed on the image reproduction unit in such a manner that an image before masking is reproduced from the generated superimposed image.

[0159] A mask image is generated by masking a part of an image obtained by photographing a new printed substrate, and then a corresponding area of ​​the CAD data is superimposed to generate a superimposed image. In addition, the generated superimposed image is input to a trained image reproduction unit to reproduce the image before masking, and the reproduced image is compared with the image before masking to determine whether the image before masking contains defects.

[0160] In this way, by configuring the training process to reproduce the image of "visual inspection result: OK" from the superimposed image, it is possible to accurately determine the situation including defects such as circuit short circuits. Therefore, according to the second embodiment, it is possible to avoid erroneous determination while enjoying the same effects as the first embodiment.

[0161] [Third Embodiment]

[0162] In the first embodiment, the mask unit 410 generates a plurality of mask images (e.g., mask images 440_1 to 440_n) for one image (e.g., image 440) during the training phase. However, the number of mask images generated from one image is not limited to multiple, and for example, only one mask image may be generated from one image.

[0163] In the first embodiment, the specific number of times the training process is performed by the training unit 122 is not mentioned. However, for example, when generating a mask image from one image,

[0164] Prepare 10,000 images as a training dataset 310,

[0165] Select 20 images and mask each part at a random position and size for each image.

[0166] Using these 20 mask images, the process for updating the model parameters of the image reproduction unit 420 is performed 500 times (= 10000 images / 20 images).

[0167] These treatments were repeated for approximately 10,000 sets until the error converged.

[0168] As a result, the inference unit 511 can output a highly accurate determination result.

[0169] In the second embodiment, CAD data is superimposed on the mask image as an example, but the data superimposed on the mask image is not limited to CAD data. For example, in the inspection stage, a corresponding area of ​​an image of "visual inspection result: OK" can be superimposed on the mask image.

[0170] Furthermore, in the above-described first embodiment and second embodiment, the case where the inspection target is a printed circuit board has been described, but the inspection target is not limited to the printed circuit board.

[0171] In the first and second embodiments, the images to be assigned to the visual inspection are determined based on the determination results. However, the method of using the determination results is not limited to this. For example, the determination results may be displayed to the inspector 111 to assist the visual inspection.

[0172] Furthermore, in the above-described first and second embodiments, the training device 120 and the estimation device 510 are configured as independent entities, but the training device 120 and the estimation device 510 may be configured as an integrated entity.

[0173] It should be noted that the present invention is not limited to the configuration shown here, for example, it is not limited to the combination of the configurations and other elements listed in the above embodiments. These contents can be changed within the scope of the purpose of the present invention and can be appropriately determined according to its application form.

Claims

1. An inspection system, comprising: an image reproducing unit trained to reproduce a first image before masking from a first masked image generated by masking a portion of the first image determined to contain no defects among images obtained by photographing the inspection object; as well as a determination unit that determines whether a defect is included in the second image based on a reproduced image reproduced by inputting a second mask image to the image reproduction unit and the second image, the second mask image being generated by masking a part of the second image obtained by photographing a new inspection object, The image reproducing unit is trained in a manner of reproducing the first image before masking from a superimposed image, wherein the superimposed image is formed by superimposing CAD data on a masked area in the generated first masked image. The determination unit determines whether a defect is included in the second image based on a reproduced image reproduced by inputting a superimposed image formed by superimposing CAD data on the generated second mask image to the image reproduction unit and the second image.

2. The inspection system according to claim 1, wherein: The determination unit When a value calculated based on an error in pixel values ​​of each pixel between the reproduced image and the second image is equal to or less than a predetermined threshold value, it is determined that the second image does not contain a defect, and When a value calculated based on an error in pixel values ​​of each pixel between the reproduced image and the second image exceeds a predetermined threshold value, it is determined that a defect is included in the second image.

3. The inspection system according to claim 2, further comprising: The output unit outputs the second image so as to perform a visual inspection on the second image when the determination unit determines that the second image contains a defect.

4. The inspection system according to claim 1, further comprising: The training unit performs a training process on the image reproduction unit in such a manner that the first image before masking is reproduced from the first mask image.

5. The inspection system according to claim 1, further comprising: The training unit performs a training process on the image reproduction unit in such a manner that the first image before masking is reproduced from the superimposed image.

6. An inspection method comprising: A determination step of determining whether a defect is included in an image reproduced by inputting a second masked image into an image reproduction unit and the second image, wherein the image reproduction unit is trained to reproduce a first image before masking from the first masked image, the first masked image being generated by masking a portion of the first image judged to contain no defect among an image obtained by photographing an inspection object, and the second masked image being generated by masking a portion of the second image obtained by photographing a new inspection object, The image reproducing unit is trained in a manner of reproducing the first image before masking from a superimposed image, wherein the superimposed image is formed by superimposing CAD data on a masked area in the generated first masked image. In the determination step, whether the second image includes a defect is determined based on a reproduced image reproduced by inputting a superimposed image into the image reproduction unit and the second image, wherein the superimposed image is formed by superimposing CAD data on the generated second mask image.

7. A computer program product, comprising a checking program, wherein the checking program is used to cause a computer to execute the following steps: A determination step of determining whether a defect is included in an image reproduced by inputting a second masked image into an image reproduction unit and the second image, wherein the image reproduction unit is trained to reproduce a first image before masking from the first masked image, the first masked image being generated by masking a portion of the first image judged to contain no defect among an image obtained by photographing an inspection object, and the second masked image being generated by masking a portion of the second image obtained by photographing a new inspection object, in, The image reproducing unit is trained to reproduce the first image before masking from a superimposed image, wherein the superimposed image is formed by superimposing CAD data on a masked area in the generated first mask image. In the determination step, whether the second image includes a defect is determined based on a reproduced image reproduced by inputting a superimposed image into the image reproduction unit and the second image, wherein the superimposed image is formed by superimposing CAD data on the generated second mask image.

Images