Teacher data generation device, teacher data generation method, and program

Abstract

To easily generate teacher data including an image where an unnecessary area other than a defective area is reduced.SOLUTION: A teacher data generation device 4 comprises: an image reception section 41 for receiving a predetermined size of defective image including a defect detection area and also receiving defective information indicating the range of the detection area in the defective image from an inspection device 2 for imaging an object so as to detect a defect; a segmented image generation section 42 for segmenting an area including the detection area from the defective image as a segmented image based on the defect information; a display control section 43 for displaying at least a part of the defective image on a display 35; a determination result reception section 44 for receiving an input of a defect type determination result by an operator with respect to the defective image displayed on the display 35; and a teacher data generation section 45 for generating teacher data by labelling the determination result to the segmented image. Thus, the teacher data including an image where an unnecessary area other than the defective area is reduced can be easily generated.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for generating training data. [Background technology]

[0002] In an inspection device that captures an image of an object to detect defects, a defect image of a predetermined size including the defect area is output when a defect is detected. It is conceivable that the type of defect indicated by such a defect image is classified using a trained model (classifier). In this case, an operator determines the defect type for a prepared defect image (i.e., by annotation), thereby generating training data in which the defect image is labeled with one defect type, and the trained model is generated by performing training using multiple training data.

[0003] Patent Document 1 discloses a method for generating learning data by obtaining an area containing a defect in an image through input by an operator, correcting the area by widening the outer edge so that the number of pixels contained within the area increases by a predetermined amount, and associating the corrected area with the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-87078 Summary of the Invention [Problem to be solved by the invention]

[0005] However, defect images output from an inspection device have a fixed size and contain many unnecessary areas other than the defect area. Therefore, even if training is performed using training data including such defect images, it is difficult to obtain a highly accurate trained model. As in the method of Patent Document 1, it is possible to obtain an image in which unnecessary areas have been reduced by having an operator input areas including defects, but this increases the workload. Therefore, there is a need for a method that can easily generate training data including images in which unnecessary areas other than the defect area have been reduced.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has as its object to easily generate training data including an image in which unnecessary areas other than defective areas have been reduced. [Means for solving the problem]

[0007] The invention of claim 1 is a teacher data generation device for generating teacher data, comprising: an image receiving unit that receives, from an inspection device that images an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; a cut-out image generating unit that cuts out an area including the detection area from the defect image as a cut-out image based on the defect information; and a display unit that displays at least one image of the defect image. the detection region a judgment result receiving unit that receives an input of a judgment result of a defect type by an operator for the defect image displayed on the display; and a teacher data generating unit that generates teacher data by labeling the judgment result to the extracted image. Each position of the object belongs to one of a plurality of area types, the defect information includes area type information indicating the area type to which the detection area belongs, the cut-out image generating unit stores an extension amount set for each area type, and includes in the cut-out image an area obtained by extending the detection area by the extension amount specified using the area type information. do.

[0009] Claim 2 The invention described in claim 1 2. The teaching data generation device according to claim 1, wherein the object is a printed circuit board, and the plurality of area types include at least a plated area and a solder resist area.

[0010] Claim 3 The invention described in Generate training data A teacher data generation device, an image receiving unit that receives, from an inspection device that images an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; a cut-out image generating unit that cuts out an area including the detection area from the defect image as a cut-out image based on the defect information; a display control unit that displays at least the detection area of ​​the defect image on a display; a judgment result receiving unit that receives an input of a judgment result of a defect type by an operator for the defect image displayed on the display; and a training data generating unit that generates training data by labeling the judgment result to the cut-out image, One of a plurality of inspection sensitivities is set for each position of the object, the defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area, the cut-out image generation unit stores the extension amount set for each inspection sensitivity, and includes in the cut-out image an area obtained by extending the detection area by the extension amount specified using the inspection sensitivity information.

[0011] Claim 4 The invention described in claims 1 to 3 wherein each position of the object belongs to one of a plurality of area types, the defect information includes area type information indicating the area type to which the detection area belongs, and the teacher data generation unit labels the extracted image with the area type to which the detection area belongs in addition to the judgment result.

[0012] Claim 5 The invention described in is a teacher data generation method for generating teacher data, comprising the steps of: a) receiving, from an inspection device that images an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) cutting out an area including the detection area from the defect image as a cut-out image based on the defect information; and c) displaying at least one of the defect images on a display. the detection region d) a step of receiving an input of a defect type determination result by an operator for the defect image displayed on the display; and e) a step of labeling the determination result to the extracted image to generate training data. each position of the object belongs to one of a plurality of area types, the defect information includes area type information indicating the area type to which the detection area belongs, and in the step b), an extension amount set for each area type is prepared, and an area obtained by extending the detection area by the extension amount specified using the area type information is included in the extracted image. do.

[0014] Claim 6 The invention described in claim 5 2. The teaching data generation method according to claim 1, wherein the object is a printed circuit board, and the plurality of area types include at least a plated area and a solder resist area.

[0015] Claim 7 The invention described in Generate training dataA teacher data generation method, a) receiving, from an inspection device that images an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as an extracted image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result for the defect image displayed on the display by an operator; and e) generating training data by labeling the determination result to the extracted image, One of a plurality of inspection sensitivities is set for each position of the object, the defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area, and in step b), an extension amount set for each inspection sensitivity is prepared, and an area obtained by extending the detection area by the extension amount specified using the inspection sensitivity information is included in the extracted image.

[0016] Claim 8 The invention described in claim 5 Or 7 a training data generation method according to any one of the above, wherein each position of the object belongs to one of a plurality of area types, the defect information includes area type information indicating the area type to which the detection area belongs, and in step e), in addition to the judgment result, the area type to which the detection area belongs is labeled on the cut-out image, and a trained model for defect classification of the one area type is generated using a plurality of training data labeled with the one area type.

[0017] Claim 9 The invention described in is a program for causing a computer to generate training data, and execution of the program by a computer includes the steps of: a) receiving, from an inspection device that picks up an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) cutting out an area including the detection area from the defect image as a cut-out image based on the defect information; and c) displaying at least one of the defect images on a display. the detection region a step of displaying the defect type; a step of receiving an input of a defect type determination result by an operator for the defect image displayed on the display; and a step of labeling the determination result to the extracted image to generate training data. each position of the object belongs to one of a plurality of area types, the defect information includes area type information indicating the area type to which the detection area belongs, and in the step b), an extension amount set for each area type is prepared, and an area obtained by extending the detection area by the extension amount specified using the area type information is included in the extracted image. do. The invention described in claim 10 is a program for causing a computer to generate training data, wherein execution of the program by a computer causes the computer to execute the following steps: a) receiving, from an inspection device that images an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) cutting out an area including the detection area from the defect image as a cropped image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result for the defect image displayed on the display by an operator; and e) labeling the determination result on the cropped image to generate training data, wherein one inspection sensitivity out of a plurality of inspection sensitivities is set for each position of the object, the defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area, and in step b), an extension amount set for each inspection sensitivity is prepared, and an area obtained by expanding the detection area by the extension amount specified using the inspection sensitivity information is included in the cropped image. [Effects of the Invention]

[0018] According to the present invention, training data including an image in which unnecessary areas other than defective areas have been reduced can be easily generated. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an inspection system. [Figure 2] FIG. 1 illustrates the configuration of a computer. [Figure 3] FIG. 1 is a diagram illustrating a configuration of a teacher data generation device. [Figure 4] FIG. 10 is a diagram showing a processing flow for generating training data. [Figure 5] FIG. [Figure 6] FIG. [Figure 7] FIG. [Figure 8A] FIG. 10 is a diagram showing a defect image. [Figure 8B] FIG. 10 is a diagram showing a defect image. [Figure 9] FIG. 10 is a diagram showing a defect image. [Figure 10] FIG. 10 is a diagram showing the vicinity of a defect area. [Figure 11] FIG. 10 is a diagram showing a defect image. [Figure 12] FIG. 10 is a diagram showing the vicinity of a defect area. [Figure 13] FIG. 10 is a diagram showing the vicinity of a defect area. [Figure 14] FIG. 10 is a diagram showing the vicinity of a defect area. [Figure 15] FIG. [Figure 16] FIG. 2 is an enlarged view of a portion of a printed circuit board. [Figure 17] FIG. 10 is a diagram illustrating another example of a classifier. DETAILED DESCRIPTION OF THE INVENTION

[0020] (First embodiment) FIG. 1 is a diagram showing the configuration of an inspection system 1 according to a first embodiment of the present invention. The inspection system 1 inspects a printed circuit board, which is an object of inspection. The inspection system 1 includes an inspection device 2 and a computer 3. In FIG. 1, the functional configuration realized by the computer 3 is enclosed in a dashed rectangle. The inspection device 2 includes an imaging unit, a movement mechanism, and a defect detection unit, all of which are not shown. The imaging unit captures an image of the printed circuit board. The movement mechanism moves the printed circuit board relative to the imaging unit. The defect detection unit detects defects from the image output from the imaging unit. When a defect is detected by the defect detection unit, a defect image of a predetermined size (also called a defect block size) that includes the defect area is output to the computer 3.

[0021] FIG. 2 shows the configuration of the computer 3. The computer 3 has a typical computer system configuration including a CPU 31, a ROM 32, a RAM 33, a fixed disk 34, a display 35, an input unit 36, a reading device 37, a communication unit 38, a GPU 39, and a bus 30. The CPU 31 performs various arithmetic operations. The GPU 39 performs various arithmetic operations related to image processing. The ROM 32 stores basic programs. The RAM 33 stores various information. The fixed disk 34 stores information. The display 35 displays various information such as images. The input unit 36 ​​includes a keyboard 36a and a mouse 36b for receiving input from an operator. The reading device 37 reads information from a computer-readable recording medium 81 such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The communication unit 38 transmits and receives signals to and from other components of the inspection system 1 and external devices. The bus 30 is a signal circuit that connects the CPU 31, the GPU 39, the ROM 32, the RAM 33, the fixed disk 34, the display 35, the input unit 36, the reading device 37, and the communication unit 38.

[0022] In computer 3, program 811 is read in advance from recording medium 81 via reader 37 and stored on fixed disk 34. Program 811 may also be stored on fixed disk 34 via a network. CPU 31 and GPU 39 execute arithmetic processing using RAM 33 and fixed disk 34 in accordance with program 811. CPU 31 and GPU 39 function as a computing unit in computer 3. Other components functioning as a computing unit may be employed in addition to CPU 31 and GPU 39.

[0023] In the inspection system 1, the computer 3 executes arithmetic processing and the like in accordance with the program 811, thereby realizing the functional configuration enclosed by the dashed line in Fig. 1. That is, the CPU 31, GPU 39, ROM 32, RAM 33, fixed disk 34 of the computer 3 and their peripheral configuration realize the teacher data generation device 4, learning unit 51, and classifier 52. All or part of these functions may be realized by dedicated electrical circuits. Furthermore, these functions may be realized by multiple computers.

[0024] The classifier 52 is a trained model that classifies defects indicated in defect images input from the inspection device 2 into true defects or false defects. The learning unit 51 generates the trained model (classifier 52) by performing learning using a plurality of training data described below. The training data generation device 4 generates training data used in the learning unit 51.

[0025] FIG. 3 is a diagram showing the configuration of the teacher data generation device 4. The teacher data generation device 4 includes an image receiving unit 41, a cropped image generation unit 42, a display control unit 43, a judgment result receiving unit 44, and a teacher data generation unit 45. The image receiving unit 41 is connected to the inspection device 2 and receives input of defect images and the like from the inspection device 2. The cropped image generation unit 42 crops cropped images (described below) from the defect images. The display control unit 43 is connected to the display 35 and displays the defect images and the like on the display 35. The judgment result receiving unit 44 is connected to the input unit 36 ​​and receives input from an operator via the input unit 36. The teacher data generation unit 45 labels the cropped images to generate teacher data.

[0026] 4 is a diagram showing a processing flow for generating teacher data by the teacher data generating device 4. First, the image receiving unit 41 receives a defect image and defect information (to be described later) from the inspection device 2 (step S11).

[0027] Here, an example of a process for detecting defects by the inspection device 2 will be described. FIG. 5 is a diagram showing a multi-tone captured image of a portion of a printed circuit board. For example, the captured image is a color image. The captured image may also be a grayscale image. Multiple types of regions are provided on the main surface of the printed circuit board. Specifically, there are provided plated regions plated with metal such as copper, solder resist regions (hereinafter also referred to as "SR regions") on the surface of which solder resist is provided, silk regions where characters or symbols are printed on the solder resist, through-hole regions where through-holes are opened, etc. Furthermore, the SR regions can be distinguished into a first SR region where the layer below the solder resist is copper foil and a second SR region where the layer below the solder resist is the base material of the printed circuit board, and the two regions have different colors. As described above, each position on the main surface of the printed circuit board belongs to one of multiple region types including the plated region, the first SR region, the second SR region, the silk region, etc.

[0028] 5 includes region 61 indicating a plated region and region 62 indicating an SR region, and region 62 includes region 621 indicating a first SR region and region 622 indicating a second SR region. In the following description, regions 61, 62, 621, and 622 will be similarly referred to as "plated region 61," "SR region 62," "first SR region 621," and "second SR region 622." For other types of regions on the printed circuit board, the corresponding regions in the captured image will be referred to by the same names.

[0029] The defect detection unit of the inspection device 2, for example, refers to design data (CAM data, etc.) to identify the area type to which each position in the captured image belongs. A normal range of gradation values ​​for each color component is set for each area type. The gradation values ​​at each position in the captured image are compared with the normal range for each color component, and a group of pixels outside the normal range is detected as a defective area. In the example of FIG. 5, there is an area 71 in the first SR area 621 that is darker than its surroundings, and this area 71 is the defective area 71 recognized by an operator who observes the captured image. In FIG. 6, the outer edge of an area 72 (hereinafter referred to as the "detection area 72") detected as a defect by the inspection device 2 is indicated by a dashed line. In the example of FIG. 6, the detection area 72 substantially coincides with the defective area 71.

[0030] When a defect is detected in the inspection device 2, an image of a predetermined size including the detection area 72 is acquired as a defect image. In addition, defect information indicating the position and shape (including size) of the detection area 72 in the defect image is acquired. Note that various well-known methods (inspection logic, etc.) may be used to detect defects, and different methods may be used for different area types.

[0031] When starting to generate training data, the inspection device 2 has previously acquired a plurality of defect images from a large number of captured images of a plurality of printed circuit boards. The plurality of defect images have the same size (defect block size) and represent areas of the same size on the printed circuit board. Each defect image is associated with defect information indicating the position and shape of the detection area 72. In step S11 of FIG. 4, the plurality of defect images and the defect information for the plurality of defect images are accepted by the image accepting unit 41. For example, the defect information for the plurality of defect images is included in one list in a state where it is associated with each of the plurality of defect images.

[0032] Next, the cropped image generating unit 42 crops out an area including the detection area 72 from each defect image as a cropped image (step S12). In the example of FIG. 6, as shown in FIG. 7, the area of ​​a circumscribing rectangle 73 (shown by a dashed line in FIG. 7) of the detection area 72 is cropped out as a cropped image. Each side of the circumscribing rectangle 73 is parallel to the up-down direction (column direction) or left-right direction (row direction) of the defect image. Depending on the design of the cropped image generating unit 42, the area of ​​the smallest circumscribing rectangle (each side may be inclined with respect to the up-down direction or left-right direction) that can be set for the detection area 72 may be cropped out as a cropped image.

[0033] The display control unit 43 then displays the defect image on the display 35 (step S13). The image displayed on the display 35 may be either the entire defect image or a portion thereof. For example, a cropped image of the defect image may be displayed, or an image obtained by expanding the cropped image by a predetermined number of pixels, i.e., an image including the detection area 72 and its surroundings, may be displayed. In this manner, the display control unit 43 displays at least a portion of the defect image on the display 35. In one example, thumbnails of a plurality of defect images are displayed in an array in a window on the display 35, and when the operator selects the thumbnail of one defect image via the input unit 36, at least a portion of the defect image (hereinafter referred to as the "selected defect image") is displayed on the display 35. The defect image to be displayed on the display 35 may be selected by various well-known methods.

[0034] The judgment result receiving unit 44 receives an input of the judgment result of whether the selected defect image displayed on the display 35 is a true defect or a false defect by the operator (step S14). In one example, a button indicating "true defect" and a button indicating "false defect" are provided in a window on the display 35 along with the selected defect image. When the operator checks the selected defect image and selects one of the buttons via the input unit 36, a judgment result indicating whether the defect shown in the selected defect image is a true defect or a false defect is input. The input of the judgment result is received by the judgment result receiving unit 44. The operator may input the judgment result using various well-known methods.

[0035] The teacher data generation unit 45 generates teacher data by labeling the cut-out image with the judgment result (step S15). The teacher data is data including a cut-out image obtained from a defect image and the judgment result of the operator on the defect image. The teacher data may also include a defect image. In practice, the operator inputs judgment results for multiple defect images, and multiple pieces of teacher data are generated. This completes the teacher data generation process, and multiple pieces of teacher data (learning data sets) are obtained.

[0036] When multiple sets of training data are generated, the learning unit 51 in FIG. 1 performs machine learning so that the output of the classifier in response to the input of cut-out images in the multiple sets of training data is approximately the same as the determination result (true defect or false defect) indicated by the multiple sets of training data, thereby generating a classifier. The classifier is a trained model that classifies defects indicated by images into true defects or false defects, and in generating the classifier, the parameter values ​​included in the classifier and the structure of the classifier are determined. The machine learning is performed, for example, by deep learning using a neural network. The machine learning may also be performed by well-known methods other than deep learning. The classifier (actually, the parameter values ​​and information indicating the structure of the classifier) ​​is transferred to and introduced into the classifier 52.

[0037] When the inspection system 1 inspects a printed circuit board, the inspection device 2 acquires multiple captured images showing multiple positions on the printed circuit board and inspects the multiple captured images for defects. When a defect is detected, an image of a predetermined size including the detection area 72 is output as a defect image to the classifier 52. The classifier 52 classifies the defect shown in the defect image as a true defect or a false defect, and stores the classification result or outputs it externally. In a preferred inspection system 1, the cropped image generation unit 42 of the computer 3 crops the area of ​​the circumscribing rectangle 73 of the detection area 72 in the defect image as a cropped image, similar to when generating the training data, and inputs the cropped image to the classifier 52. This enables the classifier 52 to more accurately classify the defect shown in the defect image as a true defect or a false defect.

[0038] Here, a comparative example of processing for generating training data will be described. FIGS. 8A and 8B are diagrams showing defect images, which include a defect area 71. In FIGS. 8A and 8B, the defect area 71 is indicated by parallel diagonal lines spaced closer together than the SR area 62, and the defect area 71 substantially coincides with the detection area acquired by the inspection device 2. In the example of FIG. 8B, a group of multiple defect partial areas 711 is detected as one defect area 71.

[0039] In the processing of the first comparative example, the entire defect image is used as the image of training data. As shown in Figures 8A and 8B, a defect image usually shows an area much larger than defect area 71. Therefore, in the first comparative example, the features of unnecessary areas other than defect area 71 are also used for learning by learning unit 51. In other words, the image of training data does not efficiently show the features of defect area 71 (detection area), and therefore the classification accuracy of the classifier is reduced.

[0040] In the processing of the second comparative example, a region of a certain size including the defect area 71 is cut out from the defect image and used as an image of training data. In FIGS. 8A and 8B, the cut-out area A1 cut out from the defect image in the second comparative example is indicated by a two-dot chain line. The size of the cut-out area A1 is determined empirically, for example. In the second comparative example, unnecessary areas other than the defect area 71 are reduced in the image of training data (image of the cut-out area A1) compared to the first comparative example, but still remain to some extent. Furthermore, if the defect area 71 is relatively large, as in the example of FIG. 8B, it extends beyond the cut-out area A1, and the image of training data will not show all the features of the defect area 71 (detection area).

[0041] Furthermore, in the first and second comparative examples, a large amount of training data is required to improve the classification accuracy of the classifier, which increases the number of times an operator inputs the judgment results for the defect images (the number of times annotates). Even if a large amount of training data is used, it may not be possible to generate a highly accurate classifier.

[0042] 3, a defect image of a predetermined size including a defect detection area 72 and defect information indicating the position and shape of the detection area 72 in the defect image are input from the inspection device 2 and accepted by the image accepting unit 41. The extracted image generating unit 42 extracts an area including the detection area 72 from the defect image as an extracted image based on the defect information. The display control unit 43 also displays at least a portion of the defect image on the display 35, and the judgment result accepting unit 44 accepts an input by the operator of a judgment result as to whether the displayed defect image is a true defect or a false defect. The teacher data generating unit 45 then labels the extracted image with the judgment result, thereby generating teacher data.

[0043] This makes it possible to easily generate training data including an image (cut-out image) in which unnecessary areas other than the defect area 71 have been reduced. Furthermore, the image shows almost all of the characteristics of the defect area 71. In this way, by using training data that efficiently shows the characteristics of the defect area 71, it is possible to generate a highly accurate trained model (classifier 52) with a small amount of training data, and it is also possible to reduce the number of annotations by an operator. In FIGS. 8A and 8B, a circumscribing rectangle 73 of the detection area cut out as the cut-out image is shown by a dashed line.

[0044] (Second embodiment) Next, a training data generation process according to a second embodiment of the present invention will be described. FIG. 9 is a diagram showing a defect image, illustrating an example in which a defect area 71 exists on a plating area 61. In FIG. 9, the defect area 71 is indicated by parallel diagonal lines spaced closer together than the SR area 62 (similar to FIGS. 11 to 14 described below). FIG. 10 is an enlarged view showing the vicinity of the defect area 71, with a detection area 72 acquired by the inspection device 2 shaded in black (similar to FIGS. 12 to 14 described below). When the defect area 71 exists on the plating area 61, the outer edge of the detection area 72 tends to roughly coincide with the outer edge of the defect area 71 recognized by an operator who observes the defect image. In FIG. 10, the entire detection area 72 roughly overlaps the entire defect area 71.

[0045] FIG. 11 is a diagram showing a defect image, illustrating an example in which a defect area 71 exists on an SR region 62. FIG. 12 is an enlarged view showing the vicinity of the defect area 71, in which a collection of multiple detected partial regions 721 is detected as a single detection region 72. In FIGS. 11 and 12, the outer edge of the defect area 71 is shown with a dashed line, indicating that the outer edge (i.e., the boundary with the surrounding area) of the defect area 71 is unclear (the same applies to FIG. 14 described below). When the defect area 71 exists on the SR region 62, the outer edge of the detection region 72 tends to be smaller than the outer edge of the defect area 71 as recognized by an operator observing the defect image. In FIG. 12, the detection region 72 overlaps only a portion of the defect area 71. Note that different defect detection methods may be used for the plating region 61 and the SR region 62.

[0046] As described above, each position on the main surface of the printed circuit board belongs to one of a plurality of area types, and the area type to which each position in the captured image belongs is also identified in the inspection device 2. In the inspection device 2 of this processing example, when a defect is detected, area type information indicating the area type to which the detection area 72 belongs is generated and included in the defect information.

[0047] In generating training data by the training data generation device 4, the image receiving unit 41 receives a defect image and defect information from the inspection device 2 (FIG. 4: step S11). As described above, the defect information includes area type information in addition to the position and shape of the detection area 72 in the defect image. The cropped image generation unit 42 crops out an area obtained by extending the circumscribing rectangle of the detection area 72 in the vertical and horizontal directions according to the area type to which the detection area 72 belongs (step S12).

[0048] Specifically, the number of pixels (a natural number; the same applies below) by which the circumscribing rectangle is expanded vertically and horizontally is set as the expansion amount for each of a plurality of region types, and the expansion amounts are stored and prepared in the cropped image generating unit 42. As described above, the outer edge of the detection region 72 on the plating region 61 tends to roughly coincide with the outer edge of the defect region 71, so the expansion amount for the plating region 61 is set to a relatively small number of pixels (e.g., 0 to 5 pixels). Therefore, in the example of FIG. 10 where the detection region 72 belongs to the plating region 61, as shown in FIG. 13, the region of the circumscribing rectangle 73 of the detection region 72 (shown by a dashed line in FIG. 13) or a region obtained by slightly expanding this region is cropped as the cropped image. The cropped image includes almost the entire defect region 71.

[0049] Furthermore, since the outer edge of the detection region 72 on the SR region 62 tends to be smaller than the outer edge of the defective region 71, the expansion amount for the SR region 62 is set to a relatively large number of pixels (for example, 10 to 20 pixels). Therefore, in the example of FIG. 12 where the detection region 72 belongs to the SR region 62, a region 74 obtained by expanding the circumscribing rectangle 73 of the detection region 72 by the expansion amount is cut out as a cut-out image, as shown in FIG. 14. In FIG. 14, the circumscribing rectangle 73 and region 74 are indicated by dashed lines. The cut-out image (i.e., region 74) includes almost the entire defective region 71. Note that region 74 obtained by expanding the circumscribing rectangle 73 of the detection region 72 by the expansion amount is the same as the circumscribing rectangle of the region obtained by expanding the detection region 72 by the expansion amount.

[0050] In the teacher data generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs the judgment result of whether the selected defect image is a true defect or a false defect, and the input is accepted (step S14). Then, the judgment result is labeled on the extracted image, thereby generating teacher data (step S15). Thereafter, similar to the above-mentioned processing example, a classifier 52 is generated using the plurality of teacher data.

[0051] In the inspection of printed circuit boards in the inspection system 1, when a defect is detected by the inspection device 2, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image, and a classification result is obtained by the classifier 52. In a preferred inspection system 1, similar to the generation of training data, an area obtained by extending the circumscribing rectangle 73 of the detection area 72 in all directions is extracted as a cropped image according to the area type to which the detection area 72 belongs, and the cropped image is input to the classifier 52. This enables the classifier 52 to more accurately classify the defect indicated in the defect image as either a true defect or a false defect.

[0052] As described above, in this processing example, the defect information includes area type information indicating the area type to which the detection area 72 belongs. The cropped image generation unit 42 stores an expansion amount set for each area type, and includes in the cropped image an area obtained by expanding the detection area 72 by the expansion amount specified using the area type information. This makes it possible to obtain a preferable cropped image that shows approximately the entire defect area 71, and to generate a highly accurate trained model (classifier 52). Since the plated area and SR area occupy the majority of a printed circuit board, from the perspective of obtaining a preferable cropped image, it is preferable that the multiple area types include at least the plated area and the solder resist area.

[0053] (Third embodiment) Next, a teacher data generation process according to a third embodiment of the present invention will be described. Fig. 15 is a diagram showing an entire printed circuit board 9. The printed circuit board 9 during manufacture includes a waste board area 92, which is a portion that will be removed in the final product. In Fig. 15, the waste board area 92 is indicated by parallel diagonal lines. Fig. 16 is an enlarged view of a portion B1 surrounded by a dashed line in the printed circuit board 9 of Fig. 15, with the waste board area 92 surrounded by a thick dashed line. As shown in Fig. 16, the printed circuit board 9 has an area 91 (the area surrounded by a thin dashed line in Fig. 16) where small plated areas are densely arranged and where thin wiring patterns are provided.

[0054] Since defects present in region 91 have a significant impact on the operation of printed circuit board 9, in the inspection device 2 in this processing example, stricter inspection sensitivity is set for region 91 than for other regions. Hereinafter, region 91 will be referred to as the "first sensitivity setting region 91." On the other hand, since defects present in the previously described waste board region 92 have almost no impact on the operation of printed circuit board 9, a looser inspection sensitivity is set for the waste board region 92 than for other regions. Hereinafter, the waste board region 92 will be referred to as the "second sensitivity setting region 92." Furthermore, an intermediate inspection sensitivity is set for region 93 other than the first sensitivity setting region 91 and the second sensitivity setting region 92. Hereinafter, region 93 will be referred to as the "third sensitivity setting region 93."

[0055] In this way, one of a plurality of inspection sensitivities is set at each position on the printed circuit board 9. In the above example in which the inspection device 2 compares the gradation value at each position on the captured image with the normal range, the inspection sensitivity is the width of the normal range. A narrower normal range is set in the first sensitivity setting region 91 than in other regions, and a wider normal range is set in the second sensitivity setting region 92 than in other regions. As described above, various methods may be used for defect detection, and the method for setting the inspection sensitivity is changed as appropriate depending on the defect detection method.

[0056] In the inspection device 2, for example, by referring to design data (CAM data, etc.), it is determined whether each position in the captured image belongs to the first sensitivity setting region 91, the second sensitivity setting region 92, or the third sensitivity setting region 93, and a normal range to be compared is obtained. Then, the gradation value of the position is compared with the normal range, and a set of pixels outside the normal range is obtained as the detection region 72. In the inspection device 2, inspection sensitivity information is included in the above-mentioned defect information. The inspection sensitivity information is information that can identify the inspection sensitivity used when detecting the detection region 72, and the inspection sensitivity information in this processing example is information that indicates any one of the first sensitivity setting region 91, the second sensitivity setting region 92, and the third sensitivity setting region 93.

[0057] In generating training data by the training data generation device 4, the image receiving unit 41 receives the defect image and defect information from the inspection device 2 (FIG. 4: step S11). As described above, the defect information includes inspection sensitivity information in addition to the position and shape of the detection area 72 in the defect image. The cropped image generation unit 42 crops out an area obtained by extending the circumscribing rectangle 73 of the detection area 72 in the vertical and horizontal directions in accordance with the inspection sensitivity used when detecting the detection area 72 (step S12).

[0058] Specifically, the extension amount is set in advance for each of a plurality of inspection sensitivities as the number of pixels by which the circumscribing rectangle 73 is extended vertically and horizontally, and is stored and prepared in the extracted image generating unit 42. With the loosest inspection sensitivity (i.e., when the detection area 72 is located in the second sensitivity setting area 92), the outer edge of the detection area 72 tends to be smaller than the outer edge of the defective area 71, so the extension amount is set to a relatively large number of pixels α (e.g., 8 to 12 pixels). With the strictest inspection sensitivity (i.e., when the detection area 72 is located in the first sensitivity setting area 91), the outer edge of the detection area 72 tends to roughly coincide with the outer edge of the defective area 71, so the extension amount is set to a relatively small number of pixels β (e.g., 0 to 3 pixels). At intermediate inspection sensitivity (i.e., when the detection area 72 is located in the third sensitivity setting area 93), the outer edge of the detection area 72 tends to be slightly smaller than the outer edge of the defective area 71, so the amount of expansion is set to a pixel number γ (e.g., 4 to 7 pixels) between the number of pixels when the inspection sensitivity is the loosest and the number of pixels when the inspection sensitivity is the strictest.

[0059] As described above, the expansion amount is largest when the inspection sensitivity is lowest, and is smallest when the inspection sensitivity is highest. In other words, α>γ>β is satisfied. As a result, the area obtained by expanding the circumscribing rectangle 73 of the detection area 72 by the expansion amount, i.e., the extracted image, includes almost the entire defect area 71.

[0060] In the teacher data generating device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs the judgment result of whether the selected defect image is a true defect or a false defect, and the input is accepted (step S14). Then, the judgment result is labeled on the extracted image, thereby generating teacher data (step S15). Thereafter, similar to the above-mentioned processing example, a classifier 52 is generated using the plurality of teacher data.

[0061] In the inspection of printed circuit boards in the inspection system 1, when a defect is detected by the inspection device 2, an image of a predetermined size including the detection area 72 is output to the computer 3 as a defect image, and a classification result is obtained by the classifier 52. In a preferred inspection system 1, similar to the generation of training data, an area obtained by extending the circumscribing rectangle 73 of the detection area 72 in all directions is extracted as a cropped image in accordance with the inspection sensitivity used when detecting the detection area 72, and the cropped image is input to the classifier 52. This enables the classifier 52 to more accurately classify the defect indicated in the defect image as either a true defect or a false defect.

[0062] As described above, in this processing example, one of multiple inspection sensitivities is set for each position on the printed circuit board, and inspection sensitivity information indicating the inspection sensitivity used when detecting the detection area 72 is included in the defect information. The cropped image generation unit 42 stores an extension amount set for each inspection sensitivity, and includes in the cropped image an area obtained by extending the detection area 72 by the extension amount specified using the inspection sensitivity information. This makes it possible to obtain a desirable cropped image that shows approximately the entire defect area 71, and to generate a highly accurate trained model (classifier 52).

[0063] (Fourth embodiment) Next, a teacher data generation process according to a fourth embodiment of the present invention will be described. As described above, each position on the main surface of a printed circuit board belongs to one of a plurality of area types. When a defect is detected, the inspection device 2 generates area type information indicating the area type to which the detection area 72 belongs, and includes the generated area type information in the defect information.

[0064] 4 in this processing example are the same as those in the first embodiment. In step S12, as in the second embodiment, a region obtained by extending the circumscribing rectangle 73 of the detection region 72 in the vertical and horizontal directions may be extracted as a cropped image depending on the region type to which the detection region 72 belongs. Also, as in the third embodiment, a region obtained by extending the circumscribing rectangle 73 of the detection region 72 in the vertical and horizontal directions may be extracted as a cropped image depending on the inspection sensitivity used when detecting the detection region 72.

[0065] The teacher data generation unit 45 generates teacher data by labeling the extracted image with the region type to which the detection region 72 belongs, in addition to the operator's determination result of the selected defect image as a true defect or a false defect (step S15). In the teacher data generation process, multiple teacher data for each region type are generated from multiple defect images. Here, it is assumed that multiple teacher data for the plating region and multiple teacher data for the SR region are generated.

[0066] The learning unit 51 performs machine learning using multiple pieces of training data for the plating region to generate a trained model 521 for the plating region shown in Fig. 17. Also, by performing machine learning using multiple pieces of training data for the SR region, a trained model 522 for the SR region is generated.

[0067] When inspection system 1 inspects a printed circuit board, inspection device 2 acquires multiple captured images showing multiple positions on the printed circuit board and inspects the multiple captured images for defects. When a defect is detected, a defect image of a predetermined size including detection area 72 is output to classifier 52 along with defect information including area type information. If detection area 72 of the defect image belongs to the plating area, classifier 52 classifies the defect indicated by the defect image into a true defect or a false defect using trained model 521 for plating area. If detection area 72 of the defect image belongs to the SR area, classifier 52 classifies the defect indicated by the defect image into a true defect or a false defect using trained model 522 for SR area.

[0068] As described above, in this processing example, the defect information includes area type information indicating the area type to which the detection area 72 belongs. In the training data generation unit 45, in addition to the operator's determination of whether the defect is true or false, the extracted image is labeled with the area type to which the detection area 72 belongs. This allows the learning unit 51 to generate a trained model for defect classification of the area type using multiple training data labeled with a single area type. In this way, generating a trained model for each area type can further improve classification accuracy.

[0069] The teacher data generating device 4 and the teacher data generating method can be modified in various ways.

[0070] The defect information input from the inspection device 2 to the teacher data generation device 4 may be information indicating the range of the detection area 72 in the defect image, and is not limited to information indicating the position and shape of the detection area 72. For example, the defect information may be information indicating the range of a circumscribing rectangle of the detection area 72 in the defect image (i.e., the range in each of the vertical and horizontal directions).

[0071] The region of the defect image cut out as the cut-out image is determined based on the defect information and may be any region that includes the detection region 72, but is preferably a region that approximately circumscribes the detection region 72. The region that approximately circumscribes the detection region 72 includes not only a region that circumscribes the detection region 72, but also a region that circumscribes a region obtained by expanding the detection region 72 by the aforementioned expansion amount.

[0072] 4, the operator inputs the judgment result of whether the defect image is a true defect or a false defect, but the judgment result of a defect type other than a true defect or a false defect (for example, foreign matter adhesion, film peeling, etc.) may also be input. That is, the judgment result receiving unit 44 receives the input of the operator's judgment result of the defect type (including the judgment result of whether the defect is a true defect or a false defect) for the defect image displayed on the display 35.

[0073] In the second embodiment, when the detection area 72 includes portions that belong to two or more different area types, the expansion amount for any of the two or more area types may be used in expanding the detection area 72. From the viewpoint of obtaining a preferable cropped image that shows approximately the entire defect area 71, it is preferable to use the largest expansion amount among the expansion amounts for the two or more area types.

[0074] In the third embodiment, when the detection area 72 includes portions detected with two or more different inspection sensitivities, the expansion amount for any of the two or more inspection sensitivities may be used to expand the detection area 72. From the viewpoint of obtaining a preferable cropped image that shows approximately the entire defect area 71, it is preferable to use the largest expansion amount among the expansion amounts for the two or more inspection sensitivities.

[0075] The object to be inspected by the inspection device 2 may be a substrate such as a semiconductor substrate or a glass substrate in addition to a printed circuit board. Furthermore, defects in objects other than substrates, such as machine parts, may be detected by the inspection device 2. The training data generation device 4 can easily generate preferable training data used to generate trained models for defect classification of various objects.

[0076] The configurations in the above-described embodiment and each modification may be combined as appropriate as long as they are not mutually contradictory. [Explanation of symbols]

[0077] 2. Inspection equipment 3. Computer 4. Teacher data generation device 9 Printed Circuit Board 35 Display 41 Image Reception Department 42 Cutout image generation section 43 Display control unit 44 Judgment result reception department 45 Teacher data generation unit 72 detection area 521,522 trained models 811 Program Steps S11 to S15

Claims

1. A teacher data generation device that generates teacher data, an image receiving unit that receives, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; a cut-out image generating unit that cuts out an area including the detection area from the defect image as a cut-out image based on the defect information; a display control unit that displays at least the detection area of ​​the defect image on a display; a judgment result receiving unit that receives an input of a judgment result of a defect type by an operator for the defect image displayed on the display; a training data generation unit that generates training data by labeling the cut-out image with the determination result; Equipped with each position of the object belongs to one of a plurality of area types; the defect information includes area type information indicating an area type to which the detection area belongs, A teacher data generation device characterized in that the cut-out image generation unit stores an extension amount set for each region type, and includes in the cut-out image an area obtained by extending the detection region by an extension amount specified using the region type information.

2. 2. The teacher data generation device according to claim 1, A teacher data generating device characterized in that the object is a printed circuit board, and the multiple area types include at least a plated area and a solder resist area.

3. A teacher data generation device for generating teacher data, an image receiving unit that receives, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; a cut-out image generating unit that cuts out an area including the detection area from the defect image as a cut-out image based on the defect information; a display control unit that displays at least the detection area of ​​the defect image on a display; a judgment result receiving unit that receives an input of a judgment result of a defect type by an operator for the defect image displayed on the display; a training data generation unit that generates training data by labeling the cut-out image with the determination result; Equipped with One of a plurality of inspection sensitivities is set for each position of the object; the defect information includes inspection sensitivity information indicating an inspection sensitivity used when detecting the detection area, A teacher data generation device characterized in that the cut-out image generation unit stores an extension amount set for each inspection sensitivity, and includes in the cut-out image an area obtained by extending the detection area by an extension amount determined using the inspection sensitivity information.

4. 4. The teacher data generation device according to claim 1, each position of the object belongs to one of a plurality of area types; the defect information includes area type information indicating an area type to which the detection area belongs, The teacher data generating device is characterized in that the teacher data generating unit labels the extracted image with the region type to which the detection region belongs in addition to the judgment result.

5. A teacher data generation method for generating teacher data, a) receiving, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as an extracted image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result by an operator for the defect image displayed on the display; e) labeling the extracted image with the determination result to generate training data; Equipped with each position of the object belongs to one of a plurality of area types; the defect information includes area type information indicating an area type to which the detection area belongs, A teacher data generation method characterized in that in step b), an extension amount set for each region type is prepared, and a region obtained by extending the detection region by the extension amount specified using the region type information is included in the cut-out image.

6. The teacher data generation method according to claim 5, A training data generating method characterized in that the object is a printed circuit board, and the plurality of area types include at least a plated area and a solder resist area.

7. A teacher data generation method for generating teacher data, comprising: a) receiving, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as an extracted image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result by an operator for the defect image displayed on the display; e) labeling the extracted image with the determination result to generate training data; Equipped with One of a plurality of inspection sensitivities is set for each position of the object; the defect information includes inspection sensitivity information indicating an inspection sensitivity used when detecting the detection area, A teacher data generation method characterized in that in step b), an extension amount set for each inspection sensitivity is prepared, and an area obtained by extending the detection area by the extension amount specified using the inspection sensitivity information is included in the extracted image.

8. 8. The teacher data generation method according to claim 5, further comprising: each position of the object belongs to one of a plurality of area types; the defect information includes area type information indicating an area type to which the detection area belongs, In the step e), in addition to the determination result, the region type to which the detected region belongs is labeled to the extracted image; A training data generation method, characterized in that a trained model for defect classification of a single area type is generated using multiple training data labeled with the single area type.

9. A program for causing a computer to generate teacher data, wherein execution of the program by the computer causes the computer to: a) receiving, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as an extracted image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result by an operator for the defect image displayed on the display; e) labeling the extracted image with the determination result to generate training data; Execute each position of the object belongs to one of a plurality of area types; the defect information includes area type information indicating an area type to which the detection area belongs, A program characterized in that in step b), an extension amount set for each area type is prepared, and an area obtained by extending the detection area by the extension amount specified using the area type information is included in the cut-out image.

10. A program for causing a computer to generate teacher data, wherein execution of the program by the computer causes the computer to: a) receiving, from an inspection device that captures an image of an object and detects defects, a defect image of a predetermined size including a defect detection area and defect information indicating the range of the detection area in the defect image; b) extracting an area including the detection area from the defect image as an extracted image based on the defect information; c) displaying at least the detection area of ​​the defect image on a display; d) receiving an input of a defect type determination result by an operator for the defect image displayed on the display; e) labeling the extracted image with the determination result to generate training data; Execute One of a plurality of inspection sensitivities is set for each position of the object; the defect information includes inspection sensitivity information indicating an inspection sensitivity used when detecting the detection area, A program characterized in that in step b), an extension amount set for each inspection sensitivity is prepared, and an area obtained by extending the detection area by the extension amount specified using the inspection sensitivity information is included in the extracted image.

Images