Teacher data generation device, teacher data generation method, and recording medium having a program recorded

Abstract

The teacher data generation device (4) comprises: an image receiving unit (41) that receives a defect image of a predetermined size, including the detection area of ​​the defect, and defect information indicating the extent of the detection area in the defect image, from an inspection device (2) that detects defects by imaging an object; a cropping image generation unit (42) that, based on the defect information, crops the area including the detection area from the defect image as a cropped image; a display control unit (43) that displays at least a portion of the defect image on a display (35); a judgment result receiving unit (44) that receives an input from an operator regarding the judgment result of the defect category of the defect image displayed on the display (35); and a teacher data generation unit (45) that annotates the cropped image with the judgment result to generate teacher data. Thus, it is possible to easily generate teacher data that includes an image in which unwanted areas other than the defect area have been reduced.

Description

Technical Field

[0001] This invention relates to a technology for generating teacher data. Background Technology

[0002] In an inspection apparatus that detects defects by imaging an object, when a defect is detected, a defect image of a specified size, including the defect area, is output. Consider classifying the defect category represented by such a defect image using a learned model (classifier). In this case, the operator generates teacher data by determining (i.e., annotating) the defect category for a pre-prepared defect image, thus labeling the defect image with a defect category. The learned model is then generated by learning from multiple teacher data sets.

[0003] It should be noted that Japanese Patent Application Publication No. 2019-87078 (Document 1) discloses a method in which a region including defects in an image is obtained by an operator input, and the outer edge of the extended region is corrected in such a way that the number of pixels contained in the region exceeds a predetermined number. Learning data is generated by associating the corrected region with the image.

[0004] However, the defect images output from the inspection device are of a fixed size and contain a large number of unwanted areas besides the defect area. Therefore, even when learning using teacher data that includes such defect images, it is difficult to obtain a high-precision learned model. As with the method in Reference 1, it is also possible to obtain an image with reduced unwanted areas by having the operator input the region including the defect, but this increases the operational burden. Therefore, there is a need for a method that can easily generate teacher data that includes images with reduced unwanted areas besides the defect area. Summary of the Invention

[0005] The present invention relates to a teacher data generation apparatus for generating teacher data, the purpose of which is to easily generate teacher data containing images in which unwanted areas, excluding defective areas, have been reduced.

[0006] The teacher data generation apparatus of the present invention comprises: an image receiving unit that receives a defect image of a predetermined size, including a detection area of ​​the defect, and defect information indicating the extent of the detection area in the defect image from an inspection device for imaging an object to detect defects; a cropped image generation unit that, based on the defect information, crops a region including the detection area from the defect image to obtain a cropped image; a display control unit that displays at least a portion of the defect image on a display screen; a determination result receiving unit that receives an input of a determination result of a defect category performed by an operator on the defect image displayed on the display screen; and a teacher data generation unit that annotates the cropped image with the determination result to generate teacher data.

[0007] According to the present invention, it is possible to easily generate teacher data that includes images containing reduced unwanted areas other than defective areas.

[0008] Preferably, each location of the object belongs to one of multiple region categories, the defect information includes region category information indicating the region category to which the detection area belongs, the cropping image generation unit stores the expansion amount set for each region category, and includes the region after the detection area is expanded using the expansion amount determined by the region category information in the cropping image.

[0009] Preferably, the object is a printed circuit board, and the plurality of area categories include at least plated areas and solder mask areas.

[0010] Preferably, an inspection sensitivity is set for each position of the object, one of a plurality of inspection sensitivities is set, the defect information includes inspection sensitivity information indicating the inspection sensitivity used when detecting in the detection area, the cropping image generation unit stores an expansion amount set for each inspection sensitivity, and includes the area after the detection area is expanded using the expansion amount determined by the inspection sensitivity information in the cropping image.

[0011] Preferably, each location of the object belongs to one of multiple region categories, and the defect information includes region category information indicating the region category to which the detected area belongs. In addition to the determination result, the teacher data generation unit marks the region category to which the detected area belongs on the cropped image.

[0012] The present invention also relates to a method for generating teacher data. The method includes: a) receiving, from an inspection apparatus that detects defects by imaging an object, a defect image of a predetermined size including a detection area of ​​the defect and defect information representing the extent of the detection area in the defect image; b) cropping, based on the defect information, a region including the detection area from the defect image as a cropped image; c) displaying at least a portion of the defect image on a display; d) receiving input of a determination result of a defect category performed by an operator on the defect image displayed on the display; and e) labeling the cropped image with the determination result to generate teacher data.

[0013] Preferably, each location of the object belongs to one of multiple region categories, and the defect information includes region category information indicating the region category to which the detection area belongs. In step b), an expansion amount is prepared for each region category, and the region after the detection area is expanded using the expansion amount determined by the region category information is included in the cropped image.

[0014] Preferably, the object is a printed circuit board, and the plurality of area categories include at least plated areas and solder mask areas.

[0015] Preferably, an inspection sensitivity is set for each location of the object, one of a plurality of inspection sensitivities, and the defect information includes inspection sensitivity information indicating the inspection sensitivity used during inspection in the detection area. In step b), an expansion amount is prepared for each inspection sensitivity, and the area after the detection area is expanded using the expansion amount determined by the inspection sensitivity information is included in the cropped image.

[0016] Preferably, each location of the object belongs to one of multiple region categories, and the defect information includes region category information indicating the region category to which the detection area belongs. In step e), in addition to the determination result, the region category to which the detection area belongs is marked on the cut image. Using multiple teacher data marked with a region category, a learned model for defect classification of the aforementioned region category is generated.

[0017] The present invention also relates to a recording medium containing a program for generating teacher data by a computer. The program of the present invention is executed by a computer, which performs the following steps: a) receiving a defect image of a predetermined size, including a detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image from an inspection apparatus that images an object to detect defects; b) cropping a region including the detection area from the defect image as a cropped image based on the defect information; c) displaying at least a portion of the defect image on a display; d) receiving input of a determination result of a defect category performed by an operator on the defect image displayed on the display; and e) annotating the cropped image with the determination result to generate teacher data.

[0018] The above-mentioned objects and other objects, features, aspects and advantages will be illustrated by the following detailed description of the invention with reference to the accompanying drawings. Attached Figure Description

[0019] Figure 1 This is a diagram showing the structure of the inspection system.

[0020] Figure 2This is a diagram showing the structure of a computer.

[0021] Figure 3 This is a diagram showing the structure of the teacher data generation device.

[0022] Figure 4 This is a diagram illustrating the process of generating teacher data.

[0023] Figure 5 This is a diagram showing the image being imaged.

[0024] Figure 6 This is a diagram showing the image being imaged.

[0025] Figure 7 This is a diagram showing the image being imaged.

[0026] Figure 8A This is a diagram showing an image of a defect.

[0027] Figure 8B This is a diagram showing an image of a defect.

[0028] Figure 9 This is a diagram showing an image of a defect.

[0029] Figure 10 This is a diagram showing the vicinity of the defect area.

[0030] Figure 11 This is a diagram showing an image of a defect.

[0031] Figure 12 This is a diagram showing the vicinity of the defect area.

[0032] Figure 13 This is a diagram showing the vicinity of the defect area.

[0033] Figure 14 This is a diagram showing the vicinity of the defect area.

[0034] Figure 15 This is a diagram showing a printed circuit board.

[0035] Figure 16 This is a magnified view showing a portion of a printed circuit board.

[0036] Figure 17 This is a diagram showing another example of a classifier.

[0037] Attached image caption:

[0038] 2: Inspection device

[0039] 3: Computer

[0040] 4: Teacher Data Generation Device

[0041] 9: Printed Circuit Board

[0042] 35: Monitor

[0043] 41: Image receiving unit

[0044] 42: Image cropping generation unit

[0045] 43: Display Control Unit

[0046] 44: Judgment Result Acceptance Department

[0047] 45: Teacher Data Generation Department

[0048] 72: Detection area

[0049] 521, 522: Learned models

[0050] 811: Program

[0051] S11~S15: Steps Detailed Implementation

[0052] (First Implementation)

[0053] Figure 1 This diagram illustrates the structure of an inspection system 1 according to a first embodiment of the present invention. The inspection system 1 inspects a printed circuit board, which is the object of the inspection. The inspection system 1 includes an inspection device 2 and a computer 3. Figure 1 In the diagram, the functional structure implemented by computer 3 is enclosed by a dashed rectangle. Inspection device 2 has an imaging unit, a moving mechanism, and a defect detection unit (not shown). The imaging unit images the printed circuit board. The moving mechanism moves the printed circuit board relative to the imaging unit. The defect detection unit detects defects based on the image output from the imaging unit. When a defect is detected in the defect detection unit, a defect image of a specified size (also called the defect block size) including the defect area is output to computer 3.

[0054] Figure 2This diagram illustrates the structure of computer 3. Computer 3 has the general structure of a computer system, including CPU 31, ROM 32, RAM 33, fixed disk 34, display 35, input unit 36, reading device 37, communication unit 38, GPU 39, and bus 30. CPU 31 performs various arithmetic operations. GPU 39 performs various arithmetic operations related to image processing. ROM 32 stores basic programs. RAM 33 stores various information. Fixed disk 34 stores information. Display 35 displays various information such as images. Input unit 36 ​​includes a keyboard 36a and a mouse 36b for accepting input from the operator. Reading device 37 reads information from computer-readable recording media 81 such as optical discs, magnetic disks, magneto-optical disks, and memory cards. Communication unit 38 sends and receives signals with other components of inspection system 1 and external devices. Bus 30 is a signal circuit connecting CPU 31, GPU 39, ROM 32, RAM 33, fixed disk 34, display 35, input unit 36, reading device 37, and communication unit 38.

[0055] In computer 3, program 811 is read in advance from recording medium 81, which is the program product, via reading device 37 and stored in fixed disk 34. Program 811 can be stored in fixed disk 34 via network. CPU 31 and GPU 39 perform arithmetic processing simultaneously using RAM 33 and fixed disk 34 according to program 811. CPU 31 and GPU 39 function as arithmetic units in computer 3. In addition to CPU 31 and GPU 39, other structures that function as arithmetic units can also be used.

[0056] In inspection system 1, computer 3 performs calculations and other processes according to program 811, thereby achieving... Figure 1 The functional structures enclosed by dashed lines in the diagram represent the CPU 31, GPU 39, ROM 32, RAM 33, fixed disk 34, and their peripheral structures in computer 3, which implement the teacher data generation device 4, learning unit 51, and classifier 52. All or part of these functions can also be implemented using dedicated circuitry. Alternatively, these functions can be implemented using multiple computers.

[0057] Classifier 52 is a learned model that classifies defects shown in the defect image input from inspection device 2 as true defects or false defects. Learning unit 51 learns by using multiple teacher data, described later, to generate the learned model (classifier 52). Teacher data generation device 4 generates teacher data used in learning unit 51.

[0058] Figure 3This diagram illustrates the structure of the teacher data generation device 4. The teacher data generation device 4 includes an image receiving unit 41, a cropped image generation unit 42, a display control unit 43, a judgment result receiving unit 44, and a teacher data generation unit 45. The image receiving unit 41 is connected to the inspection device 2 and receives input such as defect images from the inspection device 2. The cropped image generation unit 42 crops a cropped image (described later) from the defect image. The display control unit 43 is connected to the display 35 and displays the defect image, etc., on the display 35. The judgment result receiving unit 44 is connected to the input unit 36 ​​and receives input from the operator via the input unit 36. The teacher data generation unit 45 annotates the cropped image to generate teacher data.

[0059] Figure 4 This diagram illustrates the processing flow of teacher data generation device 4 in generating teacher data. First, in image receiving unit 41, defect images and defect information (described later) are received from inspection device 2 (step S11).

[0060] Here, an example of how the inspection device 2 handles defects will be explained. Figure 5 This diagram illustrates a multi-grayscale image obtained by imaging a portion of a printed circuit board. For example, the image may be a color image. The image may also be a grayscale image. Various regions are provided on the main surface of the printed circuit board. Specifically, these include plating areas where metals such as copper are plated, solder resist areas (hereinafter also referred to as "SR areas") where solder resist is applied to the surface, silkscreen areas where text or markings are printed on the solder resist, and through-hole areas, etc. Furthermore, SR areas can be distinguished into first SR areas where the solder resist is underlaid with copper foil and second SR areas where the solder resist is underlaid with the substrate of the printed circuit board, and the two have different colors. As described above, each location on the main surface of the printed circuit board belongs to any one of several region categories, including plating areas, first SR areas, second SR areas, silkscreen areas, etc.

[0061] Figure 5 Examples include region 61, representing a plating area, and region 62, representing an SR area. Region 62 includes region 621, representing a first SR area, and region 622, representing a second SR area. In the following description, regions 61, 62, 621, and 622 will also be referred to as "plating area 61," "SR area 62," "first SR area 621," and "second SR area 622." The same names are used to refer to the corresponding areas of the image in other categories of areas on the printed circuit board.

[0062] In the defect detection unit of inspection device 2, the region category of each location in the image is determined, for example, by referring to design data (CAM data, etc.). Furthermore, a normal range for the grayscale values ​​of each color component is set for each region category. In the image, the grayscale values ​​of each location are compared with the normal range for each color component, and the set of pixels outside the normal range is detected as a defect region. Figure 5 In the example, there is a darker region 71 on the first SR region 621 than the surrounding area, and this region 71 is a defect region 71 identified by the operator observing the imaging image. Figure 6 In the diagram, the outer edge of the area 72 (hereinafter referred to as "detection area 72") detected as defective by inspection device 2 is indicated by a dashed line. Figure 6 In the example, the detection area 72 is basically the same as the defect area 71.

[0063] In inspection device 2, if a defect is detected, an image of a predetermined size, including the detection area 72, is acquired as a defect image. Additionally, defect information indicating the position and shape (including size) of the detection area 72 in the defect image is acquired. It should be noted that various known methods (inspection logic, etc.) can be used for defect detection, and different methods can be used depending on the category of each area.

[0064] When generating teacher data, multiple defect images are pre-acquired from multiple imaging images of multiple printed circuit boards using inspection device 2. These multiple defect images are of the same size (defect block size), representing areas of the same size within the printed circuit board. Furthermore, defect information indicating the location and shape of the detection area 72 is associated with each defect image. Figure 4 In step S11, the image receiving unit 41 receives multiple defect images and defect information of the multiple defect images. For example, the defect information of the multiple defect images is included in a list with states associated with each of the multiple defect images.

[0065] Next, in the cropping image generation unit 42, the region including the detection region 72 is cropped from each defect image to create a cropped image (step S12). Figure 6 In the example, such as Figure 7 As shown, the outer rectangle 73 of the detection area 72 ( Figure 7 The area indicated by the dashed line is cut as a cut image. The sides of the circumscribed rectangle 73 are parallel to the vertical (column) or horizontal (row) direction of the defect image. Depending on the design of the cut image generation unit 42, an area that can be cut as a cut image can also be the smallest circumscribed rectangle set relative to the detection area 72 (each side can also be tilted relative to the vertical and horizontal directions).

[0066] Additionally, the display control unit 43 displays a defect image on the display 35 (step S13). The image displayed on the display 35 can be all or part of the defect image. For example, a cropped image of the defect image can be displayed, or an image enlarged by a predetermined number of pixels, i.e., an image including the detection area 72 and its surroundings, can be displayed. Thus, the display control unit 43 displays at least a portion of the defect image on the display 35. In one example, multiple thumbnails of defect images are displayed in a window arrangement on the display 35, and the operator selects a thumbnail of a defect image via the input unit 36, thereby displaying at least a portion of that defect image (hereinafter referred to as "selected defect image") on the display 35. The selection of the defect image displayed on the display 35 can be performed by various known methods.

[0067] In the determination result receiving unit 44, the operator's determination result regarding whether the defect shown in the selected defect image displayed on the display 35 is a true defect or a false defect is received (step S14). In one example, a window on the display 35 includes a button indicating "true defect" and a button indicating "false defect" along with the selected defect image. The operator confirms the selected defect image and selects any button via the input unit 36 ​​to input a determination result indicating whether the defect shown in the selected defect image is a true defect or a false defect. This determination result input is received by the determination result receiving unit 44. The operator can input the determination result using various known methods.

[0068] In the teacher data generation unit 45, teacher data is generated by annotating the cut image with judgment results (step S15). Teacher data includes the cut image obtained from the defective image and the judgment results made by the operator regarding that defective image. Teacher data may include the defective image. In practice, the operator inputs judgment results for multiple defective images, generating multiple sets of teacher data. Thus, the teacher data generation process is completed, resulting in multiple sets of teacher data (learning datasets).

[0069] When generating multiple teacher data sets, Figure 1 In the learning unit 51, machine learning is performed to generate a classifier in a manner that makes the output of the classifier to the input of the cropped image from multiple teacher data approximately the same as the judgment results (true defects or false defects) shown by the multiple teacher data. The classifier is a learned model that classifies defects shown in an image as true defects or false defects. In the generation of the classifier, the values ​​of the parameters included in the classifier and the structure of the classifier are determined. Machine learning is performed, for example, by using deep learning through neural networks. This machine learning can also be performed by known methods other than deep learning. The classifier (actually the values ​​of the parameters, information representing the structure of the classifier) ​​is transmitted and imported into classifier 52.

[0070] When the inspection system 1 inspects the printed circuit board, the inspection device 2 acquires multiple imaging images representing multiple locations on the printed circuit board and checks for defects in the multiple imaging images. When a defect is detected, an image of a predetermined size, including the detection area 72, is output as a defect image to the classifier 52. In the classifier 52, the defect shown in the defect image is classified as a true defect or a false defect, and the classification result is stored or output externally. In the preferred inspection system 1, in the cropping image generation unit 42 of the computer 3, similar to when generating teacher data, the region of the bounding rectangle 73 of the detection area 72 in the defect image is cropped as a cropped image, and this cropped image is input to the classifier 52. Thus, the classifier 52 can classify whether the defect shown in the defect image is a true defect or a false defect with higher accuracy.

[0071] Here, the processing of the comparative examples for generating teacher data is explained. Figure 8A and Figure 8B An image of the defect is shown, including the defect region 71. Figure 8A and Figure 8B In this process, parallel diagonal lines with narrower intervals than those in the SR region 62 are applied to the defect region 71, and the defect region 71 is substantially consistent with the detection region obtained by the inspection device 2. Furthermore, in Figure 8B In the example, a collection of multiple defective partial regions 711 is detected as a single defective region 71.

[0072] In the processing of the first comparative example, the entire defect image was used as the image of the teacher data. For example... Figure 8A and Figure 8B As shown, the defect image typically represents a region much larger than the defect region 71. Therefore, in the first comparative example, features of unwanted regions other than the defect region 71 are also used for learning in the learning unit 51. In other words, since the image of the teacher data cannot efficiently represent the features of the defect region 71 (detection region), the classification accuracy in the classifier becomes low.

[0073] In the processing of the second comparative example, a region of a certain size, including the defect region 71, was cropped from the defect image and used as the image for the teacher data. Figure 8A and Figure 8B In the second comparative example, the cut-out region A1, which was cut from the defect image, is represented by a double-dotted line. The size of the cut-out region A1 is determined empirically, for example. In the second comparative example, in the image of the teacher's data (the image of the cut-out region A1), the unwanted areas other than the defect region 71 are reduced compared to the first comparative example, but are still included to some extent. Additionally, as... Figure 8BAs in the example, when the defect region 71 is relatively large, the image of the teacher data cannot represent all the features of the defect region 71 (detection region) because it is exposed from the shear region A1.

[0074] Furthermore, in both the first and second comparative examples, a large amount of teacher data is required to improve the classification accuracy of the classifier, which increases the number of times the operator needs to input (annotate) the judgment results for the defective images. Even with a large amount of teacher data, it is sometimes impossible to generate a high-accuracy classifier.

[0075] In contrast, Figure 3 In the teacher data generation device 4, a defect image of a predetermined size, including the detection area 72 of the defect, and defect information indicating the position and shape of the detection area 72 in the defect image are input from the inspection device 2 and received in the image receiving unit 41. In the cropped image generation unit 42, based on the defect information, the area including the detection area 72 is cropped from the defect image to form a cropped image. In addition, at least a portion of the defect image is displayed on the display 35 by the display control unit 43, and the judgment result receiving unit 44 receives the input of the judgment result of the operator on the displayed defect image, indicating whether it is a true defect or a false defect. Furthermore, the teacher data generation unit 45 marks the judgment result on the cropped image to generate teacher data.

[0076] Therefore, it is possible to easily generate teacher data including an image (cropped image) containing reduced unwanted regions other than defect region 71. Furthermore, almost all the features of defect region 71 are shown in this image. Thus, by using teacher data that efficiently represents the features of defect region 71, a high-precision learned model (classifier 52) can be generated with less teacher data, and the number of operator annotations can also be reduced. It should be noted that in... Figure 8A and Figure 8B In the image, the bounding rectangle 73 of the detection region used for cropping is represented by a dashed line.

[0077] (Second Implementation)

[0078] Next, the teacher data generation process according to the second embodiment of the present invention will be described. Figure 9 This is a diagram showing a defect image, illustrating an example where a defective region 71 exists on the plating area 61. Figure 9 In the diagram, the defect area 71 is marked with parallel diagonal lines that are narrower than those in the SR area 62 (described later). Figures 11-14 (The same applies). Figure 10 This is a magnified view showing the vicinity of the defect area 71, with the inspection area 72 obtained by the inspection device 2 blacked out (described later). Figures 12-14(The same applies in the middle). When a defect region 71 exists on the plating area 61, the outer edge of the detection area 72 tends to coincide substantially with the outer edge of the defect region 71 identified by the operator observing the defect image. Figure 10 In the middle, the entire detection area 72 roughly overlaps with the entire defect area 71.

[0079] Figure 11 This is a diagram showing a defect image, illustrating an example where defect region 71 exists on SR region 62. Figure 12 This is a magnified view showing the vicinity of defect region 71, where a collection of multiple detection areas 721 are detected as a single detection area 72. Figure 11 and Figure 12 In the diagram, the outer edge of the defect region 71 is represented by a dashed line, indicating that the outer edge of the defect region 71 (i.e., the boundary with its surroundings) is unclear (as described later). Figure 14 (Similarly) In the case where a defect region 71 exists in the SR region 62, the outer edge of the detection region 72 tends to be smaller than the outer edge of the defect region 71 identified by the operator observing the defect image. Figure 12 In this context, the detection area 72 only partially overlaps with the defect area 71. It should be noted that the defect detection methods can differ between the plating area 61 and the SR area 62.

[0080] As described above, each location on the main surface of the printed circuit board belongs to one of multiple region categories. In the inspection apparatus 2, the region category to which each location in the imaging image belongs is also determined. In the inspection apparatus 2 of this processing example, when a defect is detected, region category information indicating the region category to which the detection area 72 belongs is generated and included in the defect information.

[0081] When teacher data is generated by teacher data generation device 4, defect images and defect information are received from inspection device 2 in image receiving unit 41. Figure 4 (Step S11). As described above, in addition to the position and shape of the detection area 72 in the defect image, the defect information also includes region category information. In the cropping image generation unit 42, according to the region category to which the detection area 72 belongs, the region after expanding the bounding rectangle of the detection area 72 upwards, downwards, leftwards, and rightwards is cropped as a cropping image (step S12).

[0082] Specifically, the number of pixels (a natural number; the same applies below) that expands the circumscribed rectangle in all directions is used as the expansion amount. An expansion amount is preset for each of the multiple region categories and stored in the cropping image generation unit 42 for preparation. As described above, since the outer edge of the detection region 72 on the plating region 61 tends to be substantially consistent with the outer edge of the defect region 71, the expansion amount for the plating region 61 is a relatively small number of pixels (e.g., 0 to 5 pixels). Therefore, when the detection region 72 belongs to the plating region 61... Figure 10 In the example, such as Figure 13 As shown, the outer rectangle 73 of the detection area 72 is... Figure 13 The area (represented by dashed lines) or a region that has been slightly expanded from the defect is cropped as a cropped image. This cropped image contains almost the entire defect area 71.

[0083] Furthermore, since the outer edge of the detection area 72 on the SR region 62 tends to be smaller than the outer edge of the defect region 71, the expansion amount for the SR region 62 is a relatively large number of pixels (e.g., 10-20 pixels). Therefore, in the detection area 72 belonging to the SR region 62 Figure 12 In the example, such as Figure 14 As shown, the region 74, after the outer rectangle 73 of the detection region 72 is expanded by an expansion amount, is used as the cropped image and is cropped. Figure 14 In the image, the circumscribed rectangle 73 and region 74 are represented by dashed lines. This clipped image (i.e., region 74) encompasses almost the entire defect region 71. It should be noted that region 74, after expanding the circumscribed rectangle 73 of the detection region 72 by an expansion amount, is the same as the circumscribed rectangle of the region after expanding the detection region 72 by an expansion amount.

[0084] In the teacher data generation device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs and accepts the judgment result of whether the selected defect image is a true defect or a false defect (step S14). Then, teacher data is generated by labeling the cropped image with the judgment result (step S15). Afterwards, in the same way as the above processing example, multiple teacher data generation classifiers 52 are used.

[0085] In the inspection of printed circuit boards in inspection system 1, when a defect is detected in inspection device 2, an image of a specified size, including the detection area 72, is output as a defect image to computer 3, and the classification result of classifier 52 is obtained. In the preferred inspection system 1, similar to the generation of teacher data, based on the region category to which the detection area 72 belongs, the area after expanding the bounding rectangle 73 of the detection area 72 upwards, downwards, leftwards, and rightwards is cropped as a cropped image, and this cropped image is input into classifier 52. Therefore, classifier 52 can classify the defect shown in the defect image as a real defect or a false defect with higher accuracy.

[0086] As described above, in this processing example, the region category information representing the region category to which the detection region 72 belongs is included in the defect information. In the cut image generation unit 42, the expansion amount set for each region category is stored, and the region after the detection region 72 is expanded using the expansion amount determined using the region category information is included in the cut image. Thus, a preferred cut image representing almost the entire defect region 71 can be obtained, and a high-precision learned model (classifier 52) can be generated. In a printed circuit board, since plating areas and solder resist areas constitute the majority, from the viewpoint of obtaining a preferred cut image, the aforementioned multiple region categories preferably include at least plating areas and solder resist areas.

[0087] (Third Implementation)

[0088] Next, the teacher data generation process of the third embodiment of the present invention will be described. Figure 15 This is a diagram showing the entire printed circuit board 9. The printed circuit board 9 during manufacturing includes the discarded circuit board area 92, which is the portion removed in the final product. Figure 15 In the middle, the abandoned circuit board area 92 is marked with parallel diagonal lines. Figure 16 It is shown in magnification Figure 15 The diagram shows the portion B1 of the printed circuit board 9 enclosed by dashed lines, and the abandoned circuit board area 92 enclosed by thick dashed lines. (See diagram 92.) Figure 16 As shown, in the printed circuit board 9, there are areas 91 (where small plating areas are densely arranged or where fine wiring patterns are provided) Figure 16 (The area surrounded by a thin dotted line).

[0089] Since the defects present in region 91 have a significant impact on the operation of the printed circuit board 9, the inspection device 2 in this processing example sets a stricter inspection sensitivity for region 91 than for other regions. Hereinafter, region 91 will be referred to as the "first sensitivity setting region 91". On the other hand, since the defects present in the aforementioned discarded circuit board region 92 have almost no impact on the operation of the printed circuit board 9, a more lenient inspection sensitivity is set for the discarded circuit board region 92 compared to other regions. Hereinafter, the discarded circuit board region 92 will be referred to as the "second sensitivity setting region 92". Furthermore, an intermediate inspection sensitivity is set in region 93, excluding the first sensitivity setting region 91 and the second sensitivity setting region 92. Hereinafter, region 93 will be referred to as the "third sensitivity setting region 93".

[0090] Thus, any one of multiple inspection sensitivities is set at various locations on the printed circuit board 9. In the above example, where the grayscale values ​​at various locations of the image are compared with the normal range in the inspection device 2, the inspection sensitivity is the width of the normal range. A normal range narrower than other areas is set in the first sensitivity setting area 91, and a normal range wider than other areas is set in the second sensitivity setting area 92. As described above, various methods can be used in defect detection, and the method for setting the inspection sensitivity can be appropriately modified according to the defect detection method.

[0091] In inspection device 2, by referring to, for example, design data (CAM data, etc.), it is determined which of the first sensitivity setting region 91, the second sensitivity setting region 92, and the third sensitivity setting region 93 each position in the image belongs to, and a normal range for comparison is obtained. Then, the gray value of that position is compared with the normal range, and the set of pixels outside the normal range is obtained as the detection region 72. In inspection device 2, inspection sensitivity information is included in the defect information described above. Inspection sensitivity information is information that can determine the inspection sensitivity used when detecting the detection region 72. In this processing example, the inspection sensitivity information shows information indicating any one of the first sensitivity setting region 91, the second sensitivity setting region 92, and the third sensitivity setting region 93.

[0092] When teacher data is generated by teacher data generation device 4, defect images and defect information are received from inspection device 2 in image receiving unit 41. Figure 4 (Step S11). As described above, in addition to the position and shape of the detection area 72 in the defect image, the defect information also includes inspection sensitivity information. In the cut image generation unit 42, based on the inspection sensitivity used when detecting the detection area 72, the area after expanding the circumscribed rectangle 73 of the detection area 72 in all directions is cut as a cut image (step S12).

[0093] Specifically, the number of pixels that expand the bounding rectangle 73 vertically, horizontally, and vertically is used as the expansion amount. For each of the multiple inspection sensitivities, a pre-set expansion amount is stored in the cropping image generation unit 42 and prepared. At the most lenient inspection sensitivity (i.e., when the detection area 72 is located in the second sensitivity setting area 92), the outer edge of the detection area 72 tends to be smaller than the outer edge of the defect area 71, so the expansion amount is set to a relatively large number of pixels α (e.g., 8 to 12 pixels). At the most stringent inspection sensitivity (i.e., when the detection area 72 is located in the first sensitivity setting area 91), the outer edge of the detection area 72 tends to be substantially consistent with the outer edge of the defect area 71, so the expansion amount is set to a relatively small number of pixels β (e.g., 0 to 3 pixels). At an intermediate inspection sensitivity (i.e., when the detection area 72 is located in the third sensitivity setting area 93), the outer edge of the detection area 72 tends to be slightly smaller than the outer edge of the defect area 71, so the expansion amount is set to a number of pixels γ (e.g., 4 to 7 pixels) between the number of pixels in the most lenient inspection sensitivity and the number of pixels in the most stringent inspection sensitivity.

[0094] As mentioned above, the expansion amount is largest when the inspection sensitivity is most lenient, and the expansion amount is smallest when the inspection sensitivity is most stringent. In other words, α > γ > β must be satisfied. As a result, the area after expanding the outer rectangle 73 of the detection area 72 by the expansion amount, i.e., the cropped image, contains almost the entire defect area 71.

[0095] In the teacher data generation device 4, after the selected defect image is displayed on the display 35 (step S13), the operator inputs and accepts the judgment result of whether the selected defect image is a true defect or a false defect (step S14). Then, teacher data is generated by labeling the cropped image with the judgment result (step S15). Afterwards, in the same way as the above processing example, multiple teacher data generation classifiers 52 are used.

[0096] In the inspection of printed circuit boards in inspection system 1, when a defect is detected in inspection device 2, an image of a specified size, including the detection area 72, is output as a defect image to computer 3, and the classification result of classifier 52 is obtained. In the preferred inspection system 1, similarly to when generating teacher data, based on the inspection sensitivity used when detecting the detection area 72, the area after expanding the outer rectangle 73 of the detection area 72 in all directions is cropped as a cropped image, and this cropped image is input to classifier 52. Therefore, classifier 52 can classify the defect shown in the defect image as a real defect or a false defect with higher accuracy.

[0097] As described above, in this processing example, one of a plurality of inspection sensitivities is set for each location on the printed circuit board, and the defect information includes inspection sensitivity information representing the inspection sensitivity used when detecting the detection area 72. In the cut-out image generation unit 42, the expansion amount set for each inspection sensitivity is stored, and the area after the detection area 72 is expanded using the expansion amount determined using the inspection sensitivity information is included in the cut-out image. Thus, a preferred cut-out image representing almost the entire defect area 71 can be obtained, and a high-precision learned model (classifier 52) can be generated.

[0098] (Fourth Implementation)

[0099] Next, the teacher data generation process according to the fourth embodiment of the present invention will be described. As described above, each location on the main surface of the printed circuit board belongs to one of a plurality of area categories. In the inspection device 2, when a defect is detected, area category information indicating the area category to which the detection area 72 belongs is generated and included in the defect information.

[0100] In this processing example, Figure 4 Steps S11 to S14 are the same as in the first embodiment described above. In step S12, similar to the second embodiment, the area after expanding the circumscribed rectangle 73 of the detection area 72 upwards, downwards, leftwards, and rightwards can be used as a cropped image based on the region category to which the detection area 72 belongs. Furthermore, similar to the third embodiment, the area after expanding the circumscribed rectangle 73 of the detection area 72 upwards, downwards, leftwards, and rightwards can be used as a cropped image based on the inspection sensitivity used during the detection of the detection area 72.

[0101] In the teacher data generation unit 45, in addition to the operator's determination of whether a defect is real or false based on the selected defect image, the region category to which the detection region 72 belongs is labeled on the cropped image, thereby generating teacher data (step S15). In the teacher data generation process, multiple teacher data sets for each region category are generated based on multiple defect images. Here, multiple teacher data sets for the plating region and multiple teacher data sets for the SR region are generated.

[0102] In Learning Department 51, machine learning was used to generate data from multiple teachers' data on the plating area. Figure 17 The plating area shown uses the learned model 521. Additionally, a learned model 522 for the SR area is generated by performing machine learning using multiple teacher data for the SR area.

[0103] When the inspection system 1 inspects the printed circuit board, the inspection device 2 acquires multiple imaging images representing multiple locations on the printed circuit board and checks for defects in the multiple imaging images. When a defect is detected, a defect image of a specified size, including the detection area 72, along with defect information including area type information, is output to the classifier 52. In the classifier 52, if the detection area 72 of the defect image belongs to the plating area, the defect shown in the defect image is classified as a true defect or a false defect using the plating area and the learned model 521. If the detection area 72 of the defect image belongs to the SR area, the defect shown in the defect image is classified as a true defect or a false defect using the SR area and the learned model 522.

[0104] As described above, in this processing example, the region category information indicating the region category to which the detection region 72 belongs is included in the defect information. In the teacher data generation unit 45, in addition to the operator's determination of whether it is a true or false defect, the region category to which the detection region 72 belongs is labeled on the cropped image. Therefore, in the learning unit 51, a learned model for defect classification of that region category can be generated using multiple teacher data labeled with a region category. In this way, by generating a learned model for each region category, the classification accuracy can be further improved.

[0105] Various modifications can be made to the aforementioned teacher data generation device 4 and teacher data generation method.

[0106] The defect information input from the inspection device 2 to the teacher data generation device 4 only needs to represent the range of the detection area 72 in the defect image, and is not limited to representing the position and shape of the detection area 72. For example, the defect information can represent the range of the bounding rectangle of the detection area 72 in the defect image (i.e., the range in the vertical and horizontal directions respectively).

[0107] The region of the defect image that is cut out as a cut-out image only needs to be determined based on the defect information and include the detection region 72, and is preferably a region that is approximately outside the detection region 72. The region that is approximately outside the detection region 72 includes not only the region that is outside the detection region 72, but also the region that is outside the region after the detection region 72 has been expanded by the aforementioned expansion amount.

[0108] In the above embodiments, Figure 4 In step S14, the operator inputs the determination result of whether the defect image is a true defect or a false defect. However, the operator can also input the determination result of defect categories other than true and false defects (such as foreign matter adhesion, film peeling, etc.). That is, the determination result receiving unit 44 accepts the operator's input of the determination result of the defect category (including the determination result of whether it is a true defect or a false defect) of the defect image displayed on the display 35.

[0109] In the second embodiment, when the detection area 72 includes portions belonging to two or more different region categories, the expansion amount for any one of the two or more region categories can be used in the expansion of the detection area 72. From the viewpoint of obtaining a preferred cropped image representing almost the entire defect area 71, it is preferable to use the largest expansion amount among the two or more region categories.

[0110] In the third embodiment, when the detection area 72 includes regions detected with two or more different inspection sensitivities, the expansion amount for any one of the two or more inspection sensitivities can be used in the expansion of the detection area 72. From the viewpoint of obtaining a preferred cutaway image representing almost the entire defect area 71, it is preferable to use the largest expansion amount among the two or more inspection sensitivities.

[0111] The objects inspected in inspection device 2 can be circuit boards such as semiconductor circuit boards or glass circuit boards, in addition to printed circuit boards. Furthermore, inspection device 2 can also detect defects in objects other than circuit boards, such as mechanical parts. Teacher data generation device 4 can easily generate preferred teacher data, which is used to generate a learned model for defect classification of various objects.

[0112] The structures of the above-described embodiments and their variations can be appropriately combined as long as they do not contradict each other.

[0113] While the invention has been described in detail, the foregoing description is exemplary and not restrictive. Therefore, various modifications or variations are possible without departing from the scope of the invention.

Claims

1. A teacher data generation device for generating teacher data, characterized in that, have: The image receiving unit receives a defect image of a predetermined size, including the detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image from an inspection device that images an object to detect defects. The cropped image generation unit, based on the defect information, crops the region including the detection region from the defect image to obtain a cropped image; The display control unit displays at least a portion of the defect image on the display screen; The determination result receiving unit accepts input of the operator's determination result regarding the defect category of the defect image displayed on the display screen; as well as The teacher data generation unit annotates the cropped image with the judgment result to generate teacher data. The locations of the objects belong to one of multiple region categories. The defect information includes region category information indicating the region category to which the detected area belongs. The cropped image generation unit stores an expansion amount set for each region category, and includes the region after the detected region is expanded using the expansion amount determined by the region category information in the cropped image.

2. The teacher data generation device according to claim 1, characterized in that, The object is a printed circuit board, and the multiple area categories include at least plated areas and solder mask areas.

3. A teacher data generation device for generating teacher data, characterized in that, have: The image receiving unit receives a defect image of a predetermined size, including the detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image from an inspection device that images an object to detect defects. The cropped image generation unit, based on the defect information, crops the region including the detection region from the defect image to obtain a cropped image; The display control unit displays at least a portion of the defect image on the display screen; The determination result receiving unit accepts input of the operator's determination result regarding the defect category of the defect image displayed on the display screen; as well as The teacher data generation unit annotates the cropped image with the judgment result to generate teacher data. One of a plurality of inspection sensitivities is set for each location of the object. The defect information includes inspection sensitivity information, which indicates the inspection sensitivity used during inspection in the detection area. The cropped image generation unit stores an expansion amount set for each inspection sensitivity, and includes the region in the cropped image after the detection area is expanded using the expansion amount determined using the inspection sensitivity information.

4. The teacher data generation device according to any one of claims 1 to 3, characterized in that, The locations of the objects belong to one of multiple region categories. The defect information includes region category information indicating the region category to which the detected area belongs. In addition to the determination result, the teacher data generation unit also marks the region category to which the detected region belongs on the cropped image.

5. A method for generating teacher data, characterized in that, include: a) A process in which an inspection apparatus for detecting defects by imaging an object receives a defect image of a predetermined size, including a detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image; b) Step: Based on the defect information, cut out the region containing the detection area from the defect image to obtain a cut-out image; c) The step of displaying at least a portion of the defect image on a display; d) The process of receiving input of the operator's determination of the defect category based on the defect image displayed on the display; as well as e) Step: Annotate the cut image with the determination result to generate teacher data, wherein each position of the object belongs to one of multiple region categories. The defect information includes region category information indicating the region category to which the detected area belongs. In step b), an expansion amount is prepared for each region category, and the region after the detection region is expanded using the expansion amount determined by the region category information is included in the cropped image.

6. The teacher data generation method according to claim 5, characterized in that, The object is a printed circuit board, and the multiple area categories include at least plated areas and solder mask areas.

7. A method for generating teacher data, characterized in that, include: a) A process in which an inspection apparatus for detecting defects by imaging an object receives a defect image of a predetermined size, including a detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image; b) Step: Based on the defect information, cut out the region containing the detection area from the defect image to obtain a cut-out image; c) The step of displaying at least a portion of the defect image on a display; d) The process of receiving input of the operator's determination of the defect category based on the defect image displayed on the display; as well as e) Step: Annotate the cut image with the judgment result to generate teacher data; set one of multiple inspection sensitivities for each position of the object. The defect information includes inspection sensitivity information, which indicates the inspection sensitivity used during inspection in the detection area. In step b), an expansion amount is prepared for each inspection sensitivity setting, and the area after the detection area is expanded using the expansion amount determined using the inspection sensitivity information is included in the cropped image.

8. The teacher data generation method according to any one of claims 5 to 7, characterized in that, The locations of the objects belong to one of multiple region categories. The defect information includes region category information indicating the region category to which the detected area belongs. In step e), in addition to the determination result, the region category to which the detected region belongs is also marked on the cropped image. Using data from multiple teachers labeled with a region category, a learned model is generated for defect classification of that region category.

9. A recording medium recording a program that enables a computer to generate teacher data, characterized in that, The program is executed by a computer, causing the computer to perform the following: a) A process in which an inspection apparatus for detecting defects by imaging an object receives a defect image of a predetermined size, including a detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image; b) Step: Based on the defect information, cut out the region containing the detection area from the defect image to obtain a cut-out image; c) The step of displaying at least a portion of the defect image on a display screen; d) The process of receiving input of the operator's determination of the defect category based on the defect image displayed on the display; as well as e) Step: Annotate the cut image with the judgment result to generate teacher data. The locations of the objects belong to one of multiple region categories. The defect information includes region category information indicating the region category to which the detected area belongs. In step b), an expansion amount is prepared for each region category, and the region after the detection region is expanded using the expansion amount determined by the region category information is included in the cropped image.

10. A recording medium recording a program that enables a computer to generate teacher data, characterized in that, The program is executed by a computer, causing the computer to perform the following: a) A process in which an inspection apparatus for detecting defects by imaging an object receives a defect image of a predetermined size, including a detection area of ​​the defect, and defect information representing the extent of the detection area in the defect image; b) Step: Based on the defect information, cut out the region containing the detection area from the defect image to obtain a cut-out image; c) The step of displaying at least a portion of the defect image on a display screen; d) The process of receiving input of the operator's determination of the defect category based on the defect image displayed on the display; as well as e) Step: Annotate the cut image with the judgment result to generate teacher data. One of a plurality of inspection sensitivities is set for each location of the object. The defect information includes inspection sensitivity information, which indicates the inspection sensitivity used during inspection in the detection area. In step b), an expansion amount is prepared for each inspection sensitivity setting, and the area after the detection area is expanded using the expansion amount determined using the inspection sensitivity information is included in the cropped image.

Images