Learning system and learning method

Abstract

Provided is a learning system (5) for creating a determiner (4) for determining the type of a defect image indicating a defect of an object. A learning system (5) is provided with: a learning unit (51) for creating a determiner (4), which is a learned model for determining the type of a defect image, by performing a set number of rounds of learning using a teacher image group, which is a plurality of defect images to which a label of any one of a plurality of types has been assigned, and thereby creating a determiner (4) that is a learned model for determining the type of the defect image; a storage unit (52) that stores a determination probability when each teacher image of the teacher image group is input to the determiner (4) in the production process during a period of interest during learning of a set number of rounds; and an image extraction unit (53) that extracts a target image, which is a teacher image of which the category should be confirmed, from the teacher image group on the basis of a change in the determination probability of each teacher image during the attention period. As a result, it is possible to appropriately extract the subject image from the teacher image group.

Description

[0001] Reference to related applications

[0002] This application claims priority to Japanese patent application JP2023-158110 filed on September 22, 2023, the entire contents of which are incorporated into this application. Technical Field

[0003] The invention relates to a learning system and a learning method. Background Art

[0004] In the past, an inspection system was used to detect defects by photographing objects such as printed circuit boards. In the inspection system of Japanese Patent Publication No. 2021-177154 (Document 1), there are provided: a primary inspection unit that performs defective judgment based on an image obtained by photographing the object without using machine learning; and a secondary inspection unit that uses a machine learning model to distinguish between true defective products and misjudged products based on the image of the object judged as defective by the primary inspection unit. In this way, the reduction in productivity caused by the generation of misjudged products can be suppressed.

[0005] As described above, when a learned model (machine learning model) is used to determine a real defect or a suspected defect, it is necessary to generate learning data including a plurality of teacher images representing real defects or suspected defects, and to use the learning data for learning to create the learned model. In generating the learning data, the operator assigns a real defect or suspected defect label to each teacher image by determining the category of the real defect or suspected defect (i.e., by annotation).

[0006] In addition, in cases where the judgment accuracy of the learned model is low, it is sometimes preferable to reconfirm the category of the teacher image contained in the learning data (here, real defects and suspected defects). On the other hand, since multiple teacher images are included in the learning data, it is difficult to reconfirm all teacher images. Therefore, a method for screening out candidate samples that may be incorrectly annotated from a sample set (sampleset) (learning data) is disclosed in Japanese Patent Gazette No. 2023-42582 (Document 2). In this method, the sample set is processed using the learned model to determine the predicted data for the sample set and the reliability corresponding to the predicted data. Next, the accuracy of the learned model is determined based on the comparison between the predicted data and the annotated data. The accuracy is the proportion of samples in the sample set whose annotated data matches the predicted data. Then, the samples whose number is determined based on the accuracy are selected as candidate samples in order of reliability from low to high.

[0007] The method of Document 2 can easily extract images whose categories are to be confirmed, but because only a one-time determination result based on a learned model is used, an appropriate image may not be extracted. Summary of the invention

[0008] An object of the present invention is to appropriately extract images whose categories are to be confirmed from a group of teacher images.

[0009] Mode 1 of the present invention is a learning system for producing a determiner for determining the category of a defect image representing a defect of an object, wherein the learning system comprises: a learning unit, which produces a learned model, i.e., a determiner, for determining the category of a defect image by performing a set number of rounds of learning using a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of any one of a plurality of categories; a storage unit, which stores the determination probability when each teacher image of the teacher image group is input into the determiner in the production process during a focus period in the learning of the set number of rounds; and an image extraction unit, which extracts a teacher image, i.e., an object image, of a category to be confirmed from the teacher image group based on changes in the determination probability of each teacher image during the focus period.

[0010] According to the present invention, it is possible to appropriately extract a target image whose category is to be confirmed from a group of teacher images.

[0011] According to a second aspect of the present invention, in the learning system of the first aspect, the categories of the defect images are real defects and suspected defects.

[0012] According to a third aspect of the present invention, in the learning system of the first or second aspect, a plurality of attention rounds are set in the attention period, and the image extraction unit extracts the object image based on changes in the judgment probability of each teacher image between the plurality of attention rounds.

[0013] In mode 4 of the present invention, in the learning system of mode 3, the image extraction unit extracts the object image based on the number of times the category judgment result switched by the judgement device between two consecutive attention rounds in the plurality of attention rounds, and / or the variation range of the judgment probability between the two consecutive attention rounds.

[0014] In mode 5 of the present invention, in the learning system of mode 1 or 2 (or any one of modes 1 to 4), the learning system further comprises: a display control unit, which displays the object image on a display unit; an input receiving unit, which receives an input of the category of the object image; and an image group updating unit, which updates the teacher image group by assigning a label of the category shown in the input to the object image, and the learning unit recreates the determiner by learning using the updated teacher image group.

[0015] According to a sixth aspect of the present invention, in the learning system of the fifth aspect, the display control unit displays, on the display unit, information indicating changes in the determination probability of the target image during learning using the updated teacher image group.

[0016] According to mode 7 of the present invention, in the learning system of mode 1 or 2 (or any one of modes 1 to 6), the learning system further comprises: an input receiving unit for receiving input of a change in the attention period; and a display control unit for performing display control of the display unit, wherein the storage unit stores the judgment probability of each teacher image during substantially the entire period of the learning of the set number of rounds, the display control unit displays information related to the object image extracted by the image extraction unit on the display unit, and the image extraction unit re-extracts the object image based on the change in the judgment probability of each teacher image during the attention period indicated by the changed input.

[0017] Mode 8 of the present invention is a learning method for producing a determinator for determining the category of a defect image representing a defect of an object, wherein the learning method comprises: step a), producing a learned model, i.e., a determinator, for determining the category of a defect image by performing a set number of learning rounds using a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of any one of a plurality of categories; step b), obtaining a determination probability when each teacher image of the teacher image group is input into the determinator in the production process during a focus period in the learning of the set number of rounds; and step c), extracting a teacher image, i.e., an object image, whose category should be confirmed, from the teacher image group based on a change in the determination probability of each teacher image during the focus period.

[0018] Mode 9 of the present invention, in the learning method of mode 8, the learning method comprises: step d), displaying the object image on a display unit; step e), determining the category of the object image; step f), updating the teacher image group by assigning a label of the category determined in step e) to the object image; and step g), remaking the determiner by learning using the updated teacher image group.

[0019] The above-mentioned object and other objects, features, aspects and advantages will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram showing the structure of the inspection system.

[0021] Figure 2 A diagram showing the structure of a computer.

[0022] Figure 3 This is a diagram showing the flow of processing for creating a determiner.

[0023] Figure 4 It is a diagram showing the flow of image extraction processing.

[0024] Figure 5 This is a diagram showing changes in the determination probability of the teacher image as the rounds pass.

[0025] Description of reference numerals:

[0026] 4Determinator

[0027] 5. Learning System

[0028] 35 Display

[0029] 51 Learning Department

[0030] 52 Storage

[0031] 53 Image extraction unit

[0032] 54 Display control unit

[0033] 55 Input receiving unit

[0034] 56 Image Group Update Department

[0035] 521 Learning Data

[0036] 522 Determination Probability Data

[0037] P1, P2 attention period

[0038] Steps S11 to S21, S141 to S145 DETAILED DESCRIPTION

[0039] Figure 1 1 is a block diagram showing the structure of an inspection system 1 according to an embodiment of the present invention. The inspection system 1 inspects a printed circuit board as an object. The inspection system 1 includes an inspection device 2, a computer 3, and a defect confirmation device 11. Figure 1 In the figure, a dotted rectangular frame is used to enclose the functional structure implemented by the computer 3.

[0040] The inspection device 2 includes a photographing unit, a moving mechanism, and an inspection unit 20. The photographing unit photographs the printed circuit board. The moving mechanism moves the printed circuit board relative to the photographing unit. The inspection unit 20 is implemented by, for example, a computer and / or a circuit. The inspection unit 20 performs a predetermined inspection process on the photographed image output from the photographing unit, and detects defects from the photographed image. The inspection process performed by the inspection unit 20 is, for example, a design rule check (DRC: Design Rule Check) for detecting defect candidates by comparing the photographed image with the design data of the wiring pattern, or a comparative inspection process for detecting defect candidates by comparing a main image prepared as a defect-free image with the photographed image.

[0041] The computer 3 realizes the function of the determiner 4. As described later, the determiner 4 is a learning model (AI (artificial intelligence) model) created by learning using the learning data 521. The determiner 4 determines the type of the image by inputting the image from the inspection device 2. The defect confirmation device 11 takes a picture of the printed circuit board inspected by the inspection device 2 and displays it on the display.

[0042] In the inspection of the printed circuit board in the inspection system 1, in the inspection device 2, a plurality of captured images representing a plurality of positions of the printed circuit board are obtained, and the plurality of captured images are inspected for defects using the inspection unit 20. The captured images may be either grayscale images or color images. When a defect is detected, an area containing the defect is extracted from the captured images as a defect image, and the defect image and defect-related information are output to the computer 3. For example, the defect-related information includes the position on the printed circuit board represented by the defect image (i.e., the position information of the defect image on the printed circuit board), etc.

[0043] In the determiner 4, by inputting a defect image, it is determined whether the defect shown in the defect image is a real defect or a suspected defect. Suspected defects are also called false reports or false defects. In fact, by inputting a defect image, the determination probability described later is obtained, and the type of defect is determined based on the determination probability. In the case where the determiner 4 determines that the defect shown in the defect image is a real defect, the defect image and defect-related information of the defect are output to the defect confirmation device 11.

[0044] The printed circuit board inspected by the inspection device 2 is carried into the defect confirmation device 11, and the area of ​​the defect image on the printed circuit board is photographed with reference to the defect-related information. The photographed image is displayed on the display of the defect confirmation device 11. The operator confirms the defect included in the displayed image and finally determines whether the defect is a real defect or a suspected defect. In addition, the image photographed by the defect confirmation device 11 can also be displayed on the display together with the defect image.

[0045] When the defect shown in the defect image is determined by the determiner 4 to be a suspected defect, the defect image and defect-related information of the defect are not output to the defect confirmation device 11, and the processing of the defect is completed. In this way, for the defect determined by the determiner 4 as a suspected defect, by omitting the confirmation by the operator, the operator's work hours required for confirming the defect can be reduced. In addition, the defect confirmation device 11 can also be omitted, and the determination result of the determiner 4 can be determined as the final category of the defect.

[0046] Figure 2 3 is a diagram showing the structure of a computer 3. The computer 3 has a structure of a general computer system including a CPU (Central Processing Unit) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a storage device 34, a display 35, an input unit 36, a reading device 37, a communication unit 38, a GPU (Graphics Processing Unit) 39, and a bus 30. The CPU 31 performs various calculations. The GPU 39 performs calculations related to image processing and the like. The ROM 32 stores basic programs. The RAM 33 and the storage device 34 store various information. The display 35 is a display unit that displays various information such as images. The input unit 36 ​​has a keyboard 36a and a mouse 36b that receive input from an operator. When the display 35 is a touch panel, the input unit 36 ​​also includes the touch panel. The reading device 37 reads information from a computer-readable recording medium 81 such as an optical disk, a magnetic disk, an optical magnetic disk, or a memory card. The communication unit 38 sends and receives signals to and from other components of the inspection system 1 and external devices. The bus 30 is a signal circuit that connects the CPU 31 , GPU 39 , ROM 32 , RAM 33 , storage device 34 , display 35 , input unit 36 ​​, reading device 37 , and the communication unit 38 .

[0047] In the computer 3, the program 811 is read from the recording medium 81 via the reading device 37 in advance and stored in the storage device 34. The program 811 can be stored in the storage device 34 via a network as long as it can be read by the computer. The CPU 31 and the GPU 39 perform calculation processing while using the RAM 33 and the storage device 34 according to the program 811. The CPU 31 and the GPU 39 function as a calculation unit in the computer 3. In addition to the CPU 31 and the GPU 39, other structures that function as a calculation unit can also be used.

[0048] In the inspection system 1, the computer 3 performs calculations and processing according to the program 811 to realize Figure 1The functional structure surrounded by dotted lines in FIG. 3 is shown. That is, the CPU 31, GPU 39, ROM 32, RAM 33, storage device 34 and their peripheral structures of the computer 3 realize the determination device 4 and the learning system 5. All or part of these functions can be realized by a dedicated circuit, or these functions can be realized by a separate program. In addition, these functions can also be realized by multiple computers.

[0049] The learning system 5 includes a learning unit 51, a storage unit 52, an image extraction unit 53, a display control unit 54, an input receiving unit 55, and an image group updating unit 56. The learning unit 51 uses the learning data 521 as training data to create the above-mentioned determiner 4 as a learning completed model. The learning data 521 includes a teacher image group as a plurality of defect images. The storage unit 52 stores the learning data 521, etc. The image extraction unit 53 extracts the object image as the teacher image of the category to be confirmed from the teacher image group. The display control unit 54 is connected to the display 35 to perform display control of the display 35. The input receiving unit 55 is connected to the input unit 36 ​​to receive input from the operator via the input unit 36. The image group updating unit 56 updates the teacher label described later of the teacher image group according to the input of the operator.

[0050] Figure 3 : is a diagram showing the processing flow of making the determiner 4 using the learning system 5. In making the determiner 4, first, prepare the learning data 521 (step S11). In an example of generating the learning data 521, first, the inspection unit 20 obtains a plurality of defect images from a plurality of captured images of a plurality of printed substrates. Next, the operator determines which category of the defect shown in each defect image is a true defect or a suspected defect, and annotates the defect image with the determined category. In this way, a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of a certain category of a true defect or a suspected defect (hereinafter also referred to as a "teacher label") can be obtained. The learning data 521 includes the teacher image group and label information representing the teacher label of each teacher image, and is stored in the storage unit 52 as preparation. The learning data 521 can be generated by using the learning system 5, or the learning data 521 generated in an external device can be input into the learning system 5.

[0051] When the learning data 521 is prepared, machine learning is performed in the learning unit 51 so that the output of the determiner for each teacher image included in the learning data 521 becomes substantially the same as the category (real defect or suspected defect) of the teacher image indicated by the label information, thereby creating a determiner (step S12). The determiner is a learned model that classifies defects shown in an image as real defects or suspected defects. In creating the determiner, the values ​​of the parameters included in the determiner and the structure of the determiner are determined. Figure 1In FIG. 1 , the determiner 4 in the process of being created in the learning unit 51 is represented by a dotted rectangle.

[0052] Machine learning is performed, for example, by deep learning (deep learning) using a neural network. Specifically, in the production of the determiner 4, learning using a plurality of teacher images (here, all teacher images) included in the teacher image group is taken as one epoch, and a preset number of epochs (hereinafter referred to as the "set number of epochs") of learning is repeated. The set number of epochs is the number of learning trials in the production of the determiner 4. In each epoch, the judgment probability when each teacher image of the teacher image group is input to the determiner 4 in the production process is obtained (step S13). The judgment probability is a value obtained by the determiner 4 in the production process in the epoch, indicating the probability (i.e., accuracy or reliability) that the category of the teacher image is a true defect. The judgment probability of the plurality of teacher images in all epochs is stored in the storage unit 52 as judgment probability data 522. In Figure 3 In the example, step S12 (production of the determiner 4) and step S13 (acquisition of the determination probability) are connected in series, but in fact, the two are performed in parallel. Machine learning can also be performed by a known method other than deep learning.

[0053] When the learning of the set number of rounds is completed, the image extraction unit 53 performs image extraction processing to extract a target image from the teacher image group (step S14). The target image is a teacher image whose category should be confirmed (re-evaluated). Figure 4 is a diagram showing the flow of image extraction processing. Figure 4 The processing enclosed by the dotted line (steps S143 to S145) is performed in a processing example described later, and is not performed in this processing example. Figure 5 is a diagram showing changes in the determination probability of the teacher images as the rounds pass, and the lines showing changes in the determination probability in the four teacher images are respectively labeled with reference numerals L1 to L4. Figure 5 In the example, only the decision probabilities of every 25 rounds are plotted when the number of rounds is set to 300.

[0054] In the image extraction process, first, the attention period, the determination threshold, and the allowable variation range are set for the image extraction unit 53 (step S141). Figure 5 In the figure, the arrow P1 indicates the focus period, and the thick line T1 indicates the judgment threshold. The focus period P1 is the period (area) of the round that is focused on in order to determine whether to extract each teacher image as the object image. It is typically a period in which the change in judgment probability is small in most teacher images. In the initial rounds, the judgment probability usually changes greatly, so it is preferred that the initial rounds are not included in the focus period P1. The judgment threshold T1 is the threshold of the judgment probability of judging each teacher image as a real defect or a suspected defect. Figure 5 In the example, it is 50. When the determination probability obtained by inputting the defect image to the determiner 4 is greater than the determination threshold T1, the defect shown in the defect image is determined to be a true defect. When the determination probability is less than the determination threshold T1, the defect is determined to be a suspected defect.

[0055] In the preferred image extraction process, every certain number (certain period) of rounds in the focus period P1 is set as the focus round. Typically, the number of focus rounds is less than the number of rounds included in the focus period P1. In this processing example, the focus round is every 25th round. Figure 5 In FIG. 1 , dots are used to represent the determination probability in each round of interest. The determination probability of each round of interest may also be the average of the determination probabilities of the round of interest and a plurality of rounds near the round of interest.

[0056] The absolute value of the difference in judgment probability between two consecutive rounds of attention (a pair of rounds of attention described later) is taken as the fluctuation range of the judgment probability, and the above-mentioned allowable fluctuation range is the maximum value allowed for the fluctuation range of the judgment probability. Typically, the allowable fluctuation range is a constant value, but usually the fluctuation range of the judgment probability decreases as the rounds pass, so the allowable fluctuation range may also decrease as the rounds pass. The setting of the attention period P1, the judgment threshold T1 and the allowable fluctuation range of the image extraction unit 53 is performed, for example, by the operator via the input unit 36. The operator's input is received by the input receiving unit 55, and the attention period P1, the judgment threshold T1 and the allowable fluctuation range are set in the image extraction unit 53. The attention period P1, the judgment threshold T1 and the allowable fluctuation range may also be set in advance.

[0057] In the image extraction unit 53, based on the change in the determination probability of each teacher image in the attention period P1 (i.e., the movement of the determination probability), the object image as the teacher image whose category should be confirmed is extracted from the teacher image group (step S142). In this processing example, each combination of two consecutive attention rounds is regarded as an attention round pair, and in the change of the determination probability of each teacher image, the attention round pair whose change range of the determination probability is greater than the allowable change range and the change of the determination probability crosses the determination threshold T1 is determined as a specific pair. In the specific pair, the determination probability changes with a change range larger than the allowable change range, and the determination result of the category performed by the determiner 4 is switched.

[0058] The number of specific pairs in each teacher image is compared with the number of allowed pairs. If the number of allowed pairs is exceeded, the teacher image is extracted as the target image. The target image in this processing example is an image in which the change in the determination probability during the attention period P1 is significant, and an image in which the determination result of the category performed by the determiner 4 during the attention period P1 changes frequently (fluctuates). In addition, the number of allowed pairs can also be set in step S141 in the same way as the attention period P1, etc.

[0059] exist Figure 5 In the example, for each teacher image, in the two rounds of attention that become specific pairs, the points of the judgment probability in the subsequent round of attention are surrounded by a thin circle. In the teacher image representing the change of judgment probability on line L1, the number of specific pairs is 0, and the judgment probability in all rounds of attention is greater than the judgment threshold T1. Therefore, the teacher image is considered to be a real defect. In the teacher image representing the change of judgment probability on line L4, the number of specific pairs is 0, and the judgment probability in all rounds of attention is less than the judgment threshold T1. Therefore, the teacher image is considered to be a suspected defect.

[0060] On the other hand, in the teacher image indicating the change of determination probability of line L2 and the teacher image indicating the change of determination probability of line L3, the number of specific pairs is 3 and 7 respectively, and the determination result of which category is a real defect or a suspected defect changes. In fact, it can be said that the more the number of specific pairs, the more difficult it is for the determiner 4 to determine the image. Here, the teacher image of line L3 in which the number of specific pairs exceeds the number of allowed pairs of 5 is extracted as the target image. In addition, in the extraction of the target image, since only the determination probability in the attention period P1 is used, when the attention period is pre-set, the determination probability in the attention period P1 can also be stored in the storage unit 52.

[0061] When the target image is extracted, the target image is displayed on the display 35 of the computer 3 by the display control unit 54 ( Figure 3 :Step S15). In one example, a plurality of thumbnails of object images are arranged and displayed in a window on the display 35, and the operator selects a thumbnail of an object image through the input unit 36, so that the object image can be enlarged and displayed on the display 35 (hereinafter referred to as "selected object image"). The object image displayed on the display 35 can be selected by various known methods.

[0062] Next, the operator determines the category of the selection object image displayed on the display 35 as a real defect or a suspected defect (step S16). In one example, the selection object image and a button indicating "real defect" and a button indicating "suspected defect" are displayed together in a window on the display 35. The operator confirms the selection object image and selects any button through the input unit 36 ​​to input whether the defect shown in the selection object image is a real defect or a suspected defect. The input is received by the input receiving unit 55. The input of the category of the selection object image by the operator can be performed by various well-known methods.

[0063] In the image group updating unit 56, a new teacher label of the category indicated by the input, i.e., the category determined by the operator, is assigned to the selected target image (step S17). In practice, the teacher image group is updated by repeating the above steps S15 to S17 for a plurality of (all) target images. Figure 3 In FIG. 5 , the diagram showing the repeated processing of the above steps S15 to S17 is omitted. The updating of the teacher image group is the updating of the label information of the learning data 521. In addition, according to the design of the learning system 5, in step S16, an input indicating that the selection target image is excluded from the teacher image group may be performed.

[0064] When the teacher image group is updated, the learning unit 51 performs the same learning as the process of step S12 using the updated teacher image group (i.e., the updated learning data 521), thereby reproducing the determiner 4 (step S18). At this time, similarly to the process of step S13, the determination probability of each teacher image in all rounds is obtained and stored in the storage unit 52 as new determination probability data 522 (step S19). In addition, the set number of rounds in the process of reproducing the determiner 4 may be changed according to the set number of rounds in step S12.

[0065] When the learning of the set number of rounds is completed, for each object image whose category has been confirmed, the display control unit 54 displays information indicating changes in the determination probability on the display 35 (step S20). The information indicating changes in the determination probability of the object image is obtained during the learning using the updated teacher image group. In one example, a graph showing changes in the determination probability of each object image as the rounds pass (see Figure 5). Similarly to step S142, the number of specific pairs can also be calculated for each object image, and the number of specific pairs can be displayed as information on the display 35 through the display control unit 54. For example, when the change in the determination probability of the object image in the later rounds is small (or the number of specific pairs is small), it can be captured that the category determined for the object image in the above step S16 is appropriate. In step S20, the determination accuracy of the teacher image group performed by the determiner 4 in the final round (relative to the correctness of the teacher label) and the like can also be displayed.

[0066] The operator refers to the information indicating the change of the determination probability of each target image, the determination accuracy of the newly created determination device 4, etc., and when a new target image is extracted, the operator inputs an instruction image extraction process. The input is received by the input receiving unit 55, and the process returns to step S14 to perform image extraction process (step S21). In this image extraction process, the determination probability data 522 just obtained is used. After the image extraction process, steps S15 to S20 are performed in the same manner as above.

[0067] On the other hand, when the production of the determiner 4 is terminated without extracting a new object image, the operator inputs an instruction to terminate the process. The input is received by the input receiving unit 55, and the production process of the determiner 4 is completed (step S21). The produced determiner 4 is used for the inspection of the printed circuit board described above. In addition, the processing of steps S20 and S21 may be omitted, and the production process of the determiner 4 is completed after the determiner 4 is newly produced.

[0068] The extraction of the object image in the above step S142 can be performed by various methods. For example, the specific pair is not limited to the attention round pair whose fluctuation range of the judgment probability is greater than the allowable fluctuation range and whose change of the judgment probability crosses the judgment threshold T1. The attention round pair whose fluctuation range of the judgment probability is greater than the allowable fluctuation range or whose change of the judgment probability crosses the judgment threshold T1 can also be determined as the specific pair. In addition, it is not necessarily necessary to determine a specific pair. For example, for each teacher image, when the average fluctuation range of the judgment probability in a plurality of attention round pairs or the maximum value of the fluctuation range of the judgment probability is greater than a specified value, the teacher image can also be extracted as the object image.

[0069] Furthermore, it is not necessary to use the value derived from the attention round pair. For each teacher image, when the difference between the maximum and minimum values ​​of the determination probability in a plurality of attention rounds is greater than a specified value, or when the value representing the deviation of the determination probability in a plurality of attention rounds (standard deviation, etc.) is greater than a specified value, the teacher image can be extracted as the object image. The plurality of attention rounds included in the attention period P1 can be any number of rounds or every other round, that is, all rounds included in the attention period P1. In the image extraction unit 53, the object image can be extracted by various methods based on the change of the determination probability of each teacher image in the attention period P1.

[0070] As mentioned above, Figure 1 The learning system 5 includes a learning unit 51, a storage unit 52, and an image extraction unit 53. The learning unit 51 uses a plurality of defect images, i.e., a teacher image group, each of which is labeled as a real defect or a suspected defect, to perform a set number of rounds of learning, thereby producing a learned model, i.e., a determiner 4, for determining the category of the defect image. The storage unit 52 stores the determination probability when each teacher image of the teacher image group is input to the determiner 4 in the production process during the focus period P1 in the learning of the set number of rounds. The image extraction unit 53 extracts the teacher image, i.e., the object image, of which the category should be confirmed, from the teacher image group based on the change in the determination probability of each teacher image during the focus period P1. In this way, in the learning system 5, by considering the change in the determination probability of each teacher image in the repeated learning process, it is possible to extract an appropriate object image for the determination fluctuation of the determiner 4.

[0071] Preferably, the learning system 5 includes: a display control unit 54 that causes the display 35 to display the object image; an input receiving unit 55 that receives an input of the category of the object image; and an image group updating unit 56 that updates the teacher image group by assigning a label of the category indicated by the input to the object image. Then, the learning unit 51 re-creates the determiner 4 by learning using the updated teacher image group. In this way, by re-creating the determiner 4 using the teacher image group in which the category of the object image has been re-evaluated, a high-precision determiner 4 can be obtained.

[0072] Here, the processing of the comparative example is assumed. In the processing of the comparative example, a teacher image whose category determined by the determiner 4 is different from the category of the teacher label, a teacher image whose determination probability of the determiner 4 for the category of the teacher label is low, or a teacher image whose determination probability of the determiner 4 is included in a specified range near the determination threshold is extracted as the object image. However, in these processes, since only one determination result of the determiner 4 (for example, the result in the final round) is used, the object image is sometimes not properly extracted. In addition, it is sometimes impossible to narrow the range of the object image (that is, to extract more object images), and the confirmation operation of the category of the object image takes a long time.

[0073] In contrast, the learning system 5 can extract appropriate target images by narrowing the range based on the change in the determination probability of each teacher image during the attention period P1. This can shorten the time required for confirming the type of the target image and efficiently correct the learning data 521.

[0074] Preferably, the display control unit 54 displays information indicating changes in the determination probability of the target image during learning using the updated teacher image group on the display 35. This allows the operator to easily determine whether the current teacher label of the target image is appropriate, thereby generating optimal learning data 521.

[0075] Preferably, a plurality of attention rounds are set in the attention period P1, and the image extraction unit 53 extracts the target image based on the change in the determination probability of each teacher image between the plurality of attention rounds. In this way, by reducing the number of rounds referring to the determination probability, the processing in the image extraction unit 53 can be simplified.

[0076] Preferably, the image extraction unit 53 extracts the target image based on the number of times the determination result of the category is switched by the determiner 4 between two consecutive focus rounds (the above-mentioned focus round pair) among the plurality of focus rounds and / or the variation range of the determination probability between the two consecutive focus rounds. Thus, it is possible to more reliably extract the appropriate target image.

[0077] Figure 3 The learning method includes: a process of making a determiner 4 by learning a set number of rounds using a teacher image group (step S12); a process of obtaining the determination probability when each teacher image is input to the determiner 4 in the process of making during the attention period P1 of the learning of the set number of rounds (step S13); and a process of extracting a target image from the teacher image group based on the change of the determination probability of each teacher image in the attention period P1 (step S142). Thus, the target image can be appropriately extracted from the teacher image group.

[0078] Next, Figure 4In this other processing example, Figure 4 The processing surrounded by dotted lines in (steps S143 to S145). In the image extraction processing, as described above, the attention period P1, the judgment threshold T1 and the allowable variation range are set (step S141), and the object image is extracted from the teacher image group (step S142). In this processing example, the information related to the extracted object image is displayed on the display 35 via the display control unit 54 (step S143). The information related to the object image is, for example, the number of object images. The information related to the object image can also be a curve chart showing the change in the judgment probability of each object image as the rounds pass (refer to Figure 5 ), or thumbnails of multiple object images, etc.

[0079] Next, the operator confirms the information related to the object image displayed on the display 35 to determine whether the attention period P1 needs to be changed. Whether the attention period P1 needs to be changed is determined based on the number of object images, changes in the determination probability of the object images, etc. When there is no need to change the attention period P1, the image extraction process ends (step S144), and the above-mentioned steps S15 to S21 are performed. On the other hand, when the number of object images is too large, or when there is a certain trend in the change of the determination probability of the object image, it is determined that the attention period P1 needs to be changed. In this case, the input unit 36 ​​is used to input the change of the attention period (steps S144, S145). The input is received by the input receiving unit 55, and the changed attention period is set in the image extraction unit 53. Figure 5 In the example of , the changed focus period is indicated by arrow P2.

[0080] In the image extraction unit 53, based on the change in the determination probability of each teacher image in the changed attention period P2, the object image is re-extracted from the teacher image group (step S142). In addition, the information related to the re-extracted object image is displayed on the display 35 through the display control unit 54 (step S143). In the case where the attention period P2 needs to be changed, the attention period is changed in the same way as described above (steps S144, S145), and steps S142 and 143 are performed. In the case where the attention period P2 does not need to be changed, the image extraction process ends (step S144), and the processing of steps S15 to S21 is performed.

[0081] As described above, in this processing example, after the object image is extracted by the image extraction unit 53, information related to the extracted object image is displayed on the display 35 by the display control unit 54. After the operator who has confirmed the information inputs a change in the attention period and the input receiving unit 55 accepts it, the image extraction unit 53 re-extracts the object image based on the change in the judgment probability of each teacher image in the attention period P2 shown by the changed input. In this way, the object image whose category should be confirmed can be more appropriately extracted, so that the learning data 521 can be corrected with high precision. According to the design of the learning system 5, the attention period can be replaced, or at least one of the judgment threshold, the allowable fluctuation range, and the number of allowed pairs can be changed together with the attention period, and the object image can be re-extracted using the changed value. In addition, the interval of the attention rounds contained in the attention period P1 can also be changed.

[0082] Various modifications can be made to the above-described learning system 5 and learning method.

[0083] The categories of defect images are not limited to true defects and suspected defects, and may also include detailed categories of defects such as foreign matter adhesion and film peeling. In addition, the categories of defect images are not limited to two, and may be three or more. In this case, a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of any one of a plurality of categories (for example, three or more) is prepared, and the learning unit 51 uses the teacher image group for learning to create the determiner 4.

[0084] In the case where there are three or more categories of defective images, in the determiner 4, for example, the determination probabilities are output respectively for the three or more categories. In the image extraction unit 53, for example, for each teacher image, a pair of rounds of interest whose fluctuation range of the determination probability of a specific category is greater than the allowed fluctuation range is determined as a specific pair. An example of the specific category is the category taught to each teacher image (i.e., the category of the teacher label). Then, the object image is extracted by comparing the number of specific pairs with the number of allowed pairs. Similar to the case where there are two categories of defective images, the object image can also be extracted based on the average of the fluctuation range of the determination probability, etc.

[0085] In the above-mentioned embodiment, for each teacher image, the judgment probability in all rounds (i.e., the judgment probability during the entire period of learning for a set number of rounds) is stored in the storage unit 52 as the judgment probability data 522. However, in the case where the focus period is set in advance, it is also possible to store only the judgment probability during the focus period. In this way, the storage unit 52 only needs to store the judgment probability of each teacher image obtained by the judge 4 in the production process at least during the focus period of learning for a set number of rounds. On the other hand, when performing Figure 4In the processing example of the process surrounded by the dotted line (steps S143 to S145), since the attention period is changed, the storage unit 52 stores the determination probability of each teacher image during substantially the entire period of learning of the set number of rounds (for example, excluding the initial period of learning).

[0086] The object to be inspected in the inspection unit 20 may be a substrate such as a semiconductor substrate or a glass substrate in addition to a printed circuit board. In addition, defects of objects other than substrates such as mechanical parts may be detected by the inspection unit 20. Thus, the defect image processed in the learning system 5 may also be an image showing defects of various objects.

[0087] The configurations in the above-described embodiment and various modifications may be appropriately combined as long as they do not contradict each other.

[0088] The invention has been described and illustrated in detail, but the above description is illustrative rather than restrictive. Therefore, it can be said that many modifications and aspects are possible without departing from the scope of the invention.

Claims

1. A learning system for producing a classifier for determining a type of a defect image representing a defect of an object, wherein: The learning system has: The learning unit creates a learned model, i.e., a determiner, for determining the category of a defect image by performing a set number of rounds of learning using a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of any one of a plurality of categories; a storage unit for storing, during the attention period in the learning of the set number of rounds, the determination probability when each teacher image of the teacher image group is input to the determination device in the production process; as well as The image extraction unit extracts a target image, which is a teacher image of a class to be confirmed, from the teacher image group based on a change in the determination probability of each teacher image during the attention period.

2. The learning system according to claim 1, wherein: The categories of the defect images are real defects and suspected defects.

3. The learning system according to claim 1 or 2, wherein: A plurality of attention rounds are set in the attention period. The image extraction unit extracts the target image based on changes in the determination probabilities of the teacher images between the plurality of focus rounds.

4. The learning system according to claim 3, wherein: The image extraction unit extracts the target image based on the number of times the determination result of the category performed by the determiner switches between two consecutive attention rounds among the plurality of attention rounds and / or the variation range of the determination probability between the two consecutive attention rounds.

5. The learning system according to claim 1 or 2, wherein: The learning system also has: a display control unit, which displays the object image on a display unit; An input receiving unit receiving an input of a category of the object image; as well as an image group updating unit that updates the teacher image group by assigning a label of a category indicated by the input to the target image, The learning unit recreates the determiner by performing learning using the updated teacher image group.

6. The learning system according to claim 5, wherein: The display control unit displays, on the display unit, information indicating a change in the determination probability of the target image during learning using the updated teacher image group.

7. The learning system according to claim 1 or 2, wherein: The learning system also has: an input receiving unit that receives an input of a change in the period of interest; and A display control unit performs display control of the display unit, The storage unit stores the determination probability of each teacher image during substantially the entire period of the learning of the set number of rounds, the display control unit displays information related to the target image extracted by the image extraction unit on the display unit, The image extraction unit re-extracts a target image based on a change in the determination probability of each teacher image in the attention period indicated by the changed input.

8. A learning method for producing a classifier for determining the type of a defect image representing a defect of an object, wherein: The learning method has: Step a), by using a plurality of defect images, i.e., a teacher image group, each of which is assigned a label of any one of a plurality of categories, to perform a set number of rounds of learning, thereby creating a learned model, i.e., a determiner, for determining the category of the defect image; Step b), obtaining a determination probability when each teacher image of the teacher image group is input to the determination device in the production process during the focus period of the learning of the set number of rounds; as well as Step c) extracting a target image, which is a teacher image of a class to be confirmed, from the teacher image group based on a change in the determination probability of each teacher image during the attention period.

9. The learning method according to claim 8, wherein: The learning method has: Step d), displaying the object image on a display unit; Step e), determining the category of the object image; Step f), updating the teacher image group by assigning a label of the category determined in step e) to the target image; as well as Step g) is to recreate the determiner by learning using the updated teacher image group.

Images