Learning system and learning method

Abstract

To appropriately extract an image of which the type is to be confirmed, from a teacher image group.SOLUTION: A learning system 5 generates a discriminator 4 which discriminates a type of a defect image indicating a defect of an object. The learning system 5 comprises: a learning unit 51 which generates the discriminator 4 being a trained model which discriminates the type of the defect image by using a teacher image group being a plurality of defect images each of which a label of one of a plurality of types is given, to perform learning of the set number of epochs; a storage unit 52 for storing discrimination probabilities obtained by inputting respective teacher images of the teacher image group to the discriminator 4 being under generation, in a period of interest during learning of the set number of epochs; and an image extraction unit 53 for extracting an object image being a teacher image of which the type should be confirmed, from the teacher image group on the basis of variation in discrimination probability of respective teacher images in the period of interest. Thus, the object image can be appropriately extracted from the teacher image group.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning system and a learning method. [Background technology]

[0002] Conventionally, an inspection system has been used that captures an image of an object such as a printed circuit board and detects defects. The inspection system of Patent Document 1 includes a primary inspection unit that performs defect judgment without using machine learning based on the captured image of the object, and a secondary inspection unit that uses a machine learning model to distinguish between true defective products and over-determined products based on the image of the object judged to be defective by the primary inspection unit. This suppresses a decrease in productivity caused by the occurrence of over-determined products.

[0003] As described above, when a trained model (machine learning model) is used to determine whether a defect is true or false, it is necessary to generate training data including a plurality of teacher images each of which indicates a true defect or a false defect, and to perform training using the training data to create the trained model. In generating the training data, an operator determines the type of defect (true defect or false defect) for each teacher image (i.e., by annotation), and assigns a label of true defect or false defect to the teacher image.

[0004] However, when the judgment accuracy of the trained model is low, it may be preferable to reconfirm the type of teacher image (here, true defect and pseudo defect) included in the training data. On the other hand, since the training data includes a large number of teacher images, it is difficult to reconfirm all the teacher images. Therefore, Patent Document 2 discloses a method for selecting candidate samples that may be erroneously annotated from a sample set (training data). In this method, predicted data for the sample set and a reliability corresponding to the predicted data are determined by processing the sample set using a trained model. Next, the accuracy of the trained model is determined based on a comparison between the predicted data and the annotation data. The accuracy is the proportion of samples whose annotation data matches the predicted data in the sample set. Then, the number of samples determined based on the accuracy is selected as candidate samples in order of decreasing reliability. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-177154 A [Patent Document 2] JP 2023-42582 A Summary of the Invention [Problem to be solved by the invention]

[0006] The method of Patent Document 2 makes it possible to easily extract images whose types should be confirmed, but since only one judgment result from a trained model is used, it may not be possible to extract appropriate images.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to appropriately extract images whose types should be confirmed from a group of teacher images. [Means for solving the problem]

[0008] A first aspect of the present invention is a learning system for creating a classifier that determines the type of a defect image that indicates a defect in an object, and includes a learning unit that creates a classifier that is a trained model that determines the type of defect image by performing learning for a set number of epochs using a group of teacher images, which are a plurality of defect images each of which is labeled with one of a plurality of types; a memory unit that stores the judgment probability when each teacher image of the group of teacher images is input to the classifier in the process of being created during a period of interest in learning for the set number of epochs; and an image extraction unit that extracts a target image, which is a teacher image whose type should be confirmed, from the group of teacher images based on a change in the judgment probability of each teacher image during the period of interest.

[0009] A second aspect of the present invention is the learning system of the first aspect, wherein the types of the defect images are true defects and pseudo defects.

[0010] A third aspect of the present invention is a learning system according to the first or second aspect, wherein a plurality of epochs of interest are set in the attention period, and the image extraction unit extracts the target image based on the change in the judgment probability of each teacher image between the plurality of epochs of interest.

[0011] A fourth aspect of the present invention is a learning system according to the third aspect, wherein the image extraction unit extracts the target image based on the number of times that the type judgment result by the classifier switches between two consecutive epochs of interest in the plurality of epochs of interest, and / or the range of fluctuation of the judgment probability between the two consecutive epochs of interest.

[0012] A fifth aspect of the present invention is a learning system of the first or second aspect (which may be any one of the first to fourth aspects), further comprising a display control unit that displays the target image on a display unit, an input receiving unit that receives input of the type of the target image, and an image group update unit that updates the teacher image group by assigning a label of the type indicated by the input to the target image, and the learning unit recreates the classifier by learning using the updated teacher image group.

[0013] A sixth aspect of the present invention is a learning system of the fifth aspect, wherein the display control unit displays on the display unit information indicating a change in the judgment probability of the target image in learning using the updated group of teacher images.

[0014] A seventh aspect of the present invention is a learning system of aspect 1 or 2 (which may be any one of aspects 1 to 6), further comprising an input receiving unit that receives input for changing the attention period, and a display control unit that controls the display of a display unit, wherein the memory unit stores the judgment probability of each teacher image for almost the entire period of learning the set number of epochs, the display control unit displays information about the target image extracted by the image extraction unit on the display unit, and the image extraction unit re-extracts the target image based on the change in the judgment probability of each teacher image in the attention period indicated by the input of the change.

[0015] Aspect 8 of the present invention is a learning method for creating a classifier that determines the type of a defect image that indicates a defect in an object, comprising the steps of: a) creating a classifier that is a trained model that determines the type of defect image by performing learning for a set number of epochs using a group of teacher images, which are a plurality of defect images each of which is labeled with one of a plurality of types; b) acquiring a judgment probability when each teacher image of the group of teacher images is input to the classifier in the process of being created during a period of interest in learning for the set number of epochs; and c) extracting a target image, which is a teacher image whose type should be confirmed, from the group of teacher images based on a change in the judgment probability of each teacher image during the period of interest.

[0016] A ninth aspect of the present invention is a learning method of aspect 8, comprising the steps of d) displaying the target image on a display unit, e) determining a type of the target image, f) updating the group of teacher images by assigning a label of the type determined in step e) to the target image, and g) re-creating the classifier by learning using the updated group of teacher images. Effect of the Invention

[0017] According to the present invention, it is possible to appropriately extract target images, the type of which should be confirmed, from a group of teacher images. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram showing a configuration of an inspection system. [Diagram 2] FIG. 1 illustrates a configuration of a computer. [Diagram 3] FIG. 13 is a diagram showing a process flow for creating a determinator. [Figure 4] FIG. 11 is a diagram showing the flow of an image extraction process. [Diagram 5] FIG. 13 is a diagram showing a change in the judgment probability of a teacher image over time as epochs progress. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Fig. 1 is a block diagram showing the configuration of an inspection system 1 according to an embodiment of the present invention. The inspection system 1 inspects a printed circuit board, which is an object. The inspection system 1 includes an inspection device 2, a computer 3, and a defect confirmation device 11. In Fig. 1, the functional configuration realized by the computer 3 is enclosed in a dashed rectangle.

[0020] The inspection device 2 includes an imaging unit, a moving mechanism, and an inspection unit 20. The imaging unit captures an image of the printed circuit board. The moving mechanism moves the printed circuit board relatively to the imaging unit. The inspection unit 20 is realized by, for example, a computer and / or an electric circuit. The inspection unit 20 executes a predetermined inspection process on the captured image output from the imaging unit, and detects defects from the captured image. The inspection process performed by the inspection unit 20 is, for example, a design rule check (DRC) that detects defect candidates by comparing the captured image with design data of a wiring pattern, and a comparison inspection process that detects defect candidates by comparing the captured image with a master image prepared as a defect-free image.

[0021] The computer 3 realizes the function of the determiner 4. As described later, the determiner 4 is a trained model (AI (artificial intelligence) model) created by learning using learning data 521. The determiner 4 determines the type of an image input from the inspection device 2. The defect confirmation device 11 captures an image of the printed circuit board after inspection by the inspection device 2 and displays the image on a display.

[0022] In the inspection of a printed circuit board in the inspection system 1, the inspection device 2 acquires a plurality of captured images showing a plurality of positions on the printed circuit board, and the inspection unit 20 inspects the plurality of captured images for the presence or absence of defects. The captured images may be either grayscale images or color images. When a defect is detected, an area including the defect is extracted from the captured image as a defect image, and the defect image and defect-related information are output to the computer 3. For example, the defect-related information may include the position on the printed circuit board indicated by the defect image (i.e., position information of the defect image on the printed circuit board), etc.

[0023] In response to an input of a defect image, the determiner 4 determines whether the defect indicated by the defect image is a real defect or a pseudo defect. A pseudo defect is also called a false report or a false defect. In practice, a determination probability, which will be described later, is obtained by inputting the defect image, and the type of the defect is determined based on the determination probability. If the determiner 4 determines that the defect indicated by the defect image is a real defect, the defect image and defect-related information of the defect are output to the defect confirmation device 11.

[0024] The printed circuit board inspected by the inspection device 2 is carried into the defect confirmation device 11, and the defect image area on the printed circuit board is imaged by referring to defect-related information. The image thus captured is displayed on the display of the defect confirmation device 11. An operator confirms the defect contained in the displayed image, and finally determines whether the defect is a real defect or a pseudo defect. The display may also display the defect image together with the image captured by the defect confirmation device 11.

[0025] When the judger 4 judges that the defect indicated by the defect image is a pseudo defect, the defect image and defect-related information of the defect are not output to the defect confirmation device 11, and processing for the defect ends. In this way, by omitting confirmation by an operator of a defect judged to be a pseudo defect by the judger 4, the number of worker steps required for confirming the defect can be reduced. Note that the defect confirmation device 11 may be omitted, and the judgment result by the judger 4 may be used to determine the final type of the defect.

[0026] FIG. 2 is a diagram showing the configuration of the computer 3. The computer 3 has a configuration of a general computer system including a CPU 31, a ROM 32, a RAM 33, a storage device 34, a display 35, an input unit 36, a reading device 37, a communication unit 38, a GPU 39, and a bus 30. The CPU 31 performs various arithmetic processing. The GPU 39 performs arithmetic processing related to image processing and the like. The ROM 32 stores basic programs. The RAM 33 and the storage device 34 store various information. The display 35 is a display unit, and displays various information such as images. The input unit 36 ​​includes a keyboard 36a and a mouse 36b that accept input from an operator. If the display 35 is a touch panel, the input unit 36 ​​also includes the touch panel. The reading device 37 reads information from a computer-readable recording medium 81 such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The communication unit 38 transmits and receives signals between other components of the inspection system 1 and external devices. The bus 30 is a signal circuit that connects the CPU 31, the GPU 39, the ROM 32, the RAM 33, the storage device 34, the display 35, the input unit 36, the reading device 37, and the communication unit 38.

[0027] In the computer 3, the program 811 is read in advance from the recording medium 81 via the reading device 37 and stored in the storage device 34. The program 811 may be computer-readable and may be stored in the storage device 34 via a network. The CPU 31 and the GPU 39 execute arithmetic processing while using the RAM 33 and the storage device 34 in accordance with the program 811. The CPU 31 and the GPU 39 function as a calculation unit in the computer 3. Other configurations functioning as a calculation unit other than the CPU 31 and the GPU 39 may be employed.

[0028] In the inspection system 1, the computer 3 executes arithmetic processing and the like according to the program 811, thereby realizing the functional configuration surrounded by the dashed line in Fig. 1. That is, the CPU 31, GPU 39, ROM 32, RAM 33, storage device 34, and their peripheral configuration of the computer 3 realize the determiner 4 and the learning system 5. All or part of these functions may be realized by dedicated electric circuits, or these functions may be realized by individual programs. Furthermore, these functions may be realized by multiple computers.

[0029] The learning system 5 includes a learning unit 51, a storage unit 52, an image extraction unit 53, a display control unit 54, an input reception unit 55, and an image group update unit 56. The learning unit 51 uses learning data 521, which is training data, to create the above-mentioned classifier 4, which is a learned model. The learning data 521 includes a teacher image group, which is a plurality of defect images. The storage unit 52 stores the learning data 521 and the like. The image extraction unit 53 extracts a target image, which is a teacher image whose type should be confirmed, from the teacher image group. The display control unit 54 is connected to the display 35 and controls the display of the display 35. The input reception unit 55 is connected to the input unit 36 ​​and receives input from an operator via the input unit 36. The image group update unit 56 updates the teacher label, which will be described later, of the teacher image group according to the input from the operator.

[0030] FIG. 3 is a diagram showing a flow of a process for creating the classifier 4 using the learning system 5. In creating the classifier 4, first, learning data 521 is prepared (step S11). In one example of generating the learning data 521, first, a plurality of defect images are acquired from a large number of captured images of a plurality of printed circuit boards by the inspection unit 20. Next, an operator determines whether the defect indicated by each defect image is a true defect or a pseudo defect, and the determined type is labeled for the defect image. This results in a group of teacher images, which are a plurality of defect images to which a label (hereinafter also referred to as a "teacher label") of either a true defect or a pseudo defect is assigned. The learning data 521 includes the group of teacher images and label information indicating the teacher label of each teacher image, and is stored and prepared in the storage unit 52. The learning data 521 may be generated using the learning system 5, or the learning data 521 generated in an external device may be input to the learning system 5.

[0031] When the learning data 521 is prepared, the learning unit 51 performs machine learning so that the output of the classifier for each input of a teacher image included in the learning data 521 is approximately the same as the type of the teacher image indicated by the label information (true defect or pseudo defect), and creates a classifier (step S12). The classifier is a trained model that classifies defects indicated by an image into true defects or pseudo defects, and in creating the classifier, the values ​​of parameters included in the classifier and the structure of the classifier are determined. In Fig. 1, a classifier 4 in the process of being created in the learning unit 51 is indicated by a dashed rectangle.

[0032] Machine learning is performed by, for example, deep learning using a neural network. Specifically, in the creation of the classifier 4, learning using a plurality of teacher images (here, all teacher images) included in the teacher image group is defined as one epoch, and learning is repeated for a preset number of epochs (hereinafter, "set number of epochs"). The set number of epochs is the number of learning trials in the creation of the classifier 4. In each epoch, a judgment probability is obtained when each teacher image in the teacher image group is input to the classifier 4 in the process of creation (step S13). The judgment probability is a value obtained by the classifier 4 in the process of creation in the epoch, and indicates the probability (i.e., accuracy or reliability) that the type of the teacher image is a true defect. The judgment probabilities of the plurality of teacher images in all epochs are stored in the storage unit 52 as judgment probability data 522. In FIG. 3, step S12 (creation of the classifier 4) and step S13 (acquisition of the judgment probability) are connected in series, but in reality, both are performed in parallel. Machine learning may be performed by known methods other than deep learning.

[0033] When the learning of the set number of epochs is completed, the image extraction unit 53 performs an image extraction process to extract a target image from the group of teacher images (step S14). The target image is a teacher image whose type should be confirmed (reviewed). FIG. 4 is a diagram showing the flow of the image extraction process. The process surrounded by a dashed line in FIG. 4 (steps S143 to S145) is performed in a processing example described later, and is not performed in this processing example. FIG. 5 is a diagram showing the change in the judgment probability of a teacher image over the course of an epoch, and the lines showing the change in judgment probability for four teacher images are given the symbols L1 to L4, respectively. In the example of FIG. 5, the set number of epochs is 300, and only the judgment probability for every 25th epoch is plotted.

[0034] In the image extraction process, first, a period of interest, a judgment threshold, and an allowable fluctuation range are set for the image extraction unit 53 (step S141). In FIG. 5, the period of interest is indicated by an arrow P1, and the judgment threshold is indicated by a thick line T1. The period of interest P1 is a period (area) of an epoch to be noted for determining whether or not to extract each teacher image as a target image, and is typically a period in which the fluctuation of the judgment probability is small for many teacher images. In an early epoch, the judgment probability usually changes significantly, so it is preferable that the early epoch is not included in the period of interest P1. The judgment threshold T1 is a judgment probability threshold for judging each teacher image as either a true defect or a false defect, and is 50 in the example of FIG. 5. When the judgment probability obtained by inputting a defect image to the judge 4 is equal to or greater than the judgment threshold T1, the defect indicated by the defect image is judged to be a true defect, and when the judgment probability is less than the judgment threshold T1, the defect is judged to be a false defect.

[0035] Here, in a preferred image extraction process, an epoch every fixed number (fixed period) in the period of interest P1 is set as an epoch of interest. Typically, the number of epochs of interest is less than the number of epochs included in the period of interest P1, and in this processing example, the epoch of interest is every 25 epochs. In FIG. 5, the determination probability in each epoch of interest is indicated by a dot. The determination probability in each epoch of interest may be, for example, the average of the determination probabilities in the epoch of interest and a plurality of epochs in the vicinity of the epoch of interest.

[0036] The absolute value of the difference in judgment probability between two consecutive epochs of interest (an epoch pair of interest described later) is the fluctuation range of the judgment probability, and the above-mentioned allowable fluctuation range is the maximum value allowed for the fluctuation range of the judgment probability. Typically, the allowable fluctuation range is a constant value, but since the fluctuation range of the judgment probability usually becomes smaller as the epoch progresses, the allowable fluctuation range may become smaller as the epoch progresses. The attention period P1, the judgment threshold T1, and the allowable fluctuation range are set for the image extraction unit 53 by, for example, an operator via the input unit 36. The input by the operator is accepted by the input acceptance unit 55, and the attention period P1, the judgment threshold T1, and the allowable fluctuation range are set in the image extraction unit 53. The attention period P1, the judgment threshold T1, and the allowable fluctuation range may be set in advance.

[0037] Based on the change in the judgment probability of each teacher image during the attention period P1 (i.e., the behavior of the judgment probability), the image extraction unit 53 extracts target images, which are teacher images whose types should be confirmed, from the teacher image group (step S142). In this processing example, each combination of two consecutive epochs of interest is set as an epoch pair of interest, and an epoch pair of interest in which the fluctuation range of the judgment probability in the change in the judgment probability of each teacher image is larger than the allowable fluctuation range and the change in the judgment probability crosses the judgment threshold T1 is specified as a specific pair. In the specific pair, the judgment probability changes by a fluctuation range larger than the allowable fluctuation range, and the type judgment result by the judger 4 is switched.

[0038] The number of specific pairs in each teacher image is compared with a predetermined allowable number of pairs, and if the number exceeds the allowable number of pairs, the teacher image is extracted as a target image. The target image in this processing example is an image in which the fluctuation of the judgment probability in the attention period P1 is remarkable, and the judgment result of the type by the judger 4 in the attention period P1 frequently changes (is blurred). Note that the allowable number of pairs may be set in step S141, similarly to the attention period P1, etc.

[0039] In the example of Fig. 5, for each teacher image, of the two epochs of interest that form a specific pair, the dots of the judgment probability in the later epoch of interest are surrounded by a thin-line circle. In the teacher image showing the change in judgment probability of line L1, the number of specific pairs is 0, and the judgment probabilities in all epochs of interest are greater than the judgment threshold T1, so the teacher image is considered to be a true defect. In the teacher image showing the change in judgment probability of line L4, the number of specific pairs is 0, and the judgment probabilities in all epochs of interest are smaller than the judgment threshold T1, so the teacher image is considered to be a false defect.

[0040] On the other hand, in the teacher image showing the change in the determination probability of the line L2 and the teacher image showing the change in the determination probability of the line L3, the numbers of specific pairs are 3 and 7, respectively, and the determination result as to whether the type of defect is a real defect or a pseudo defect fluctuates. In reality, the greater the number of specific pairs, the more uncertain the image is for the determiner 4, and here, the teacher image of the line L3 in which the number of specific pairs exceeds the allowable number of pairs, 5, is extracted as the target image. Note that, since only the determination probability in the attention period P1 is used in the extraction of the target image, in cases such as when the attention period is set in advance, only the determination probability in the attention period P1 may be stored in the storage unit 52.

[0041] When the target image is extracted, the display control unit 54 displays the target image on the display 35 of the computer 3 (FIG. 3: step S15). In one example, thumbnails of a plurality of target images are displayed in an array in a window on the display 35, and when the worker selects a thumbnail of one target image via the input unit 36, the target image (hereinafter referred to as the "selected target image") is displayed large on the display 35. The selection of the target image to be displayed on the display 35 may be performed by various well-known methods.

[0042] Next, the operator determines the type of the selection target image displayed on the display 35 as either a real defect or a pseudo defect (step S16). In one example, a button indicating a "real defect" and a button indicating a "pseudo defect" are displayed in a window on the display 35 together with the selection target image. The operator checks the selection target image and selects one of the buttons via the input unit 36, thereby inputting the type indicating whether the defect shown by the selection target image is a real defect or a pseudo defect. The input is accepted by the input acceptance unit 55. The operator may input the type of the selection target image by using various well-known methods.

[0043] In the image group update unit 56, a new teacher label of the type indicated by the input, i.e., the type determined by the operator, is assigned to the selection target image (step S17). In practice, the teacher image group is updated by repeating the above steps S15 to S17 for a plurality (all) of the target images. In FIG. 3, the process showing the repetition of the above steps S15 to S17 is omitted. The teacher image group is updated by updating the label information of the learning data 521. Depending on the design of the learning system 5, it may be possible to input an input indicating the exclusion of a selection target image from the teacher image group in step S16.

[0044] When the group of teacher images is updated, the learning unit 51 uses the updated group of teacher images (i.e., updated learning data 521) to perform learning similar to the processing in step S12, thereby recreating the classifier 4 (step S18). At this time, similar to the processing in step S13, the judgment probability of each teacher image in all epochs is acquired and stored in the storage unit 52 as new judgment probability data 522 (step S19). Note that the number of epochs set in the recreation of the classifier 4 may be changed from the number of epochs set in step S12.

[0045] When the learning for the set number of epochs is completed, the display control unit 54 displays information indicating the change in the judgment probability for each target image for which the type has been confirmed on the display 35 (step S20). The information indicating the change in the judgment probability of the target image is acquired in learning using the updated teacher image group. In one example, a graph (see FIG. 5) indicating the change in the judgment probability of each target image over the course of epochs is displayed. As in step S142, the number of specific pairs may be found for each target image, and the display control unit 54 may display the number of specific pairs as the information on the display 35. For example, if the change in the judgment probability of the target image is small in the later epochs (or the number of specific pairs is small), it can be considered that the type determined for the target image in step S16 above was appropriate. In step S20, the judgment accuracy of the teacher image group by the classifier 4 in the final epoch (correct answer rate for the teacher label), etc. may be displayed.

[0046] When the operator refers to information indicating the change in the determination probability of each target image and the determination accuracy of the recreated determiner 4 and the like to extract a new target image, the operator inputs an instruction to perform image extraction processing. The input is received by the input receiving unit 55, and the process returns to step S14 to perform image extraction processing (step S21). In the image extraction processing, the previously acquired determination probability data 522 is used. After the image extraction processing, steps S15 to S20 are performed in the same manner as described above.

[0047] On the other hand, when a new target image is not to be extracted and the creation of the determinator 4 is to be ended, an input command to end the process is made by the operator. The input is accepted by the input accepting unit 55, and the process of creating the determinator 4 is completed (step S21). The created determinator 4 is used for the inspection of the printed circuit board as described above. Note that the processes of steps S20 and S21 may be omitted, and the process of creating the determinator 4 may be completed after the determinator 4 is recreated.

[0048] The extraction of the target image in step S142 may be performed by various methods. For example, the specific pair is not limited to a target epoch pair in which the fluctuation range of the judgment probability is larger than the allowable fluctuation range and the change in the judgment probability crosses the judgment threshold T1, and a target epoch pair in which the fluctuation range of the judgment probability is larger than the allowable fluctuation range or a target epoch pair in which the change in the judgment probability crosses the judgment threshold T1 may be specified as the specific pair. In addition, it is not necessary to determine a specific pair. For example, when the average of the fluctuation range of the judgment probability in a plurality of target epoch pairs or the maximum value of the fluctuation range of the judgment probability for each teacher image is larger than a predetermined value, the teacher image may be extracted as the target image.

[0049] Furthermore, it is not necessary to use a value derived from the epoch pair of interest, and when the difference between the maximum and minimum values ​​of the judgment probability in the multiple epochs of interest for each teacher image is greater than a predetermined value, or when a value (such as a standard deviation) indicating the variation in the judgment probability in the multiple epochs of interest is equal to or greater than a predetermined value, the teacher image may be extracted as the target image. The multiple epochs of interest included in the period of interest P1 may be an arbitrary number of epochs, or may be every epoch, that is, all epochs included in the period of interest P1. The image extraction unit 53 can extract the target image by various methods based on the change in the judgment probability of each teacher image in the period of interest P1.

[0050] As described above, the learning system 5 in FIG. 1 includes a learning unit 51, a storage unit 52, and an image extraction unit 53. The learning unit 51 creates a classifier 4, which is a trained model for determining the type of defect image, by performing learning for a set number of epochs using a group of teacher images, which are a plurality of defect images each of which is labeled with a type of either a true defect or a pseudo defect. The storage unit 52 stores the determination probability when each teacher image of the group of teacher images is input to the classifier 4 in the process of being created during the attention period P1 in the learning for the set number of epochs. The image extraction unit 53 extracts a target image, which is a teacher image whose type should be confirmed, from the group of teacher images based on the change in the determination probability of each teacher image during the attention period P1. In this way, the learning system 5 can extract an appropriate target image whose determination by the classifier 4 is unstable by considering the change in the determination probability of each teacher image during the repetition of learning.

[0051] Preferably, the learning system 5 further includes a display control unit 54 that displays a target image on the display 35, an input receiving unit 55 that receives an input of the type of the target image, and an image group update unit 56 that updates the teacher image group by assigning a label of the type indicated by the input to the target image. Then, the learning unit 51 performs learning using the updated teacher image group to recreate the classifier 4. In this way, by recreating the classifier 4 using the teacher image group in which the type of the target image has been revised, a highly accurate classifier 4 can be obtained.

[0052] Here, a process of the comparative example is assumed. In the process of the comparative example, a teacher image whose type determined by the determiner 4 is different from the type of teacher label, whose determination probability by the determiner 4 for the type of teacher label is low, or whose determination probability by the determiner 4 is included in a predetermined range near the determination threshold is extracted as a target image. However, in all of these processes, since only one determination result by the determiner 4 (for example, the result in the final epoch) is used, the target image may not be properly extracted. Also, the target images may not be narrowed down (i.e., many target images are extracted), and it may take a long time to confirm the type of the target image.

[0053] In contrast, the learning system 5 makes it possible to narrow down and extract appropriate target images based on the change in the judgment probability of each teacher image during the attention period P1. This makes it possible to reduce the time required to confirm the type of target image, and realizes efficient correction of the learning data 521.

[0054] Preferably, the display control unit 54 displays information indicating a change in the judgment probability of the target image in learning using the updated teacher image group on the display 35. This allows the operator to easily determine the suitability of the current teacher label of the target image, and enables the generation of preferable learning data 521.

[0055] Preferably, multiple epochs of interest are set in the period of interest P1, and the image extraction unit 53 extracts target images based on changes in the judgment probability of each teacher image between the multiple epochs of interest. In this way, by reducing the number of epochs for which the judgment probability is referenced, the processing in the image extraction unit 53 can be simplified.

[0056] Preferably, the image extraction unit 53 extracts the target image based on the number of times the type determination result by the determiner 4 switches between two consecutive epochs of interest (the above-mentioned epoch pair of interest) among the multiple epochs of interest, and / or the fluctuation range of the determination probability between the two consecutive epochs of interest. This makes it possible to more reliably extract an appropriate target image.

[0057] 3 includes a step of creating a classifier 4 by performing learning for a set number of epochs using a group of teacher images (step S12), a step of acquiring a judgment probability when each teacher image is input to the classifier 4 being created during the learning for the set number of epochs during a period of attention P1 (step S13), and a step of extracting a target image from the group of teacher images based on a change in the judgment probability of each teacher image during the period of attention P1 (step S142). This allows the target image to be appropriately extracted from the group of teacher images.

[0058] Next, another example of the image extraction process in FIG. 4 will be described. In this example, the process surrounded by the dashed line in FIG. 4 (steps S143 to S145) is performed. In the image extraction process, as described above, the attention period P1, the judgment threshold T1, and the allowable fluctuation width are set (step S141), and the target image is extracted from the teacher image group (step S142). In this example, information about the extracted target image is displayed on the display 35 by the display control unit 54 (step S143). The information about the target image is, for example, the number of target images. The information about the target image may be a graph showing the change in the judgment probability of each target image over the course of an epoch (see FIG. 5), thumbnails of multiple target images, or the like.

[0059] Next, the worker checks the information on the target images displayed on the display 35, and decides whether or not to change the attention period P1. Whether or not to change the attention period P1 is determined based on the number of target images, changes in the determination probability of the target images, and the like. If it is not necessary to change the attention period P1, the image extraction process is completed (step S144), and the processes of steps S15 to S21 described above are performed. On the other hand, if the number of target images is excessively large, or if there is some tendency in the change in the determination probability of the target images, it is determined that it is necessary to change the attention period P1. In this case, an input for changing the attention period is made via the input unit 36 ​​(steps S144, S145). The input is accepted by the input accepting unit 55, and the changed attention period is set in the image extracting unit 53. In the example of FIG. 5, the changed attention period is indicated by an arrow P2.

[0060] In the image extraction unit 53, the target image is re-extracted from the teacher image group based on the change in the judgment probability of each teacher image in the changed attention period P2 (step S142). Furthermore, information on the re-extracted target image is displayed on the display 35 by the display control unit 54 (step S143). If the attention period P2 needs to be changed, the attention period is changed in the same manner as described above (steps S144, S145), and steps S142, S143 are performed. If the attention period P2 does not need to be changed, the image extraction process is completed (step S144), and the processes of steps S15 to S21 are performed.

[0061] As described above, in this processing example, after the image extraction unit 53 extracts the target image, the display control unit 54 displays information about the extracted target image on the display 35. When the operator who has confirmed the information inputs a change in the attention period and the input is accepted by the input acceptance unit 55, the image extraction unit 53 re-extracts the target image based on the change in the judgment probability of each teacher image in the attention period P2 indicated by the input of the change. This makes it possible to more appropriately extract the target image whose type should be confirmed, and to accurately correct the learning data 521. Depending on the design of the learning system 5, at least one of the judgment threshold, the allowable fluctuation range, and the allowable number of pairs may be changeable instead of or together with the attention period, and the target image may be re-extracted using the changed value. In addition, the interval between the attention epochs included in the attention period P1 may be changeable.

[0062] The above learning system 5 and learning method can be modified in various ways.

[0063] The types of defect images are not limited to true defects and pseudo defects, and may include detailed types of defects such as foreign matter adhesion, film peeling, etc. Furthermore, the types of defect images are not limited to two types, and may be three or more types. In this case as well, a group of teacher images, which are a plurality of defect images each labeled with one of a plurality of (e.g., three or more) types, is prepared, and the learning unit 51 performs learning using the group of teacher images, thereby creating the determiner 4.

[0064] When there are three or more types of defect images, the determiner 4 outputs, for example, a determination probability for each of the three or more types. For example, the image extraction unit 53 identifies, as a specific pair, an epoch pair of interest in which the fluctuation range of the determination probability of a specific type for each teacher image is larger than the allowable fluctuation range. An example of the specific type is the type taught for each teacher image (i.e., the type of teacher label). Then, the number of specific pairs is compared with the allowable number of pairs to extract a target image. As in the case where there are two types of defect images, the target image may be extracted based on the average fluctuation range of the determination probability, etc.

[0065] In the above embodiment, the judgment probability for all epochs for each teacher image (i.e., the judgment probability for the entire period in learning the set number of epochs) is stored in the storage unit 52 as judgment probability data 522, but in cases where the attention period is set in advance, only the judgment probability for the attention period may be stored. In this way, the storage unit 52 only needs to store the judgment probability of each teacher image acquired by the classifier 4 in the middle of creation at least during the attention period in learning the set number of epochs. On the other hand, in a processing example in which the process surrounded by the dashed line in FIG. 4 (steps S143 to S145) is performed, since the attention period is changed, it is preferable that the storage unit 52 stores the judgment probability of each teacher image for almost the entire period in learning the set number of epochs (for example, the period excluding the initial period of learning).

[0066] The object to be inspected by the inspection unit 20 may be a substrate such as a semiconductor substrate or a glass substrate, in addition to a printed circuit board. Furthermore, defects in objects other than substrates, such as machine parts, may be detected by the inspection unit 20. In this manner, the defect images handled by the learning system 5 may be images showing defects in various objects.

[0067] The configurations in the above-described embodiment and each of the modified examples may be combined as appropriate as long as they are not mutually inconsistent. [Explanation of symbols]

[0068] 4 Judgment device 5. Learning System 35 Display 51 Learning Department 52 Storage section 53 Image Extraction Section 54 Display control unit 55 Input reception section 56 Image Group Update Unit 521 Learning Data 522 Probability of Judgment Data P1, P2 attention period S11~S21, S141~S145 steps

Claims

1. A learning system for creating a classifier for classifying a type of a defect image showing a defect of an object, comprising: a learning unit that performs learning for a set number of epochs using a teacher image group that is a plurality of defect images to which a label of one of a plurality of types is assigned, thereby creating a classifier that is a trained model that classifies the type of the defect image; a storage unit that stores a judgment probability when each teacher image of the teacher image group is input to the classifier in the process of being created during a period of interest in learning the set number of epochs; an image extraction unit that extracts, from the group of teacher images, a target image that is a teacher image whose type should be confirmed based on a change in the determination probability of each teacher image during the attention period; A learning system comprising:

2. The learning system according to claim 1 , A learning system in which the defect images are classified into true defects and false defects.

3. 3. A learning system according to claim 1 or 2, A plurality of epochs of interest are set in the period of interest; A learning system in which the image extraction unit extracts the target image based on a change in the judgment probability of each of the teacher images between the multiple epochs of interest.

4. The learning system according to claim 3, A learning system in which the image extraction unit extracts the target image based on the number of times that the type judgment result by the classifier switches between two consecutive epochs of interest in the multiple epochs of interest, and / or the range of fluctuation of the judgment probability between the two consecutive epochs of interest.

5. 3. A learning system according to claim 1 or 2, a display control unit that displays the target image on a display unit; an input receiving unit that receives an input of a type of the target image; an image group update unit that updates the teacher image group by assigning a label of the type indicated by the input to the target image; Further equipped with The learning system includes a learning unit that performs learning using the updated teacher image group to recreate the classifier.

6. The learning system according to claim 5, A learning system in which the display control unit displays, on the display unit, information indicating a change in the judgment probability of the target image during learning using the updated group of teacher images.

7. 3. A learning system according to claim 1 or 2, an input receiving unit for receiving an input for changing the attention period; A display control unit that controls display of the display unit; Further equipped with the storage unit stores the judgment probability of each teacher image for almost the entire period of learning of the set number of epochs; the display control unit displays information about the target image extracted by the image extraction unit on the display unit; A learning system in which the image extraction unit re-extracts a target image based on a change in the judgment probability of each teacher image during the period of interest indicated by the input of the change.

8. A learning method for creating a classifier for classifying a type of a defect image showing a defect of an object, comprising: a) creating a classifier that is a trained model for classifying a type of a defect image by performing training for a set number of epochs using a teacher image group that is a plurality of defect images to which a label of any one of a plurality of types is assigned; b) acquiring a judgment probability when each teacher image of the teacher image group is input to the classifier in the process of being created during a period of interest in learning the set number of epochs; c) extracting, from the group of teacher images, a target image, which is a teacher image whose type should be confirmed, based on a change in the determination probability of each teacher image during the attention period; A learning method that provides:

9. The learning method according to claim 8, d) displaying the target image on a display unit; e) determining a type of the target image; f) updating the teacher image group by assigning the label of the type determined in the e) step to the target image; g) recreating the classifier by performing learning using the updated teacher image group; A learning method that provides:

Images