Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning technology configured to improve performance of a model constructed by machine learning, especially to determination accuracy on image data belonging to a class of low appearance frequency, by additional learning.SOLUTION: A processor (12) of a machine learning apparatus (1) executes: class inference processing (S1) of inferring a class to which each image data included in a dataset belongs, using a model constructed by machine learning; data selection processing (S2) of selecting image data of likelihood of inference equal to or higher than a threshold, out of image data inferred to belong to a specific class; labeling processing (S3) of putting a label indicating the specific class to image data determined by an examiner to belong to the specific class, out of the selected image data; and additional learning processing (S4) of executing additional learning of the model using a learning dataset including the image data with the label.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a machine learning device, a machine learning method, and a machine learning program for inputting image data and improving the performance of a model constructed by machine learning through additional learning. [Background technology]

[0002] There is known a method for inspecting an object using a model constructed by machine learning (hereinafter, also referred to as a trained model). For example, Patent Document 1 discloses an inspection device that inspects an object using a trained model that receives image data including an object as a subject and outputs the class of the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-042754 Summary of the Invention [Problem to be solved by the invention]

[0004] The trained model outputs one of a number of classes as the class to which the input image data belongs. However, some of these classes may have a very low frequency of occurrence. In this case, learning for image data belonging to such a class may be insufficient, and as a result, the accuracy of determining image data belonging to such a class may decrease.

[0005] In order to deal with such a problem, it is effective to perform additional learning using a learning dataset that contains a sufficient amount of image data belonging to a class with a low occurrence frequency. However, it is impossible or extremely difficult for an inspector to visually pick out image data belonging to a class with a low occurrence frequency from a dataset containing a huge number of image data. In addition, a method of picking out image data using a trained model is also possible, but since the trained model has low judgment accuracy for image data belonging to a class with a low occurrence frequency, this method also lacks effectiveness.

[0006] The present invention has been made in consideration of the above problems, and its purpose is to realize a machine learning technique that can improve the performance of a model constructed by machine learning, in particular, the judgment accuracy for image data belonging to a class that occurs infrequently, through additional learning. [Means for solving the problem]

[0007] The machine learning device according to aspect 1 of the present invention includes at least one processor, and the processor is characterized by executing a class estimation process that estimates a class to which each image data included in a dataset belongs using a model constructed by machine learning; a data selection process that selects from the dataset image data estimated to belong to a specific class in the class estimation process, image data whose likelihood of estimation is equal to or greater than a threshold; a labeling process that assigns a label indicating the specific class to image data selected in the data selection process that an inspector has determined to belong to the specific class; and an additional learning process that performs additional learning of the model using a learning dataset including image data to which a label has been assigned in the labeling process.

[0008] According to the above configuration, by performing the above processes with a class having a low frequency of occurrence as a specific class, it is possible to improve the accuracy of judgment for image data belonging to a class having a low frequency of occurrence. Moreover, according to the above configuration, the image data for which the inspector needs to visually judge the class is limited to the image data selected in the data selection process. Therefore, it is possible to reduce the burden on the inspector.

[0009] In the machine learning device of aspect 2 of the present invention, in addition to the configuration of the machine learning device of aspect 1, the processor is configured to repeatedly execute the class estimation process, the data selection process, the label assignment process, and the additional learning process.

[0010] According to the above configuration, it is possible to further improve the accuracy of determination for image data that belongs to a class with a low occurrence frequency.

[0011] A machine learning device according to aspect 3 of the present invention has a configuration in which, in addition to the configuration of the machine learning device according to aspect 1 or 2, it further includes a threshold setting process for setting the threshold.

[0012] According to the above configuration, the amount of image data selected in the data selection process, i.e., the amount of image data that needs to be visually determined by the inspector to determine the class, can be appropriately adjusted, and therefore the burden on the inspector can be appropriately adjusted.

[0013] The machine learning method according to aspect 4 of the present invention is characterized in that it includes a class estimation process in which at least one processor estimates a class to which each image data included in a dataset belongs using a model constructed by machine learning; a data selection process in which the processor selects from the dataset image data estimated to belong to a specific class in the class estimation process, the image data having a likelihood of the estimation equal to or greater than a threshold; a labeling process in which the processor assigns a label indicating the specific class to image data selected in the data selection process that an inspector has determined to belong to the specific class; and an additional learning process in which the processor performs additional learning of the model using a learning dataset including image data to which a label has been assigned in the labeling process.

[0014] According to the above configuration, by performing the above processes with a class having a low frequency of occurrence as a specific class, it is possible to improve the accuracy of judgment for image data belonging to a class having a low frequency of occurrence. Moreover, according to the above configuration, the image data for which the inspector needs to visually judge the class is limited to the image data selected in the data selection process. Therefore, it is possible to reduce the burden on the inspector.

[0015] The machine learning program of aspect 5 of the present invention is a program for operating a computer as a machine learning device of any of aspects 1 to 3, and is a program for causing a processor provided in the computer to execute each of the above processes.

[0016] According to the above configuration, the above machine learning device can be realized using a computer. Effect of the Invention

[0017] According to one aspect of the present invention, it is possible to realize a machine learning technique that can improve the performance of a model constructed by machine learning, in particular the judgment accuracy for image data belonging to a class that occurs infrequently, through additional learning. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a configuration of a machine learning device according to an embodiment of the present invention. [Diagram 2] 2 is a flowchart showing the flow of a machine learning method implemented by the machine learning device of FIG. 1. [Diagram 3] (a) is a diagram for explaining what kind of image data is selected in the data selection process when the threshold (the lower limit of the likelihood) is large, and (b) is a diagram for explaining what kind of image data is selected in the data selection process when the threshold (the lower limit of the likelihood) is small. [Figure 4] 1A is a diagram showing an example of image data belonging to class C1 (a class other than a specific class) in an application example (visual inspection of an LD chip) of the present invention, and FIG. 1B is a diagram showing an example of image data belonging to class C4 (a specific class) in the same application example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (Machine learning device configuration) The configuration of a machine learning device 1 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the machine learning device 1.

[0020] The machine learning device 1 is a device for implementing a machine learning method S for improving the performance of a model M constructed by machine learning through additional learning. As shown in Fig. 1, the machine learning device 1 includes a memory 11, a processor 12, a storage 13, and a bus 14. The memory 11, the processor 12, and the storage 13 are connected to each other via the bus 14.

[0021] The memory 11 is configured to record a machine learning program P for implementing the machine learning method S, and a model M constructed and evaluated in the machine learning method S. Note that as the memory 11, for example, a semiconductor RAM (Random Access Memory) or the like can be used.

[0022] The processor 12 is configured to execute a machine learning method S in accordance with a machine learning program P stored in the memory 11. As the processor 12, for example, a central processing unit (CPU), a graphic processing unit (GPU), a microprocessor, a digital signal processor, a microcontroller, an application specific integrated circuit (ASIC) such as a tensor processing unit (TPU), or a combination of these can be used.

[0023] The storage 13 is configured to store a dataset D referenced in the machine learning method S. As the storage 13, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these can be used.

[0024] Although the configuration in which the machine learning method S is executed by a single processor 12 provided in a single computer has been described here, the present invention is not limited to this. In other words, it is also possible to adopt a configuration in which the machine learning method S is executed jointly by multiple processors provided in a single computer or distributed across multiple computers.

[0025] Also, although the configuration in which the data set D is stored in a single storage 13 provided in a single computer has been described here, the present invention is not limited to this. In other words, it is also possible to adopt a configuration in which the data set D is stored in a single computer or in a distributed manner in multiple storages provided in multiple computers.

[0026] The machine learning program P for causing the processor 12 to execute the machine learning method S may be recorded in a computer-readable, non-transitory, tangible recording medium. This recording medium may be the memory 11, the storage 13, or another recording medium. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit may be used as the other recording medium.

[0027] (Machine learning methods) The flow of the machine learning method S performed by the machine learning device 1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the machine learning method S.

[0028] As described above, the machine learning method S is a method for improving the performance of the model M constructed by machine learning through additional learning. More specifically, it is a method for improving the performance of the model M so that the class can be correctly estimated even for image data belonging to a specific class that has a low occurrence frequency. As shown in Fig. 2, the machine learning method S includes a class estimation process S1, a data selection process S2, a label assignment process S3, and an additional learning process S4.

[0029] The dataset DS is a collection of image data. The model M is an algorithm that takes image data as input and outputs one of a number of classes, and is constructed by machine learning. The model M outputs the class to which the input image data is estimated to belong, along with the likelihood of the estimation. In this embodiment, a convolutional neural network (CNN) is used as the model M, but a neural network other than CNN, a logistic regression model, a support vector machine, a random forest, or the like may also be used as the model M.

[0030] The class estimation process S1 is a process for estimating the class to which each image data included in the data set D belongs, using a model M. In this embodiment, the class estimation process S1 is executed by the processor 12, and the estimated class is stored in the memory 11 together with the likelihood of the estimation.

[0031] The data selection process S2 is a process for selecting, from the data set DS, image data that has been estimated to belong to a specific class in the class estimation process S1 and whose likelihood of the estimation is equal to or greater than a predetermined threshold value α. In this embodiment, the data selection process S2 is executed by the processor 12, and the selected image data is stored in the memory 11.

[0032] The image data selected in the data selection process S2 is provided to an inspector, who visually determines the class to which each image data selected in the data selection process S2 belongs.

[0033] The labeling process S3 is a process of assigning a label indicating a specific class to image data that the inspector has determined to belong to a specific class among the image data selected in the data selection process S2. In this embodiment, the labeling process S3 is executed by the processor 12, and the image data to which the label has been assigned is stored in the memory 11.

[0034] The additional learning process S4 is a process of additionally learning the model M using a learning dataset DL including image data to which labels have been assigned in the labeling process S3. The learning dataset DL may include only image data to which labels have been assigned in the labeling process S3, or may include other image data. As an example of the latter, a set of image data selected in the data selection process S2 may be used as the learning dataset DL. In this embodiment, the additional learning process S4 is executed by the processor 12, and the model M after the additional learning is stored in the memory 11.

[0035] The processor 12 may repeatedly execute the class estimation process S1, the data selection process S2, the labeling process S3, and the additional learning process S4. This allows the performance of the model for image data belonging to a specific class to be gradually improved while gradually increasing the amount of image data belonging to the specific class included in the learning dataset DL. The threshold value α described above may be kept constant or gradually increased when these processes are repeated.

[0036] The termination conditions for repeating these processes are arbitrary. Examples include (1) when the determination accuracy of the model M for image data belonging to a specific class reaches a predetermined determination accuracy, (2) when the determination accuracy of the model M for any image data reaches a predetermined determination accuracy, and (3) when the number of repetitions reaches a predetermined number.

[0037] (Adjusting the threshold value α) The adjustment of the threshold value α will be described with reference to FIG. 3. In FIG. 3, image data is shown projected onto a two-dimensional plane. Triangle marks in FIG. 3 correspond to image data belonging to a specific class, and square marks in FIG. 3 correspond to image data belonging to a class other than the specific class. Curve γ in FIG. 3 represents the watershed of whether or not data is selected in the data selection process S2. Image data located to the lower left of curve γ is selected as image data belonging to the specific class in the data selection process S2. Image data located to the upper right of curve γ is not selected as image data belonging to the specific class in the data selection process S2.

[0038] When the threshold value α is increased, the number of image data selected in the data selection process S2 is reduced, as shown in (a) of Fig. 3. As a result, the burden on the inspector to visually determine the class to which each image data selected in the data selection process S2 belongs is reduced. However, the number of image data belonging to a specific class that can be incorporated into the learning dataset DL is reduced, so the effect of additional learning is reduced.

[0039] Conversely, if the threshold value α is made smaller, the number of image data selected in the data selection process S2 increases, as shown in (b) of Fig. 3. As a result, the number of image data belonging to a specific class that can be incorporated into the learning data set DL increases, and the effect of additional learning increases. However, the burden on the inspector who visually determines the class to which each image data item selected in the data selection process S2 belongs increases.

[0040] For this reason, it is preferable to set the above-mentioned threshold value α to an optimal value taking into consideration both the effect of additional learning and the burden on the examiner. For this reason, it is preferable that the machine learning method S further includes a threshold setting process for setting the above-mentioned threshold value α. The threshold value α set in this threshold setting process may be a value specified by the user, or may be a value derived by the machine learning device 1.

[0041] (Examples) An application example of the machine learning method S will be described with reference to FIG. 4. The model M to be applied in this application example is a trained model used for visual inspection of LD chips. The input of the model M is image data including an LD chip as a subject, and the output of the model M is one of five classes C1, C2, C3, C4, and C5 that represent the state of the LD chip. An example of image data belonging to class C1 is shown in FIG. 4(a). As shown in FIG. 4(a), class C1 is a class corresponding to a normal LD ​​chip. Also, an example of image data belonging to class C5 is shown in FIG. 4(b). As shown in FIG. 4(b), class C4 is a class corresponding to a cracked LD chip. Class C5 is a class with an extremely low occurrence frequency that corresponds to the above-mentioned specific class.

[0042] The given model M had low judgment accuracy for image data belonging to class C5, and additional learning was necessary. For this reason, a data set D consisting of 500,000 pieces of image data including LD chips as subjects was prepared. It is impossible or extremely difficult for an inspector to visually pick out image data belonging to class C5 from 500,000 pieces of image data. Therefore, additional learning of model M was performed using machine learning method S.

[0043] In the first class estimation process S1, a given model M was used. In the first data selection process S2, the threshold value α was set to 10%. As a result, 146 pieces of image data were selected in the first data selection process S2. In the first labeling process S3, of these 146 pieces of image data, 20 pieces of image data that were visually confirmed by an inspector to belong to class C5 were given a label indicating class C5. In the first additional learning process S4, additional learning of the model M was performed using a learning dataset DL including these 20 pieces of image data. As a result, the judgment accuracy of the model M for image data belonging to class C5 was 46%.

[0044] In the second class estimation process S1, the model M constructed in the first additional learning process S4 was used. In the second data selection process S2, the threshold α was set to 10% again. As a result, 72 pieces of image data were selected in the second data selection process S2. In the second labeling process S3, of these 72 pieces of image data, 41 pieces of image data that were visually confirmed by the inspector to belong to class C5 were given a label indicating class C5. In the second additional learning process S4, additional learning of the model M was performed using a learning dataset DL including these 41 pieces of image data. As a result, the judgment accuracy of the model M for image data belonging to class C5 was 85%.

[0045] From the above, it was confirmed that the machine learning method S can improve the judgment accuracy of the model M for image data belonging to a class with a low occurrence frequency. If such processing is further repeated, it is expected that the judgment accuracy of the model M for image data belonging to a class with a low occurrence frequency will be further improved. In addition, the number of image data on which the inspector performed visual class discrimination for the first labeling processing S3 was 146, and the number of image data on which the inspector performed visual class discrimination for the second labeling processing S3 was 72. Therefore, it was confirmed that the performance improvement of the model M described above can be realized without placing an excessive burden on the inspector.

[0046] (Additional Notes) The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0047] 1 Machine learning device 11. Memory 12 processors 13. Storage 14 Bus S Machine Learning Methods S1 Class estimation process S2 Data sorting and processing S3 labeling process S4 Additional learning process D Dataset DL training dataset

Claims

1. At least one processor; The processor executes a class estimation process for estimating a class to which each image data included in a data set belongs using a model constructed by machine learning; a data selection process for selecting, from the data set, image data estimated to belong to a specific class in the class estimation process, image data having a likelihood of the estimation equal to or greater than a threshold; a labeling process for assigning a label indicating the specific class to image data determined by an inspector to belong to the specific class from the image data selected in the data selection process; and an additional learning process for performing additional learning of the model using a learning data set including image data to which a label has been assigned in the labeling process, The specific class is a class that is preselected as a class that occurs less frequently. A machine learning device characterized by:

2. The processor repeatedly executes the class estimation process, the data selection process, the label assignment process, and the additional learning process. The machine learning device according to claim 1 .

3. The method further includes a threshold setting process for setting the threshold. The machine learning device according to claim 1 or 2.

4. A class estimation process in which at least one processor estimates a class to which each image data included in the dataset belongs using a model constructed by machine learning; a data selection process in which the processor selects, from the data set, image data that is estimated to belong to a specific class in the class estimation process and whose likelihood of the estimation is equal to or greater than a threshold; a labeling process in which the processor assigns a label indicating the specific class to image data that an inspector has determined to belong to the specific class, from among the image data selected in the data selection process; The processor includes an additional learning process for additionally learning the model using a learning dataset including image data to which labels have been added in the labeling process, The specific class is a class that is preselected as a class that occurs less frequently. A machine learning method comprising:

5. A machine learning program for operating a computer as the machine learning device described in any one of claims 1 to 3, the machine learning program causing a processor provided in the computer to execute each of the processes.

Images