Machine learning program, device, and method

Abstract

A machine learning device according to the present invention (A) uses, for each image included in an image data set, which is operation data, a machine learning model 20 to acquire classification results in which each pixel is classified, and determines a classification result of an image in which the average value of classification scores indicative of the confidence level of the classification result is equal to or more than a threshold to be "good" and determines a classification result of an image in which the average value is less than the threshold to be "bad", (B) uses, from among the images in which the classification results are "good", the classification results of images that were captured at the same image capturing location and in the same image capturing direction, so as to generate a synthetic pseudolabel, (C) generates expanded images by expanding the images of the operation data, (D) labels each pixel of the images having the "good" classification results with the classification result of said pixel and uses the synthetic pseudolabel to label the images having the "bad" classification results and the expanded images, so as to generate training data, and (E) uses the generated training data to train the machine learning model 20.

Description

[Technical field]

[0001] The disclosed technology relates to a machine learning program, a machine learning device, and a machine learning method. [Background technology]

[0002] In recent years, machine learning models have been increasingly introduced into processes such as data judgment and classification executed in systems used by companies and the like. Machine learning models judge and classify data based on training data used during training at the time of system development. Therefore, if the trend of operational data used during system operation changes from the trend of training data, the judgment accuracy, classification accuracy, etc. of the machine learning model will decrease. In order to maintain the accuracy of the machine learning model during system operation, a value indicating accuracy such as the accuracy rate is calculated by manually checking the accuracy of the output result of the machine learning model on a regular basis, that is, by a human being. Then, when the value decreases, the system trains the machine learning model using training data that has been manually checked for accuracy and labeled with a correct answer.

[0003] Furthermore, as a technique for judging and classifying data using a machine learning model, there is a technique called semantic segmentation, which classifies the type of object into classes for each small region, such as a pixel unit, of an image, thereby dividing an image into regions for each type of object. In the task of semantic segmentation, as in the above, the accuracy of the machine learning model may decrease due to changes in the operation data during the operation of a system using the machine learning model. In response to this, a technique has been proposed in which operation data after changes during system operation are assumed and prepared in advance, and training data including the operation data after the changes is used to train the machine learning model used in the system. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang, "Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training", Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 289-305. [Non-Patent Document 2] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos, "Bidirectional Learning for Domain Adaptation of Semantic Segmentation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6929-6938. Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, in the task of semantic segmentation, class classification is performed for each small region such as a pixel unit. Therefore, if correct labels are to be assigned to the operational data during system operation, the operational costs will be enormous. In addition, if it is unclear how the operational data will change during system operation, it is difficult to prepare the operational data after the change in advance and train a machine learning model.

[0006] In one aspect, the disclosed technology aims to maintain the accuracy of machine learning models in the task of semantic segmentation. [Means for solving the problem]

[0007] In one embodiment, the disclosed technology classifies a first image based on a value less than a threshold value. In this case, the disclosed technology labels a first region with a classification result of a second region of the second image corresponding to a position of the first region of the first image based on a classification result of the machine learning model classifying a second image based on a value equal to or greater than the threshold value. In this way, the disclosed technology generates training data and trains the machine learning model based on the training data. Effect of the Invention

[0008] One aspect is that it has the effect of maintaining the accuracy of machine learning models in semantic segmentation tasks. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining a decrease in accuracy of a machine learning model. [Diagram 2] FIG. 1 is a diagram for explaining semantic segmentation. [Diagram 3] FIG. 1 is a diagram for explaining the degradation of accuracy of a machine learning model in a semantic segmentation task. [Figure 4] FIG. 1 is a functional block diagram of a machine learning device. [Diagram 5] FIG. 2 is a diagram for explaining each process of the machine learning device. [Figure 6] FIG. 13 is a diagram for explaining generation of a composite pseudo label. [Figure 7] FIG. 13 is a diagram for explaining labeling of an image with a classification result of “bad.” [Figure 8] This is a graph showing the progress of accuracy of a machine learning model in operation. [Figure 9] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as a machine learning device. [Figure 10] 1 is a flowchart illustrating an example of a machine learning process. [Figure 11]13A and 13B are schematic diagrams showing example images and classification results when a situation change occurs. [Figure 12] 13A and 13B are schematic diagrams showing example images and classification results when a situation change occurs. [Figure 13] FIG. 13 is a diagram illustrating generation of training data in an application example. [Figure 14] 1 is a schematic diagram of an example of an image, classification results, and accuracy in an application example. [Figure 15] 1 is a schematic diagram of an example of an image, classification results, and accuracy in an application example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Before describing the details of the embodiment, a decrease in accuracy of the machine learning model during system operation will be described.

[0011] For example, in training a machine learning model used in an image classification system that estimates subjects appearing in an image, image features that are useful for classification are trained from images that are training data. However, the features of images input to the system during operation may change from the features of images used during training of the machine learning model. Causes of this include, for example, the surface of the camera that captures the images becoming dirty, the camera being misaligned, or the sensitivity deteriorating. Such changes in the features of images acquired during operation cause a decrease in the accuracy of the machine learning model. For example, while a machine learning model had an accuracy of 99% at the beginning of operation, after a certain period of time has passed since the start of operation, a decrease in accuracy occurs such that the model can only produce an accuracy of 60%.

[0012] The cause of such a decrease in accuracy will be explained. Figure 1 shows a schematic diagram of the boundary plane for each label and the feature values ​​extracted from each image projected onto the feature space. As shown in the left diagram of Figure 1, immediately after training of the machine learning model, in the feature space, the feature values ​​are clearly separated for each label at the boundary plane. Then, when a change occurs in the features of the acquired image, as shown in the right diagram of Figure 1, the feature values ​​extracted from the image move to areas of different labels (dashed line in Figure 1), or areas of multiple labels are connected (dotted line in Figure 1). This makes it easier for the machine learning model to make erroneous classification results, resulting in a decrease in accuracy.

[0013] Here, the distribution of features in the feature space has a characteristic that the distribution of features of the same label has one or more points with high density, and the density often becomes thinner toward the outside of the distribution. Therefore, the following reference method is considered to utilize this characteristic to automatically label images, which are operational data. The reference method calculates the density for each cluster of features of each label in the feature space before the accuracy is reduced, and records the number of clusters. The reference method also records the center of an area in each cluster where the density is equal to or higher than a certain level, or the point with the highest density, as the cluster center. Then, the reference method calculates the density of the features of the images, which are operational data, for each point in the feature space after operation. The reference method extracts features included in an area in the feature space where the density is equal to or higher than a threshold as a cluster. Then, the reference method searches for a minimum threshold value that changes the threshold value so that the number of extracted clusters becomes the number of clusters recorded before the accuracy is reduced. The reference method matches the cluster center of each cluster clustered at the minimum threshold value with the cluster center recorded before the accuracy is reduced. The reference method then assigns a label corresponding to the cluster before the accuracy degradation to an image corresponding to the feature included in the matched cluster. In this way, images of the operational data are labeled. The reference method trains a machine learning model using the labeled operational data, thereby suppressing the degradation of the accuracy of the machine learning model during operation.

[0014] Here, we consider the task of semantic segmentation. As shown in FIG. 2, semantic segmentation is a technology that inputs an input image to a machine learning model, classifies the type of object for each small region of the image such as a pixel unit, and outputs a classification result in which the image is divided into regions for each type of object. As shown in FIG. 3, in the task of semantic segmentation, as in the above image classification problem, the accuracy of the machine learning model decreases due to the passage of time during operation and changes in conditions such as weather. FIG. 3 shows an example in which an image taken at night is input during operation in a system using a machine learning model trained using images taken outdoors during the day as training data. For example, the accuracy of the machine learning model decreases due to brightness changes between daytime images and nighttime images, and reflections of outdoor lights that are not present in daytime images (dashed line area in FIG. 3), etc.

[0015] The above-mentioned reference method can be applied to address the deterioration of accuracy during operation in such semantic segmentation tasks. However, in semantic segmentation, class classification is performed in units of small regions such as each pixel in an image, so the number of instances handled during operation becomes enormous, making clustering like the reference method difficult. For example, when processing 100 images of 320 pixels x 240 pixels in each batch, the number of instances to be clustered is 100 in the case of an image classification problem. In contrast, in the case of a semantic segmentation problem, the number is 320 x 240 x 100 = 7,680,000.

[0016] Therefore, in this embodiment, without using clustering as in the reference method, appropriate labeling is performed by following changes in operation data during operation. The machine learning device according to this embodiment will be described in detail below. Note that in the following embodiment, a semantic segmentation problem in which class classification is performed for each pixel of an image will be described as an example.

[0017] As shown in Fig. 4, a data set of images is input as operational data to the machine learning device 10. Functionally, the machine learning device 10 includes a determination unit 11, a generation unit 12, and a training unit 16. The generation unit 12 further includes a label generation unit 13, an augmented image generation unit 14, and a training data generation unit 15. In addition, a machine learning model 20 is stored in a predetermined storage area of ​​the machine learning device 10.

[0018] The machine learning model 20 is a machine learning model used to execute a semantic segmentation task in an operational system. The machine learning model 20 is configured, for example, by a deep neural network (DNN).

[0019] As shown in A of FIG. 5, the determination unit 11 acquires a data set of images, which are operational data input to the machine learning device 10. For each of the acquired images, the determination unit 11 acquires a classification result in which each pixel is classified into a class using the machine learning model 20. Then, the determination unit 11 judges whether the classification result for each image is good or bad. Specifically, the determination unit 11 calculates a classification score indicating the degree of certainty of the classification result together with the classification result. For example, when the machine learning model 20 is a DNN, the classification score may be a score based on the output value of the layer one layer before the final layer, that is, the value before the application of the softmax function.

[0020] More specifically, in the case of a semantic segmentation problem in which N classes are classified, for pixel (k, l) of image x_i, a classification score vector v obtained from the machine learning model 20 is (x_i,k,l) is expressed by the following formula (1). In this case, the classification score S (x_i,k,l) can be expressed as the following equation (2). v (x_i,k,l) =[s (x_i,k,l,1) , ,s (x_i,k,l,N) ] (1) S (x_i,k,l) =arg max s (s (x_i,k,l,1) , ,s (x_i,k,l,N) ) (2) However, s (x_i,k,l,n) (n=1,...,N) is the probability that pixel (k,l) of image x_i is of class n.

[0021] The determination unit 11 calculates the average value of the classification scores for all pixels of the image. If the average value is equal to or greater than the threshold, the determination unit 11 determines the classification result of the image as "good", and if the average value is less than the threshold, the determination unit 11 determines the classification result of the image as "bad". This makes it possible to determine the deterioration of the accuracy of the machine learning model 20 during operation without teacher data. Note that an image with a classification result of "bad" is an example of a "first image" of the disclosed technology, and an image with a classification result of "good" is an example of a "second image" of the disclosed technology.

[0022] The generation unit 12 generates training data for retraining the machine learning model 20. Below, each of the label generation unit 13, the augmented image generation unit 14, and the training data generation unit 15 will be described in detail.

[0023] The label generating unit 13 generates a synthetic pseudo label by using the classification results of images captured in the same shooting location and direction among the images with the classification result of "good" as shown in B of Fig. 5. Specifically, as shown in Fig. 6, the label generating unit 13 generates a synthetic pseudo label by using the classification results of images captured in the same shooting location and direction among the images with the classification result of "good". W The classification score vector v for pixel (k,l) of each image x_i in (x_i,k,l) Using the above, we obtain a synthetic pseudo label c for pixel (k, l) as shown in the following equation (3). (k,l) Generate.

[0024]

number

[0025] That is, the label generation unit 13 generates a label for an image x_i∈X W For each pixel (k, l) in (k,l) Generate it as:

[0026] The extended image generating unit 14 generates an extended image by extending the image of the operational data, as shown in C of Fig. 5. A conventionally known method may be adopted as a method for generating an extended image. For example, the extended image generating unit 14 may generate an extended image by alpha blending an image classified as "good" and an image classified as "poor." When synthesizing two or more images to generate an extended image, the extended image generating unit 14 uses images taken at the same shooting location and in the same shooting direction.

[0027] As shown in D of FIG. 5, the training data generation unit 15 generates training data by labeling each pixel with the classification result of the pixel for an image classified as "good". In addition, the training data generation unit 15 generates training data by labeling each image classified as "bad" and the augmented image with a synthetic pseudo label. Specifically, as shown in FIG. 7, the training data generation unit 15 generates training data by labeling pixel (k, l) of an image classified as "bad" with a synthetic pseudo label c generated from an image classified as "good" that was photographed at the same photographing location and in the same photographing direction as the image. (k,l) Similarly, for the extended image, the training data generating unit 15 assigns a synthetic pseudo label c (k,l) is granted.

[0028] As shown in E of Fig. 5, the training unit 16 trains the machine learning model 20 using the training data generated by the generation unit 12. That is, the training unit 16 retrains the machine learning model 20 using training data in which the classification result by the machine learning model 20 currently in operation is labeled as a correct answer label for operation data acquired during operation. The retrained machine learning model 20 is output and applied to the system in operation.

[0029] FIG. 8 shows a schematic diagram of the relationship between the elapsed time during operation and the accuracy of the machine learning model. In the example of FIG. 8, the solid line shows the progress of the accuracy when the classification result obtained during operation is appropriate, and the dashed line shows the progress of the accuracy when the classification result obtained during operation is inappropriate. In this way, when the classification result for the operation data is significantly different from the true classification result, the accuracy may not be maintained even if the model is retrained using training data labeled with the classification result as a correct answer label, or the accuracy may be reduced by retraining. In this embodiment, after determining whether the classification result for the operation data is good or bad, a label based on the classification result of an image with a classification result of "good" is assigned to an image with a classification result of "bad". Therefore, as in the example shown by the solid line in FIG. 8, the accuracy reduction of the machine learning model during operation can be suppressed.

[0030] The machine learning device 10 may be realized by, for example, a computer 40 shown in FIG. 9. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input / output device 44 such as an input unit and a display unit, and an R / W (Read / Write) unit 45 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 46 that is connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to one another via a bus 47.

[0031] The storage unit 43 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 43 as a storage medium stores a machine learning program 50 for causing the computer 40 to function as the machine learning device 10. The machine learning program 50 has a determination process 51, a generation process 52, and a training process 56. The storage unit 43 also has an information storage area 60 in which information constituting the machine learning model 20 is stored.

[0032] The CPU 41 reads out the machine learning program 50 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the machine learning program 50. The CPU 41 operates as the judgment unit 11 shown in FIG. 4 by executing the judgment process 51. The CPU 41 also operates as the generation unit 12 shown in FIG. 4 by executing the generation process 52. The CPU 41 also operates as the training unit 16 shown in FIG. 4 by executing the training process 56. The CPU 41 also reads out information from the information storage area 60 and expands the machine learning model 20 in the memory 42. As a result, the computer 40 that has executed the machine learning program 50 functions as the machine learning device 10. The CPU 41 that executes the program is hardware.

[0033] The functions realized by the machine learning program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like.

[0034] Next, the operation of the machine learning device 10 according to the present embodiment will be described. A machine learning model 20 used in a system in operation is stored in the machine learning device 10, and a data set of images, which is operation data, is input to the machine learning device 10. Then, when an instruction is given to re-learn the machine learning model 20, the machine learning device 10 executes the machine learning process shown in Fig. 10. Note that the machine learning process is an example of a machine learning method of the disclosed technology.

[0035] In step S11, the determination unit 11 acquires a data set of images, which are operational data input to the machine learning device 10. Then, for each of the acquired images, the determination unit 11 acquires a classification result in which each pixel is classified into a class using the machine learning model 20. Next, in step S12, the determination unit 11 calculates the average value of the classification scores, which indicate the confidence of the classification result of each pixel, for all pixels of the image, and determines the classification result of an image whose average value is equal to or greater than a threshold value as "good" and the classification result of an image whose average value is less than the threshold value as "poor."

[0036] Next, in step S13, the label generation unit 13 generates a composite pseudo label using the classification results of images taken at the same shooting location and direction among the images classified as "good". Next, in step S14, the extended image generation unit 14 generates an extended image by extending the image of the operational data. Next, in step S16, the training data generation unit 15 generates training data by labeling each pixel of the image classified as "good" with the classification result of that pixel. In addition, the training data generation unit 15 generates training data by labeling each of the images classified as "bad" and the extended image with a composite pseudo label.

[0037] Next, in step S17, the training unit 16 trains the machine learning model 20 using the training data generated by the generation unit 12. Then, the machine learning process ends.

[0038] As described above, the machine learning device according to the present embodiment judges whether a classification result is good or bad based on the classification score of the classification result when the machine learning model performs semantic segmentation on an image, which is operational data. In addition, the machine learning device generates training data in which each pixel of an image whose classification result is judged to be "bad" is labeled with the classification result of each pixel of the image whose classification result is "good" corresponding to that pixel, and trains the machine learning model based on the generated training data. This makes it possible to maintain the accuracy of the machine learning model while suppressing operational costs in the task of semantic segmentation.

[0039] Here, an application example will be described in which the machine learning model trained by the machine learning device according to this embodiment is applied to a system that detects flooding of a river. The task of this application example is to perform semantic segmentation on an image of a river, and to determine whether or not the river is flooded based on an area classified as a river (water surface). In this application example, a data set of images taken at intervals of 10 to 20 minutes for four days at each of 8 non-flooded locations and 7 flooded locations out of 15 shooting locations is used as operational data, and the results of the verification will be described. In addition, as a verification condition, CPNet (Reference 1) was used as the initial machine learning model.

[0040] Reference 1: C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, N. Sang, "Context Prior for Scene Segmentation," IEEE Conference on Computer Vision and Pattern Recognition, pp. 12416-12425, 2020.

[0041] In addition, the method of generating the augmented images was grayscaling, flipping, and random erasing. In addition, every two hours, the machine learning model was trained and re-learned by fine tuning using images (approximately 150 to 250 images) from the previous four hours and a part of the training data (300 images) used in training the initial machine learning model. In addition, in the fine tuning, the initial value of the learning rate was set to 0.00001 and the number of epochs was set to 500. For reference, the time required for the above fine tuning is just under 10 minutes with one GPU. On the other hand, when training the initial machine learning model, it takes about five hours if the initial value of the learning rate is set to 0.001 and the number of epochs is set to 20,000.

[0042] For images taken between 18:00 and 22:00, the average classification score of images determined to be "good" was 0.946, and the average classification score of images determined to be "bad" was 0.871. Figures 11 and 12 show schematic diagrams of example images and classification results when images taken at the same shooting location and direction are taken at different shooting times, that is, when there is a change in the situation between the two images. The upper part of Figure 11 is an example of an image taken around 18:00 when it is still bright, and the classification score of the classification result is 0.959, which is determined to be "good". On the other hand, the lower part of Figure 11 is an example of an image taken when it is dark after sunset, and the classification score of the classification result is 0.885, which is determined to be "bad". Figure 12 also shows an example of an image with a similar change in the situation, and the image example in the upper part of Figure 12 has a classification score of 0.973, which is determined to be "good". On the other hand, the image example in the lower part of Figure 12 had a classification score of 0.885, and was judged to be "bad." In this way, the classification score decreases with the change in the situation, and it is possible to detect a decrease in the accuracy of the machine learning model without using the correct answer label.

[0043] In the application example, as shown in FIG. 13, a synthetic pseudo label was generated from the classification result determined to be "good", and the generated synthetic pseudo label was labeled to the image and the augmented image whose classification result was determined to be "bad" to generate training data. FIG. 14 and FIG. 15 show an example of an image example, classification result, and accuracy in this case. FIG. 14 shows an example of an image taken in a time period when it is still bright around 18:00, and FIG. 15 shows an example of an image taken at night. The accuracy represents the average correct answer rate of the classification result of the class "water surface". As shown in FIG. 14, for images taken in a time period when it is bright, both the classification result by the machine learning model before re-learning and the classification result by the machine learning model after re-learning in the application example maintain high accuracy. Also, as shown in FIG. 15, in the example of an image taken at night when a change in situation occurs, the classification result by the machine learning model before re-learning significantly decreases in accuracy. In contrast, the classification result by the machine learning model after re-learning in the application example maintains high accuracy. In other words, this application example makes it possible to maintain the accuracy of the machine learning model without incurring operational costs such as manually assigning correct labels, even when the situation changes during operation.

[0044] In the above embodiment, the semantic segmentation is described as classifying each pixel of an image, but the class classification is not limited to pixel-by-pixel. For example, the class classification may be performed in units of small regions such as 2 pixels x 2 pixels, 3 pixels x 3 pixels, etc.

[0045] In the above embodiment, the process of generating synthetic pseudo labels and the process of labeling are described with respect to images taken at the same shooting location and direction, but the present invention is not limited to this. Even if the images are taken at different shooting locations and directions, it is sufficient that the positions on the images corresponding to the same point can be matched to each other.

[0046] In the above embodiment, the case where the classification result is judged on an image-by-image basis has been described, but the present invention is not limited thereto. The machine learning device may judge the quality of each class classification unit. In this case, there are areas with a classification result of "good" and areas with a classification result of "bad" in one image. In this case, the machine learning device generates a synthetic pseudo label for each area with a classification result of "good" instead of generating it on an image-by-image basis. Then, the machine learning device may assign a synthetic pseudo label generated from an area with a classification result of "good" to an area with a classification result of "bad" in each image, the synthetic pseudo label corresponding to the position of the area. In addition, the machine learning device may assign the classification result of the area to an area with a classification result of "good" in each image as a label.

[0047] In the above embodiment, the machine learning program is stored (installed) in the storage unit in advance, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory. [Explanation of symbols]

[0048] 10 Machine Learning Device 11 Judgment section 12 Generation part 13 Label Generation Unit 14. Extended Image Generation Unit 15 Training data generation unit 16 Training Department 20 Machine Learning Models 40 Computer 41 CPU 42 Memory 43 Storage section 44 I / O Devices 45 R / W section 46 Communication I / F 47 Bus 49 Storage medium 50 Machine Learning Programs 51 Judgment Process 52 Generation Process 56 Training Process 60 Intelligence Memory Field

Claims

1. When the machine learning model classifies the first image based on a classification score indicating the confidence of the classification result being less than a threshold, the machine learning model generates training data in which the classification result of a second region of the second image corresponding to the same point as the first region of the first image is labeled for the first region based on a classification result in which the machine learning model classifies a second image based on a classification score being equal to or greater than the threshold; training the machine learning model based on the training data; A machine learning program that causes a computer to execute processing.

2. When the machine learning model classifies the first image based on a value of the classification score that is less than a threshold, the average of the classification scores output when each of a plurality of regions including the first region in the first image is classified is less than the threshold. The machine learning program according to claim 1 .

3. When the machine learning model classifies the first image based on a value in which the classification score is less than a threshold, the classification score output when the first region of the first image is classified is less than the threshold, The classification result in which the machine learning model classifies the second image based on the classification score being equal to or greater than the threshold is a classification result of the second region of the second image obtained by inputting the second image to the machine learning model, the classification result being obtained by classifying the second region of the second image based on the classification score being equal to or greater than the threshold. The machine learning program according to claim 1 .

4. the process of generating the training data includes a process of generating the training data in which, when the third region of the first image is classified with the classification score being equal to or greater than the threshold, the third region of the first image is labeled with the classification result of the third region. The machine learning program according to any one of claims 1 to 3.

5. the process of generating the training data includes a process of generating training data in which a classification result of a second region of the second image corresponding to the same point as a fourth region of a third image generated using at least one of the first image and the second image is labeled for the fourth region; The machine learning program according to any one of claims 1 to 4.

6. the classification result of the second region is a probability that the second region is classified into each of a plurality of classes; The labeling process includes assigning, to the first region, a label corresponding to a class into which the second region is most likely to be classified, based on classification results of the second regions of the second images. The machine learning program according to any one of claims 1 to 5.

7. When the machine learning model classifies the first image based on a classification score indicating the confidence of the classification result being less than a threshold, the machine learning model generates training data in which the classification result of a second region of the second image corresponding to the same point as the first region of the first image is labeled for the first region based on a classification result in which the machine learning model classifies a second image based on a classification score being equal to or greater than the threshold; training the machine learning model based on the training data; A machine learning device comprising a control unit that executes processing.

8. When the machine learning model classifies the first image based on a classification score indicating the confidence of the classification result being less than a threshold, the machine learning model generates training data in which the classification result of a second region of the second image corresponding to the same point as the first region of the first image is labeled for the first region based on a classification result in which the machine learning model classifies a second image based on a classification score being equal to or greater than the threshold; training the machine learning model based on the training data; A machine learning method comprising the steps of: causing a computer to execute a process.

Images