Computer-readable recording medium, machine learning device, and information

Abstract

To provide a machine-learning program, a machine-learning device, and an information-processing system that enable stable AI operation.SOLUTION: An inference result obtained by inputting operation data to a learned machine learning model trained based on learning data is collected, density-based clustering is executed on the collected inference result to generate a cluster, an estimated label corresponding to the cluster is estimated for each cluster from correct labels corresponding to all correct answers that can be the inference result, and the learned machine learning model is fine-tuned based on the operation data belonging to the cluster and the estimated label corresponding to the cluster.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a machine learning program, a machine learning device, and an information processing system. [Background technology]

[0002] Business operations using machine learning are carried out using the following procedure. A trained machine learning model to be used in business is generated by repeatedly performing machine learning on a pre-trained machine learning model using training data. Training data is also called teacher data. By inputting operational data into the generated trained machine learning model, the machine learning model outputs the prediction results required for business.

[0003] However, as machine learning models continue to be used in business, the external environment changes over time, and the trends in the operational data input to the machine learning model may change to differ from the trends in the training data used to train the machine learning model. As a result, problems arise such as a deterioration in the inference accuracy of the machine learning model due to differences between the training data used when the machine learning model was developed and the operational data, or changes in the statistical trends of the operational data input during operation. Therefore, technologies are being developed to address this deterioration in the inference accuracy of machine learning models.

[0004] For example, a technology has been proposed that attempts to automatically restore the accuracy of a machine learning model based on operational data input during operation. This technology attempts automatic accuracy recovery using the following procedure. The operational data input during operation is represented in a data space. The operational data represented in the data space is separated by a boundary called a decision boundary using a machine learning model. Next, the operational data represented in the data space is represented in a feature space, which is a mathematical space that represents the characteristics of data distribution as data groups. The data groups formed by the operational data in the feature space are captured as shapes, and their changes are tracked. The classification results of the operational data in the feature space are then labeled as pseudo labels for the operational data represented in the data space. The machine learning model is then re-trained using the labeled operational data, thereby automatically restoring the accuracy. The pseudo labels are, for example, correct labels assigned by estimation to unlabeled data.

[0005] In such automatic recovery techniques using feature space, for example, weights other than those in the BN (Batch Normalization) and Fully Connected (FC) layers are fixed, and the BN layer and FC layer are tuned in sequence. In this case, tuning is performed by minimizing entropy using a loss function, which is a loss function. By performing retraining before each cluster crosses the decision boundary through data tracking, it is ensured that the shape of the data group in the feature space is not disrupted as much as possible. And by ensuring the shape of the data group in the feature space, the inference accuracy of the machine learning model is maintained.

[0006] The following techniques exist for updating machine learning models. For example, a technique has been proposed in which output data is clustered based on similarity using one or more of a distance measure based on the spherical distance between images, a likelihood based on the maximum a posteriori probability, or a distance measure based on the spherical distance between centers of gravity, and then associated with tags for learning. Another technique has been proposed in which target parameters are determined based on the difference in data distribution between an inference dataset and a training dataset, and labeled data appropriate for the current inference data distribution is obtained from existing labeled data according to the parameters to update an existing AI model. Another technique has been proposed in which a machine learning model is updated using dynamically updated training data when significant movement is detected using a histogram. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-125340 [Patent Document 2] Special Publication No. 2023-535227 [Patent Document 3] U.S. Patent Application Publication No. 2021 / 0390455 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in automatic recovery techniques using feature space, operational data may not be labeled properly, and when learning is performed by entropy minimization, there is a risk that additional learning will not necessarily be performed in the direction of the correct answer for each operational data.

[0009] Furthermore, a technology that clusters output data based on information about the spherical distance between images and associates it with tags for learning does not take into account changes in data over time. Furthermore, a technology that updates an AI model using labeled data obtained from parameters based on the difference in data distribution between an inference dataset and a training dataset uses the difference between the inference data and the training data, but does not take into account changes in the data over time. Furthermore, a technology that updates a machine learning model when significant movement is detected using a histogram also does not take into account changes in data over time. Therefore, regardless of which technology is used, it is difficult to appropriately adjust the range of data tracking in response to changes in the data.

[0010] The disclosed technology has been made in consideration of the above, and aims to provide a machine learning program, a machine learning device, and an information processing system that enable stable AI operation. [Means for solving the problem]

[0011] In one aspect of the machine learning program, machine learning device, and information processing system disclosed in the present application, operational data is input into a learned machine learning model trained based on learning data, and the inference results obtained are collected, density-based clustering is performed on the collected inference results to generate clusters, and for each cluster, an estimated label corresponding to the cluster is estimated from correct answer labels that are all correct answers that can be inference results, and the learned machine learning model is fine-tuned based on the operational data belonging to the cluster and the estimated label corresponding to the cluster. [Effects of the Invention]

[0012] In one aspect, the present invention enables stable AI operation. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a machine learning device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an outline of recovery learning and additional learning of a machine learning model. [Figure 3] FIG. 3 is a diagram showing an example of the result of density-based clustering. [Figure 4] FIG. 4 is a diagram illustrating density-based clustering. [Figure 5] FIG. 5 is a flowchart of a learning process performed by the machine learning device according to the embodiment. [Figure 6] FIG. 6 is a hardware configuration diagram of the machine learning device. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, examples of the machine learning program, the machine learning device, and the information processing system disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the machine learning program, the machine learning device, and the information processing system disclosed in the present application are not limited to the following examples. [Example]

[0015] 1 is a block diagram of a machine learning device according to an embodiment. In an information processing system 100 according to this embodiment, a machine learning device 1 is connected to an operation data generating device 2.

[0016] The operational data generation device 2 generates operational data for use by the machine learning device 1 for inference and provides it to the machine learning device 1. Operational data is data used in system operation for business operations, and is data for which the correct answer for inference is unknown. There are many pieces of operational data, and they form a dataset.

[0017] For example, the operation data generation device 2 generates operation data such as images taken with a camera of a large number of products manufactured in an operating factory. By using these images, it is possible to determine whether the products manufactured in the factory are good or bad. Then, the operation data generation device 2 transmits the generated operation data to the machine learning device 1.

[0018] The machine learning device 1 receives input of operation data from the operation data generation device 2. Then, the machine learning device 1 performs inference using the input operation data. For example, when an image of a product manufactured in a factory is acquired as operation data, the machine learning device 1 can infer whether the product in the image is good or bad.

[0019] 1, the machine learning device 1 includes a learning execution unit 11, a recovery learning execution unit 12, an inference result output unit 13, a machine learning model 14, and a data accumulation unit 15. The machine learning device 1 further includes a low-dimensional mapping unit 16, a density-based clustering execution unit 17, a fine-tuning execution unit 18, a cluster check unit 19, and a label estimation unit 20.

[0020] The learning execution unit 11 causes the machine learning model 14 to perform machine learning using the learning data, thereby generating a trained machine learning model 14. The learning data is, for example, training data including input data and a correct answer to an inference for the input data. A large amount of training data exists to form a data set. For example, the learning execution unit 11 causes the machine learning model 14 to learn to infer the quality of a product based on the image of the product manufactured in the factory, using training data including images of the product manufactured in the factory and information on the quality of the product.

[0021] The machine learning model 14 is, for example, a model using a DNN (Deep Neural Network). The machine learning model 14 is an AI (Artificial Intelligence) that receives input of operation data generated by the operation data generation device 2 and performs predetermined inference based on the input operation data. For example, the machine learning model 14 receives input of images of products manufactured in a factory as operation data and infers whether the product is good or bad based on the images.

[0022] The inference result output unit 13 acquires the inference result for the operational data by the machine learning model 14. Then, the inference result output unit 13 outputs the acquired inference result. For example, the inference result output unit 13 may display the inference result on a display device such as a monitor (not shown), or may transmit the inference result to a terminal device of a user who uses the inference result.

[0023] Here, it is conceivable that the quality of the operational data may change due to influences at the time of generation, such as changes in the environment in which the operational data is generated. For example, accumulation of dirt on the lens of a camera that captures images of a product may cause changes in the operational data, such as darkening of the images generated by the operational data generation device 2. When changes occur in the operational data in this way, it is conceivable that the tendencies of the learning data used for learning by the machine learning model 14 may differ from those of the operational data.

[0024] If the trends of the training data and the operational data differ, the machine learning model 14, which has been trained using the training data, may be unable to make appropriate inferences when using changed operational data. For example, when the machine learning model 14 performs inference using an image of a product that would normally be determined to be a good product as input, it may determine that the product is defective due to changes in the operational data. In this way, the inference accuracy of the machine learning model 14 may deteriorate over time.

[0025] The data accumulation unit 15 acquires and accumulates the operational data input from the operational data generation device 2. The data accumulation unit 15 also acquires the inference results from the machine learning model 14 and accumulates them in association with the operational data. The data accumulation unit 15 also acquires the operational data and the inference results when inference is performed by the machine learning model 14 that has been tuned by the recovery learning execution unit 12, and accumulates them in association with each other.

[0026] In this way, the data accumulation unit 15 collects inference results obtained from operational data based on the learned machine learning model 14 that has been trained using the learning data. The data accumulation unit 15 also collects inference results obtained by inputting operational data into the machine learning model 14 that has been tuned by the recovery learning execution unit 12.

[0027] When the inference accuracy of the machine learning model 14 deteriorates due to changes in the operational data as described above, the recovery learning execution unit 12 causes the machine learning model 14 to perform recovery learning using the operational data at that time to recover the inference accuracy. For example, the recovery learning execution unit 12 represents the operational data as points on a coordinate system and separates the operational data by label at a boundary line. The recovery learning execution unit 12 then continuously tracks changes in the distribution of the operational data. Based on the tracking results, the recovery learning execution unit 12 fixes weights other than those of the BN layer and the FC layer and tunes the BN layer and the FC layer to minimize entropy due to a loss function, thereby automatically adjusting the machine learning model 14 to recover the inference accuracy. This tuning by the recovery learning execution unit 12 is called recovery learning, and the machine learning model 14 on which recovery learning has been performed is called an adjusted machine learning model 14.

[0028] Here, the loss function is an example of a "predetermined loss function." The recovery learning execution unit 12 acquires an inference result obtained by inputting operational data into the trained machine learning model 14 and a tracking label obtained from the ground truth label by tracking changes in the operational data. Then, based on the inference result and the tracking label, the recovery learning execution unit 12 causes the trained machine learning model 14 to perform learning that minimizes entropy in the predetermined loss function, thereby generating an adjusted machine learning model 14.

[0029] When performing recovery learning, some operational data may be difficult to assign a label to or may be assigned a label different from the correct label. Therefore, performing recovery learning using inaccurately labeled operational data may result in learning in an inaccurate direction due to entropy minimization. Therefore, the machine learning device 1 performs additional learning as follows to improve inference accuracy.

[0030] The low-dimensional mapping unit 16 acquires the inference results of the adjusted machine learning model 14 from the data accumulation unit 15. Then, the low-dimensional mapping unit 16 maps the inference results, which have dimensions equal to the number of classes, which is the number of different labels assigned to the operational data, into a low-dimensional space. In this embodiment, the low-dimensional mapping unit 16 maps the inference results into two dimensions. In this way, the low-dimensional mapping unit 16 maps the data of the inference results, which have dimensions equal to the number of correct labels, into a low-dimensional space.

[0031] The density-based clustering execution unit 17 acquires the inference results mapped into two dimensions by the low-dimensional mapping unit 16. Then, the density-based clustering execution unit 17 performs density-based clustering on the inference results mapped into two dimensions, in which the number of clusters matches the number of classes.

[0032] Density-based clustering is a method of forming individual clusters for data groups with high density from a data set. For example, the density-based clustering execution unit 17 performs density-based clustering with the idea that the point with the highest density in each data group is set as the center and data groups within a predetermined distance from that center are clustered. Density-based clustering is capable of detecting outliers and noise, and can treat outliers as data that are not included in any cluster. By removing outliers and noise from the data that make up the clusters, the density-based clustering execution unit 17 can generate clusters made up of data groups that accurately correspond to labels.

[0033] The density-based clustering execution unit 17 performs the density-based clustering described above, thereby classifying the inference results mapped onto two dimensions into clusters equal to the number of classes. The density-based clustering execution unit 17 outputs the clustering results to the label estimation unit 20.

[0034] Thereafter, when the density-based clustering execution unit 17 receives an instruction to re-execute clustering from the cluster check unit 19, it adjusts parameters of density-based clustering, such as the distance from the point with the highest density when generating clusters. Thereafter, the density-based clustering execution unit 17 re-executes density-based clustering with the adjusted parameters, and outputs the result of the re-execution of clustering to the label estimation unit 20.

[0035] In this way, the density-based clustering execution unit 17 performs density-based clustering on the inference results after mapping. Furthermore, the density-based clustering execution unit 17 performs density-based clustering so that the number of clusters matches the number of correct labels.

[0036] The label estimation unit 20 receives an input of the clustering results of the inference results mapped into two dimensions from the density-based clustering execution unit 17. Next, the label estimation unit 20 estimates the label supported by each cluster using the labels assigned to each class and the inference results from the machine learning model 14. Then, the label estimation unit 20 outputs information on the estimated label for each cluster to the cluster check unit 19. In this way, the label estimation unit 20 estimates, for each cluster, an estimated label corresponding to the cluster from among all correct labels that are the correct answer and can be the inference result.

[0037] The cluster check unit 19 receives input of estimated label information for each cluster from the label estimation unit 20. Next, the cluster check unit 19 determines whether the number of different estimated labels matches the number of clusters. That is, the cluster check unit 19 determines whether the estimated labels correspond one-to-one to the correct labels. If the number of estimated labels is less than the number of clusters and the number of different estimated labels does not match the number of clusters, the cluster check unit 19 determines whether density-based clustering has been re-executed.

[0038] If density-based clustering has not been re-executed, cluster check unit 19 instructs density-based clustering execution unit 17 to re-execute clustering. On the other hand, if density-based clustering has been re-executed, cluster check unit 19 instructs recovery learning execution unit 12 to increase the amount of operational data and then perform recovery learning again.

[0039] On the other hand, if the number of different estimated labels matches the number of clusters, the cluster check unit 19 outputs information on the estimated labels for each cluster to the fine tuning execution unit 18.

[0040] The fine tuning execution unit 18 receives input of information on estimated labels for each cluster from the cluster check unit 19. Next, the fine tuning execution unit 18 acquires operational data belonging to each cluster from the data accumulation unit 15. Then, the fine tuning execution unit 18 combines the operational data with the estimated labels to create paired data.

[0041] Thereafter, the fine-tuning execution unit 18 causes the adjusted machine learning model 14 to perform additional learning using the paired data. By causing the adjusted machine learning model 14 to perform additional learning, the fine-tuning execution unit 18 can improve the inference accuracy of the adjusted machine learning model 14. In this way, the fine-tuning execution unit 18 fine-tunes the trained machine learning model 14 based on the operational data belonging to a cluster and the estimated labels corresponding to the cluster. Furthermore, the fine-tuning execution unit 18 performs fine-tuning when the number of clusters and the number of correct labels match and the estimated labels correspond one-to-one to the correct labels of the class.

[0042] Fig. 2 is a diagram showing an overview of recovery learning and additional learning of a machine learning model. Next, an overview of recovery learning and additional learning of the machine learning model 14 will be explained together with reference to Fig. 2. Here, the machine learning model 14 that has been trained by the learning execution unit 11 is referred to as the machine learning model 101.

[0043] The machine learning model 101 performs inference on input operational data and outputs the inference result (step S1) while the machine learning device 1 is operating. As time passes, the operational data changes, and the inference accuracy of the machine learning model 101 deteriorates.

[0044] Therefore, based on the results of continuously tracking the distribution changes of the operational data, the recovery learning execution unit 12 fixes weights other than those of the BN layer and the FC layer and tunes the BN layer and the FC layer to minimize the entropy due to the loss function. As a result, the recovery learning execution unit 12 automatically adjusts the machine learning model 101 to generate a machine learning model 102 in which the inference accuracy is restored (step S2).

[0045] Next, the machine learning model 102 receives input operational data, executes inference, and outputs inference results for the input operational data. The data accumulation unit 15 collects and accumulates inference results for a large amount of operational data by the machine learning model 102 along with the operational data. The low-dimensional mapping unit 16 then acquires the inference results from the data accumulation unit 15 and maps the inference results into a low-dimensional space to obtain mapping results 103 (step S3). The mapping results 103 represent classes of inference results that differ depending on the data group to which the added patterns differ.

[0046] The density-based clustering execution unit 17 executes density-based clustering on the mapping result 103 so that the number of classes in the inference matches the number of clusters (step S4). Furthermore, the label estimation unit 20 executes estimation of the label of each cluster.

[0047] 3 is a diagram showing an example of the result of density-based clustering. For example, the density-based clustering execution unit 17 performs density-based clustering on the mapping result 103 to classify the data of the inference result into clusters 201 to 210. Here, data 220 represents outliers and noise data. The density-based clustering execution unit 17 performs clustering by excluding the data 220 that corresponds to outliers and noise.

[0048] FIG. 4 is a diagram for explaining density-based clustering. Here, for example, a case will be described in which product images are used as operational data. Image 301 is training data, and the data included in range 302 is operational data. The direction of arrow P indicates the passage of time, and data drift occurs as time passes, causing the operational data to change. Due to changes in the operational data over time, image 301 and image 303 are both images of good products, but there is a difference in data quality.

[0049] Here, data distribution 310 indicates the distribution of inference results for training data. Furthermore, data distribution 320 indicates the distribution of inference results at the time image 303 is generated. As shown by data distribution 310 and data distribution 320, the inference results shift, but the distribution shapes are approximately the same, and the dense portions of data distribution 310 correspond to the dense portions of data distribution 320. Furthermore, data group 311 in the dense portion of data distribution 310 is considered to be likely, and data group 321 in the dense portion of data distribution 320 is also considered to be likely. In other words, the label of data group 321 is likely to match the label of data group 311. In density-based clustering, data group 321 from data distribution 320 is organized into a cluster. In other words, it can be said that density-based clustering can perform clustering that appropriately corresponds to labels.

[0050] For example, when k-means clustering is used, clustering is performed so that all data is included in one of the clusters. Furthermore, k-means clustering does not take into account the density of data distribution. In contrast, the density-based clustering execution unit 17 performs density-based clustering that excludes outliers and noise data, thereby enabling clustering that more appropriately corresponds to labels. This allows the label estimation unit 20 to estimate labels with high accuracy.

[0051] Returning to Figure 2, the explanation will continue. The fine tuning execution unit 18 generates paired data of the labels estimated by the label estimation unit 20 and the operational data. Then, the fine tuning execution unit 18 uses the paired data to perform additional learning on the machine learning model 14 (step S5). This allows the fine tuning execution unit 18 to generate the machine learning model 105 with restored inference accuracy.

[0052] 5 is a flowchart of the learning process by the machine learning device according to the embodiment. Next, the flow of the learning process by the machine learning device 1 according to the embodiment will be described with reference to FIG.

[0053] The learning execution unit 11 causes the machine learning model 14 to perform machine learning using the learning data, and generates a trained machine learning model 14 (step S101).

[0054] The data accumulation unit 15 acquires the operation data input from the operation data generation device 2. The data accumulation unit 15 also acquires the inference result by the machine learning model 14. Then, the data accumulation unit 15 associates the operation data with the inference result and accumulates them (step S102).

[0055] Based on the results of continuously tracking the distribution changes of the operational data, the recovery learning execution unit 12 fixes weights other than those of the BN layer and the fully connected layer and tunes the BN layer and the fully connected layer to minimize the entropy due to the loss function. As a result, the recovery learning execution unit 12 automatically adjusts the machine learning model 101 to generate an adjusted machine learning model 14 (step S103).

[0056] The data accumulation unit 15 collects, associates, and accumulates the operational data and the inference results obtained by inputting the operational data using the adjusted machine learning model 14 (step S104).

[0057] The low-dimensional mapping unit 16 acquires the inference result by the adjusted machine learning model 14 from the data storage unit 15. Then, the low-dimensional mapping unit 16 maps the inference result of the number of dimensions equal to the number of classes into a low dimension (step S105).

[0058] The density-based clustering execution unit 17 performs density-based clustering, in which the number of clusters matches the number of classes, on the inference results mapped into two dimensions (step S106).

[0059] The label estimation unit 20 estimates the label supported by each cluster using the label assigned to each class and the estimation result by the machine learning model 14 (step S107).

[0060] The cluster check unit 19 determines whether the number of different estimated labels matches the number of clusters (step S108).

[0061] If the number of different estimated labels does not match the number of clusters (step S108: No), the cluster check unit 19 determines whether or not density-based clustering has been re-executed (step S109).

[0062] If the density-based clustering has not been re-executed (step S109: No), the cluster check unit 19 instructs the density-based clustering execution unit 17 to re-execute clustering. The density-based clustering execution unit 17 adjusts the parameters of the density-based clustering (step S110). Thereafter, the learning process returns to step S106.

[0063] On the other hand, if the density-based clustering has been re-executed (step S109: Yes), the cluster check unit 19 instructs the low-dimensional mapping unit 16 to start processing again after increasing the operational data. Then, the learning process returns to step S102.

[0064] On the other hand, if the number of different estimated labels matches the number of clusters (step S108: Yes), the cluster check unit 19 outputs information on the estimated labels for each cluster to the fine tuning execution unit 18. The fine tuning execution unit 18 acquires operational data belonging to each cluster from the data accumulation unit 15. Then, the fine tuning execution unit 18 combines the operational data and the estimated labels to create paired data (step S111).

[0065] Thereafter, the fine tuning execution unit 18 executes fine tuning by causing the adjusted machine learning model 14 to perform additional learning using the paired data (step S112).

[0066] In the above description, the machine learning device 1 performs additional learning after clustering so that the number of clusters corresponds one-to-one to the labels of the inference results, but this is not limited to this. Even if the number of clusters generated by clustering does not correspond one-to-one to the labels, the labels of the clusters may be estimated, and the adjusted machine learning model 14 may perform additional learning using the operational data belonging to each cluster and the estimated labels. For example, even if a cluster corresponds to a portion of the label, the machine learning device 1 can improve inference accuracy by using the cluster to train the adjusted machine learning model 14.

[0067] In the above description, the machine learning device 1 performed fine tuning on the adjusted machine learning model 14 that had undergone recovery learning using data that had been labeled by tracking changes in operational data, but this is not limited to this. The machine learning device 1 can perform fine tuning using the density-based clustering described above on any machine learning model 14 that performs inference using data that changes over time and that can be classified to some extent.

[0068] As described above, the machine learning device 1 fixes weights other than those in the BN layer and the fully connected layer based on the tracking results of continuously tracking changes in the distribution of operational data, performs tuning by minimizing entropy using a loss function, and generates an adjusted machine learning model 14. Furthermore, the machine learning device 1 maps the inference results obtained by inputting the operational data into the adjusted machine learning model 14 to a lower dimension, and performs density-based clustering on the mapped inference results. The machine learning device 1 then estimates the labels of the clusters, and causes the adjusted machine learning model 14 to perform additional learning using the operational data belonging to the clusters and the estimated labels.

[0069] This makes it possible to reduce the degradation of the inference accuracy of the machine learning model 14 even when the operational data changes uniformly due to data drift. In other words, it is possible to improve the robustness of the machine learning model 14 against data drift. In this way, stable AI operation is possible even if there is no label that indicates the correct answer for inference on the operational data.

[0070] For example, a comparative study of inference accuracy is performed using CIFAR-10 as training data and setting the destruction rate of operational data to the training data to the maximum. In this case, a machine learning model 14 obtained by additionally training an adjusted machine learning model 14 has higher inference accuracy than a trained machine learning model 14 or an adjusted machine learning model 14 using internal parameter adjustment technology. In other words, the machine learning device 1 has improved robustness to changes in operational data compared to a trained machine learning model or an adjusted machine learning model.

[0071] (Hardware configuration) 6 is a diagram showing the hardware configuration of the machine learning device 1. Next, an example of the hardware configuration for realizing each function of the machine learning device 1 will be described with reference to FIG.

[0072] 6, the machine learning device 1 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0073] The network interface 94 is an interface for communication between the machine learning device 1 and an external device. The network interface 94 relays communication between the operation data generation device 2 and the CPU 91, for example.

[0074] The hard disk 93 is an auxiliary storage device. The hard disk 93 stores the machine learning model 14 illustrated in FIG. 1. The hard disk 93 can also be used as a data storage location for the data storage unit 15. The hard disk 93 also stores various programs, including the programs described below. For example, the hard disk 93 stores programs for implementing the functions of the learning execution unit 11, the recovery learning execution unit 12, the inference result output unit 13, and the data storage unit 15 illustrated in FIG. 1. The hard disk 93 also stores programs for implementing the functions of the low-dimensional mapping unit 16, the density-based clustering execution unit 17, the fine-tuning execution unit 18, the cluster check unit 19, and the label estimation unit 20 illustrated in FIG. 1.

[0075] The memory 92 is a main storage device and may be, for example, a dynamic random access memory (DRAM).

[0076] The CPU 91 reads various programs from the hard disk 93, expands them into the memory 92, and executes them. As a result, the CPU 91 realizes the functions of the learning execution unit 11, the recovery learning execution unit 12, the inference result output unit 13, and the data accumulation unit 15, all of which are illustrated in Fig. 1. The CPU 91 also realizes the functions of the low-dimensional mapping unit 16, the density-based clustering execution unit 17, the fine-tuning execution unit 18, the cluster check unit 19, and the label estimation unit 20, all of which are illustrated in Fig. 1. [Explanation of symbols]

[0077] 1. Machine learning device 2 Operational data generator 11 Learning Execution Department 12 Recovery Learning Executive Department 13 Inference result output section 14 Machine Learning Models 15 Data storage unit 16 Low-dimensional mapping section 17 Density-based clustering execution unit 18 Fine Tuning Execution Department 19 Cluster Check Section 20 Label estimation part

Claims

1. Based on a trained machine learning model that has been trained using training data, we collect inference results obtained from operational data, performing density-based clustering on the collected inference results to generate clusters; For each cluster, estimate an estimated label corresponding to the cluster from among all correct answer labels that are correct answers that can be inference results; Fine-tuning the trained machine learning model based on the operational data belonging to the cluster and the estimated label corresponding to the cluster. A machine learning program that causes a computer to execute processing.

2. and causing the computer to further execute a process of generating an adjusted machine learning model by causing the trained machine learning model to perform learning that minimizes entropy in a predetermined loss function based on an inference result obtained by inputting the operational data into the trained machine learning model and a tracking label obtained from the ground truth label by tracking changes in the operational data, and The process of collecting the inference results includes a process of inputting the operational data into the adjusted machine learning model and collecting the inference results obtained. The machine learning program according to claim 1 .

3. The cluster generation process includes: Mapping the data of the inference result having a dimension equal to the number of correct labels into a lower dimension; Execute the density-based clustering on the inference results after mapping.

2. The machine learning program according to claim 1, further comprising:

4. the process of generating clusters includes a process of performing the density-based clustering so that the number of clusters matches the number of correct answer labels; further causing the computer to perform a process of determining whether the estimated labels correspond one-to-one to the ground truth labels; The fine-tuning process is performed when the number of clusters and the number of correct labels match and the estimated labels correspond one-to-one to the correct labels of the classes. The machine learning program according to claim 1 .

5. a data accumulation unit that inputs operational data into a learned machine learning model that has been trained based on learning data and collects inference results; a density-based clustering execution unit that performs density-based clustering on the inference results collected by the data accumulation unit to generate clusters; a label estimation unit that estimates, for each of the clusters generated by the density-based clustering execution unit, an estimated label corresponding to the cluster from among all correct answer labels that are correct answers that can be inference results; a fine-tuning execution unit that fine-tunes the trained machine learning model based on operational data belonging to the cluster and the estimated label corresponding to the cluster; A machine learning device comprising:

6. An information processing system having a machine learning device and an operational data generation device, The machine learning device includes: a data accumulation unit that inputs operational data obtained from the operational data generation device into a learned machine learning model that has been trained based on learning data and collects inference results obtained; a density-based clustering execution unit that performs density-based clustering on the inference results collected by the data accumulation unit to generate clusters; a label estimation unit that estimates, for each of the clusters generated by the density-based clustering execution unit, an estimated label corresponding to the cluster from among all correct answer labels that are correct answers that can be inference results; a fine-tuning execution unit that fine-tunes the trained machine learning model based on the operational data belonging to the cluster and the estimated label corresponding to the cluster. An information processing system comprising:

Images