Evaluation program, evaluation method, and accuracy evaluation device
The described method updates machine learning models using feature-based evaluation processes to adapt to changing operational data, ensuring accurate estimation and preventing accuracy decline, addressing the limitations of existing evaluation methods.
Patent Information
- Application Number
- JP2023539537
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-08-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-06
AI Technical Summary
Existing methods for evaluating the accuracy of machine learning models are impractical due to high costs of obtaining correct labels, difficulty in evaluating accuracy when both models degrade similarly, and challenges in determining the accuracy of new data classification, leading to delayed responses and decreased estimation accuracy.
A computer executes processes involving second machine learning by updating parameters based on first and second data features, creating a third machine learning model with correct labels, and evaluating its prediction accuracy using a tracking accuracy index to ensure model adaptation and accuracy maintenance.
The solution effectively maintains the estimation accuracy of machine learning models by adapting to changes in operational data, enabling accurate evaluation without relying on correct labels and preventing estimation accuracy deterioration.
Smart Images

Figure 0007589819000001 
Figure 0007589819000002 
Figure 0007589819000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an evaluation program, an evaluation method, and an accuracy evaluation device. [Background technology]
[0002] Business operations using machine learning are carried out in the following steps. First, in the training phase, machine learning is repeated using prepared training data to create a machine learning model to be used in business. The training data is also called teacher data. Then, in the application phase, operational data to be used in business is input into the created machine learning model, and prediction results are output. Here, as the machine learning model continues to be used in business, the external environment changes over time, and the tendency of the operational data input to the machine learning model may change from the training data used to train the machine learning model.
[0003] As a result of changes in operational data, the estimation accuracy of machine learning models deteriorates due to differences between the training data used when developing the machine learning model and the operational data, changes in the statistical trends of the input operational data during operation, etc. Therefore, technology is being developed to deal with such deterioration in the estimation accuracy of machine learning models.
[0004] As one of the techniques to deal with the deterioration of estimation accuracy, a technique that attempts to automatically recover the accuracy of a machine learning model according to the operational data input during operation has been proposed. In this technique, automatic accuracy recovery is attempted in the following procedure. For example, the operational data input during operation is represented in a data space. The operational data represented in the data space is separated by a boundary called a decision boundary by a machine learning model. Next, the operational data represented in the data space is projected into a feature space, which is a mathematical space that represents the characteristics of data distribution as a data group, so that the data group formed by the operational data is captured as a shape and changes are tracked. After that, the classification result of the operational data projected on the feature space is labeled as a pseudo label for the operational data represented in the data space. Then, the labeled operational data is used for retraining to automatically recover the accuracy of the machine learning model. This makes it possible to prevent the accuracy of the machine learning model, and ultimately the AI (artificial intelligence) system, from deteriorating even if the quality of the operational data changes. A pseudo label is, for example, a correct answer label that is assigned by estimation to unlabeled data.
[0005] However, whether automatic accuracy recovery works depends on the data distribution, and if the operational data changes significantly, it may be difficult to recover from accuracy degradation even if automatic accuracy recovery is performed. Even if input monitoring is performed, the accuracy of the machine learning model may deteriorate because the input operational data contains abnormal values other than the monitored values. Therefore, in order to ensure the quality of the AI system, it is important to monitor the output results of the machine learning model and detect anomalies and predict or detect accuracy degradation in advance.
[0006] For example, we will explain the case where a model with a feature extraction layer such as CNN (Convolution Neural Network) and a fully connected layer that constructs a decision boundary is used as the configuration of a machine learning model that performs deep learning. In addition, as an unsupervised adaptation, we will introduce a gradient reversal layer that inverts the sign of the gradient when going backward, and use BCE (Binary Cross Entropy) Loss to learn from feature extraction of input data, and use adversarial training as an example. In this case, automatic accuracy recovery is performed using pseudo labels using density clustering. Then, the feature extraction layer and the fully connected layer are retrained by a model that additionally introduces a metric layer after the fully connected layer. Here, the output from the feature extraction layer to the metric layer is fixed. The metric layer is a layer that performs distance learning and creates a feature space.
[0007] In such cases, one method to evaluate the degree of accuracy recovery is to obtain new correct labels to check whether the output predicted labels are correct, and evaluate the extent to which the accuracy of the machine learning model has been recovered. Another method is to input data into the machine learning model before automatic accuracy recovery, and then input the same data into the machine learning model that has been automatically recovered, to evaluate the degree of accuracy recovery.
[0008] Furthermore, as a conventional technique for the accuracy of a machine learning model, there is a technique for calculating a qualitative rate by acquiring the number of input data, acquiring the number of corrections from the correction information of the classification result, and calculating a qualitative rate. Also, as a binary classifier technique for providing an accuracy index, there is a technique for calculating a score indicating whether an unlabeled sample is positive example data or not, calculating an evaluation value model and a probability model based on the score and the sample, and learning a score function for binary classification. There is also a technique for dividing an input screen image into left and right images, extracting a first candidate for the left and right eyes using a morphological operator, and extracting a second candidate for the left and right eyes using a binary classifier for the data of the first candidate, thereby identifying the positions of the left and right eyes. [Prior art documents] [Patent documents]
[0009] [Patent Document 1] International Publication No. 2020 / 036192 [Patent Document 2] JP 2017-126158 A [Patent Document 3] US Patent Application Publication No. 2006 / 0110030 Summary of the Invention [Problem to be solved by the invention]
[0010] However, in the technology of obtaining correct labels to evaluate the accuracy of a machine learning model, if the cost of obtaining correct labels is high, it is not realistic to perform the accuracy evaluation.In addition, in the technology of evaluating the accuracy from the prediction difference of the output from a machine learning model before and after automatic accuracy recovery, if the accuracy of both models is similarly degraded, the prediction difference becomes small and it is difficult to perform an appropriate evaluation.
[0011] In addition, the technique of calculating the qualitative rate from the number of input data and the number of corrections concerns the extent to which new data can be classified by the current machine learning model, and it is difficult to evaluate the accuracy of the machine learning model itself. In addition, the technique of calculating an evaluation value model and a probability model based on the score of whether an unlabeled sample is positive data and the sample is an application method of binary classification learning of unlabeled data in logistic regression, and it is difficult to evaluate the accuracy of the machine learning model. In addition, the technique of extracting a second candidate from a first candidate extracted using a morphological operator using a binarization classifier to specify the positions of the left and right eyes can improve the classification accuracy by using the binarization classifier in the final stage, but it is difficult to evaluate the accuracy of the machine learning model. Therefore, regardless of which technique is used, it is difficult to accurately determine the deterioration of the accuracy of the machine learning model, and it is difficult to suppress the deterioration of the estimation accuracy due to the deterioration of the accuracy of the machine learning model due to a delay in response.
[0012] The disclosed technology has been made in consideration of the above, and aims to provide an evaluation program, an evaluation method, and an accuracy evaluation device that maintain the estimation accuracy of a machine learning model. [Means for solving the problem]
[0013] In one embodiment of the evaluation program, evaluation method, and learning model evaluation device disclosed in the present application, the following processes are executed by a computer. One is a process of generating a second machine learning model by updating parameters of a first machine learning model based on a first feature amount obtained from first data based on parameters of the first machine learning model. One is a process of generating a third machine learning model based on first training data including a second feature amount obtained from second data based on parameters of the second machine learning model and a correct answer label indicating first information, and second training data including a third feature amount obtained from third data based on parameters of the second machine learning model and a correct answer label indicating second information. One is a process of evaluating the second machine learning model based on the prediction accuracy of the generated third machine learning model. Effect of the Invention
[0014] In one aspect, the present invention can maintain the estimation accuracy of a machine learning model. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a block diagram of an accuracy evaluation device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an overall flow of automatic accuracy recovery and estimation accuracy evaluation in the embodiment. [Diagram 3] FIG. 3 is a diagram showing an overview of the estimation accuracy evaluation of a model by the accuracy evaluation device. [Figure 4] FIG. 4 is a diagram illustrating an example of the feature extractor. [Diagram 5]FIG. 5 is a flowchart of an estimation accuracy evaluation process performed by the accuracy evaluation device according to the embodiment. [Figure 6] FIG. 6 shows the change in accuracy of model fit against concept drift. [Figure 7] FIG. 7 is a hardware configuration diagram of the accuracy evaluation device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] The following describes in detail the embodiments of the evaluation program, evaluation method, and accuracy evaluation device disclosed in the present application with reference to the drawings. Note that the following embodiments do not limit the evaluation program, evaluation method, and accuracy evaluation device disclosed in the present application. EXAMPLES
[0017] 1 is a block diagram of an accuracy evaluation device according to an embodiment. The accuracy evaluation device 1 according to the embodiment is connected to a machine learning device 2. In addition, both the accuracy evaluation device 1 and the machine learning device 2 are connected to a training data database 31 and an operation data database 32.
[0018] Training data for training the machine learning model is stored in the training data database 31. The training data is teacher data that includes input data and correct output results. In the following description, the machine learning model is simply referred to as a model.
[0019] The operational data database 32 stores operational data that is the subject of inference for which inference is performed using a trained model. Here, in this embodiment, an example is described in which the operational data is already stored in the operational data database 32, but a configuration in which operational data is stored in the operational data database 32 from a device that generates the operational data while the machine learning device 2 is in operation is also possible.
[0020] The machine learning device 2 trains a model using training data, and then applies the trained model to make an estimation on operational data. The machine learning device 2 includes a machine learning unit 21, an estimation unit 22, and an automatic recovery unit 23.
[0021] The machine learning unit 21 holds a basic model before training. The machine learning unit 21 acquires training data from the training data database 31. Next, the machine learning unit 21 inputs input data included in the training data to the held model to obtain an output from the model. Then, the machine learning unit 21 trains the model by adjusting the parameters of the model so that the error between the output from the model and a correct output result is reduced. The machine learning unit 21 outputs the trained model to the estimation unit 22. This trained model by the machine learning unit 21 is an example of a "first machine learning model."
[0022] Fig. 2 is a diagram showing an overall flow of automatic accuracy recovery and estimation accuracy evaluation in the embodiment. The machine learning unit 21 executes learning of the model 101 shown in step S1 in Fig. 2. Data X1 and Y1 in Fig. 2 are training data.
[0023] Returning to FIG. 1, the explanation continues. The estimation unit 22 receives an input of a trained model from the machine learning unit 21. The estimation unit 22 also acquires operation data from the operation data database 32. The estimation unit 22 then inputs the operation data to the trained model and obtains an output from the model as an estimation result. Thereafter, the estimation unit 22 notifies the user of the estimation result, for example, by transmitting the estimation result to a user terminal (not shown). The estimation unit 22 repeats the estimation of the operation data. The operation data input to the model before tracking is an example of "first data".
[0024] Furthermore, the estimation unit 22 outputs the estimation result to the automatic restoration unit 23. After that, when the estimation unit 22 receives an input of a model updated by automatic accuracy recovery from the automatic restoration unit 23, the estimation unit 22 performs estimation using the updated model. That is, the estimation unit 22 performs estimation of the operational data using the model that is successively updated.
[0025] As described above, the external environment changes over time as the machine learning model continues to be used in business, and the trends of the operational data input to the machine learning model may change from the training data used to train the machine learning model. For example, examples of data that change include email spam whose contents change, electricity demand and stock prices that change from year to year, poker hands that change depending on the situation, and fixed-point images that change seasonally between summer and winter and between morning and night. When the statistical characteristics of the objective variable that the machine learning model predicts change unexpectedly over time, this is sometimes called "concept drift." When concept drift occurs, restoring the estimation accuracy of the model in a way that corresponds to the change may be called "following" the change in data. Furthermore, correcting the model so that inference can be performed appropriately after concept drift occurs may be called "fitting the model." In other words, when concept drift occurs, the estimation accuracy of the model is restored and the model is adapted to the change in data, so that the change in data can be followed.
[0026] In concept drift, when considering the joint probability of two random variables at time t0 and time t1, p, which represents the joint distribution that represents the probability of all possible combinations, does not match. There are several possible causes for this. The first is when the distribution of the training data does not change, but the correct label changes. In this case, the concept drift is called virtual drift. The second is when the conditional probability changes. The third is when the decision boundary changes.
[0027] A technology that can adapt a model to changes in operational data when it is assumed that the true decision boundary of concept drift does not change or changes very little is called domain adaptation technology. For example, the above assumption is met when the operational data is image data. In the case of image data, even if an image of a cat is rotated and its color changes, the object that is reflected is the same, so it can be said that the decision boundary does not change. In this way, domain adaptation is essentially a technology in the image processing field, and it can be said that it targets virtual drift.
[0028] On the other hand, the automatic accuracy recovery technology that projects a data group into a feature space and tracks it is an adaptation technology that targets concept drift in general. This technology can handle various types of data such as image data and table data as the estimation target. However, even with this technology, when the estimation target is image data, it is essentially an adaptation targeting virtual drift. The following explanation targets automatic accuracy recovery that projects a data group into a feature space and tracks it, and it is possible to handle various types of data other than image data, such as table data.
[0029] 2, the estimation unit 22 performs an estimation process using the tracked model 102 that has been subjected to automatic accuracy recovery for the model 101 and tracked to changes in the operational data. Data Y2' is the estimation result for data X2 by the estimation unit 22. Data Y2 is also a correct label for data X2. If the estimation accuracy of the tracked model 102 deteriorates, data Y2 and data Y2' will no longer match.
[0030] When the model training is completed, the automatic repair unit 23 acquires the training data and the trained model from the machine learning unit 21. Next, the automatic repair unit 23 projects each training data arranged in the data space into a feature space, which is a mathematical space indicating the feature of the data. Specifically, the automatic repair unit 23 projects each training data into the feature space and converts it into a vector using a function that extracts the feature of the training data included in each class in the data space. As a result, training data of the same class are represented as vectors at a grouped position in the feature space because the features are close. This allows the automatic repair unit 23 to represent the training data represented in the high-dimensional data space in a grouped manner in the low-dimensional feature space.
[0031] In this projection into the feature space, if the machine learning model is a neural network, the values calculated up to the final layer of the trained neural network or a specific layer before the final layer are used as the features of the data. That is, the projection is performed by inputting data into the neural network and acquiring the features of the data by performing forward propagation calculations. The space in which the vectors representing the acquired features are arranged is the feature space. That is, it can be said that the automatic restoration unit 23 projects the data into the feature space based on the parameters of the trained model.
[0032] Next, the automatic restoration unit 23 performs density-based clustering on the training data represented in the feature space to determine data groups (clusters) in the feature space. Next, the automatic restoration unit 23 identifies data groups in the feature space corresponding to each class in the data space. Then, the automatic restoration unit 23 sets the correct answer label of each class in the data space as the correct answer label of each data group in the corresponding feature space. That is, at this stage, the automatic restoration unit 23 only checks the correspondence between the data groups corresponding to each cluster of the training data projected onto the feature space and the classes indicated by the correct answer labels.
[0033] After that, when the model enters the operation stage, the automatic repair unit 23 acquires the operation data used for the estimation from the estimation unit 22. Then, the automatic repair unit 23 projects the acquired operation data from the data space to the feature space. This projection to the feature space is performed based on the features of the training data extracted using the model, and the features extracted for the projection to the feature space are an example of a "first feature". Then, the automatic repair unit 23 performs clustering of the operation data in the feature space, identifies a class corresponding to the generated cluster, and determines a pseudo label. Next, the automatic repair unit 23 generates training data for retraining by labeling the operation data with the result of the clustering as a pseudo label, and performs retraining of the machine learning model. As a result, the automatic repair unit 23 creates a new machine learning model having an updated decision boundary in the data space. The operation data labeled with the result of this clustering as a pseudo label is an example of a "third training data". Then, the automatic repair unit 23 outputs the model updated by retraining to the estimation unit 22. In this way, the automatic restoration unit 23 performs automatic accuracy restoration of the model, and makes the model follow the changes in the operational data to ensure the inference accuracy. This model that follows the changes in the operational data is an example of a "second machine learning model."
[0034] Data X2 in Fig. 2 is operational data. Then, the automatic restoration unit 23 performs tracking of the change in the operational data shown in step S2 in Fig. 2 in order to adapt the model 101 to the change, thereby generating a tracked model 102.
[0035] The accuracy evaluation device 1 obtains a tracking accuracy index serving as an index for evaluating the estimation accuracy from a tracked model that has undergone automatic accuracy recovery and is used for estimation by the estimation unit 22, and evaluates the estimation accuracy of the tracked model using the obtained tracking accuracy index. Then, when the estimation accuracy of the tracked model deteriorates, the accuracy evaluation device 1 notifies the user of the deterioration of the estimation accuracy. The accuracy evaluation device 1 will be described in detail below. The accuracy evaluation device 1 has a control unit 10. As shown in FIG. 1, the control unit 10 has a feature extraction unit 11, a combined data generation unit 12, a classification unit 13, a judgment unit 14, and a notification unit 15.
[0036] The feature extraction unit 11 extracts features for accuracy evaluation from the training data and the operational data shown in step S4 in Fig. 2. The operation of the feature extraction unit 11 will be described in detail below.
[0037] Fig. 3 is a diagram showing an overview of the estimation accuracy evaluation of a model by the accuracy evaluation device, and Fig. 4 is a diagram for explaining an example of a feature extractor.
[0038] The feature extraction unit 11 has the feature extractor 105 in FIG. 3. For example, a case will be described in which the model used by the estimation unit 22 is the CNN shown in FIG. 4. In this case, the feature extractor 105 outputs the output of the layer before the final layer 106 in the CNN as a feature. That is, the feature extractor 105 includes each layer in which a fully connected network except for the final layer 106 is added to a convolutional network. For example, in the case of VGG (Visual Geometry Group) 16, the feature extractor 105 includes up to the layer before the final layer 106 of 1000 class classification. That is, in the case of VGG 16, the feature output from the feature extractor 105 is a first-order tensor (vector) of 4096.
[0039] The feature extraction unit 11 acquires a tracked model that has undergone automatic accuracy recovery and that is used for estimation by the estimation unit 22 of the machine learning device 2 from the estimation unit 22. In addition, the feature extraction unit 11 acquires training data from the training data database 31, and acquires operation data from the operation data database 32. Next, the feature extraction unit 11 inputs the acquired training data to the feature extractor 105 to obtain the feature of the training data as an output. In addition, the feature extraction unit 11 inputs the acquired operation data to the feature extractor 105 to obtain the feature of the operation data as an output. Thereafter, the feature extraction unit 11 outputs the feature of the training data and the feature of the operation data to the combined data generation unit 12.
[0040] In this embodiment, the training data is an example of "second data", and the feature amount of the training data is an example of "second feature amount". Information representing that it is training data is an example of "first information", and a set of data of the feature amount of the training data and the information representing that it is training data is an example of "first training data". Operational data is an example of "third data", and the feature amount of the operation data is an example of "third feature amount". Information representing that it is operational data is an example of "second information", and a set of data of the feature amount of the operation data and the information representing that it is operational data is an example of "second training data".
[0041] 3, the feature extraction unit 11 inputs data X1, which is training data, to the feature extractor 105 and obtains a feature f(X1) as the output. Also, the feature extraction unit 11 inputs data X2, which is operational data, to the feature extractor 105 and obtains a feature f(X2) as the output.
[0042] Here, if the model is properly adapted to the change in the operation data, it is considered that the feature amount f(X1) and the feature amount f(X2) are similar, and the internal space of the model is learned as a space in which the feature amount f(X1) and the feature amount f(X2) are difficult to distinguish. Therefore, as shown in FIG. 3, the feature amount f(X1) and the feature amount f(X2) are input to a detection model 103 that detects whether the data that is the source of each feature amount is training data or operation data, and it is possible to determine whether the tracking of the change in the operation data has been successful or not depending on the degree of detection accuracy. In other words, if the feature amount f(X1) and the feature amount f(X2) are properly classified by the detection model 103 and the original data is detected, it can be said that the internal space of the model is a space in which the feature amount f(X1) and the feature amount f(X2) are difficult to distinguish, and the tracking of the change in the operation data has not been properly performed. As described below, the accuracy evaluation device 1 according to this embodiment calculates a tracking accuracy index that indicates the degree of tracking of the model from the output when the feature f(X1) and the feature f(X2) are input to the detection model 103, and judges whether the model is tracking successfully.
[0043] The combined data generation unit 12 acquires the features of the training data from the feature extraction unit 11. The combined data generation unit 12 also acquires the features of the operation data from the feature extraction unit 11. Then, the combined data generation unit 12 mixes the features of the training data and the features of the operation data to generate combined data in which a plurality of features of the training data and a plurality of features of the operation data are included in a state in which it is difficult to distinguish between the training data and the operation data from information other than the respective features. Then, the combined data generation unit 12 outputs the combined data to the classification unit 13.
[0044] 2, the classification unit 13 inputs the features of the training data and the features of the operation data to the detection model 103 and obtains the output results from the detection model 103. The operation of the classification unit 13 will be described in detail below.
[0045] The classification unit 13 has a detection model 103 which is a binary classifier for classifying the feature amount of the training data and the feature amount of the operation data into two. For example, the classification unit 13 acquires the training data from the training data database 31. The classification unit 13 also acquires the operation data from the operation data database 32. Then, the classification unit 13 performs machine learning using the detection model 103 so as to classify each data into the training data and the classification data using the acquired training data and classification data. In this way, the classification unit 13 can acquire the detection model 103 which classifies the input feature amount into the training data and the operation data. Here, the data used for learning the detection model 103 may be the training data and the operation data which are the source of the feature amount included in the combined data, or may be different training data and operation data.
[0046] Thereafter, the classification unit 13 receives the combined data from the combined data generation unit 12. Then, the classification unit 13 inputs each data included in the combined data to the detection model 103 to classify it into training data and operation data, and estimates whether each data is a feature of the training data or the operation data. Thereafter, the classification unit 13 outputs the classification result to the determination unit 14.
[0047] The determination unit 14 calculates a tracking accuracy index that indicates the degree of conformance of the model to the concept drift, i.e., the degree of tracking to changes in the operational data, as shown in step S6 in Fig. 2. After that, the determination unit 14 uses the calculated tracking accuracy index, and the operation of the determination unit 14 will be described in detail below.
[0048] The determination unit 14 receives an input of the classification result of the training data and the operational data using the detection model 103 from the classification unit 13. Then, the determination unit 14 uses the classification result of the training data and the operational data using the detection model 103 to calculate a tracking accuracy index that indicates the degree of tracking of changes in the operational data.
[0049] For example, the determination unit 14 measures the ROC (Receiver Operating Characteristic) / AUC (Area Under the Curve) score of the prediction accuracy by the detection model 103 and obtains it as a tracking accuracy index. The ROC / AUC score is an effective value as an accuracy index when the data is unbalanced, that is, in the case of two-class classification, when one class has excessive data and the other class has little data. The ROC curve is a curve in a space where the horizontal axis represents the false positive rate and the vertical axis represents the recall. The ROC can be obtained by calculating the value of the recall when the false positive rate is changed. The ACU corresponds to the area under the ROC curve, and it can be determined that the model is reliable when this area is large. The maximum value of the AUC is 1 and the minimum value is 0.5. An AUC of 0.5 is the same value as the result when random prediction is performed. In general, an ROC / AUC score of 0.7 to 0.8 can be regarded as a reliable learning model.
[0050] The determination unit 14 determines whether or not the change in the operational data has been followed using the calculated tracking accuracy index. For example, the determination unit 14 determines the classification accuracy of the detection model 103 by determining whether or not the ROC / AUC score is less than a predetermined accuracy threshold, and determines whether or not the change in the operational data has been followed based on the classification accuracy. For example, when the ROC / AUC score is used as the tracking accuracy index, the accuracy threshold can be set to 0.7 or the like.
[0051] That is, if the ROC / AUC score is less than a predetermined accuracy threshold, the determination unit 14 determines that the classification accuracy of the detection model 103 is low and unreliable. If the detection model 103 cannot perform appropriate classification, it can be said that the internal space in the model has been learned as a space in which the features of the training data and the features of the operation data are difficult to distinguish, and therefore the determination unit 14 determines that tracking of changes in the operation data has been successful.
[0052] On the other hand, if the ROC / AUC score is equal to or greater than a predetermined accuracy threshold, the determination unit 14 determines that the classification accuracy of the detection model 103 is high and reliable. If the detection model 103 is performing appropriate classification, it can be said that the internal space in the model has been learned as a space in which the features of the training data and the features of the operation data can be distinguished, and therefore the determination unit 14 determines that tracking of changes in the operation data has failed.
[0053] When the determination unit 14 determines that the tracking of the change in the operational data has been successful, it continues the estimation process and the automatic recovery process by the machine learning device 2. On the other hand, when the determination unit 14 determines that the tracking of the change in the operational data has failed, it instructs the notification unit 15 to notify the tracking failure.
[0054] Here, in this embodiment, the prediction accuracy of the detection model 103 is obtained depending on whether the training data and the operation data can be classified into different classes, and the estimation accuracy of the model is evaluated based on this, but the classes for evaluating the accuracy of the detection model 103 are not limited to this. For example, the training data and the operation data used up to the previous automatic accuracy recovery may be the "second data", and the operation data used from the previous automatic accuracy recovery to this time may be the "third data". In this case, the feature amount of the second data is an example of the "second feature amount", and the feature amount of the third data is an example of the "third feature amount". Then, the information indicating that the training data and the operation data used up to the previous automatic accuracy recovery are the "first information", and the set of the second feature amount and the first information is the "first training data".
[0055] Furthermore, information indicating that the operation data has been used from the previous automatic accuracy recovery to the current time may be defined as "second information", and a set of the third feature amount and the second information may be defined as "second training data". In this case, the detection model 103 can be trained using the first training data and the second training data, and the prediction accuracy of the detection model 103 can be obtained. In addition, the operation data used up to the previous automatic accuracy recovery may be defined as "second data", and the operation data used from the previous automatic accuracy recovery to the current time may be defined as "third data".
[0056] Notification unit 15 receives an instruction to notify of the tracking failure from determination unit 14. Then, notification unit 15 sends a notification of the failure of automatic accuracy repair to a terminal device (not shown) used by an administrator or user of machine learning device 2, and notifies the administrator or user of machine learning device 2 that the automatic accuracy repair of the model has not been performed appropriately.
[0057] 5 is a flowchart of the estimation accuracy evaluation process performed by the accuracy evaluation device according to the embodiment. Next, the flow of the estimation accuracy evaluation process performed by the accuracy evaluation device 1 according to the embodiment will be described with reference to FIG.
[0058] The machine learning unit 21 acquires training data from the training data database 31. Then, the machine learning unit 21 trains a model using the acquired training data (step S101). After that, the machine learning unit 21 outputs the trained model to the estimation unit 22.
[0059] The estimation unit 22 receives an input of the model from the machine learning unit 21. Next, the estimation unit 22 acquires operation data from the operation data database 32. Then, the estimation unit 22 inputs the operation data to the model and obtains an output, thereby executing inference using the operation data (step S102).
[0060] The automatic restoration unit 23 projects the operational data arranged in the data space onto the feature space, performs density clustering in the feature space, and performs automatic accuracy recovery by assigning pseudo-labels to each operational data and retraining the model (step S103).
[0061] The feature extraction unit 11 generates the feature extractor 105 from the model used by the estimation unit 22 for estimation. Next, the feature extraction unit 11 acquires training data from the training data database 31. Also, the feature extraction unit 11 acquires operation data from the operation data database 32. Then, the feature extraction unit 11 inputs the training data and operation data to the feature extractor 105 and extracts features of the training data and operation data to be output (step S104). Then, the feature extraction unit 11 outputs the extracted features of the training data and operation data to the combined data generation unit 12.
[0062] The combined data generation unit 12 receives the feature amounts of the training data and the feature amounts of the operation data from the feature amount extraction unit 11. Then, the combined data generation unit 12 mixes the feature amounts of the training data and the feature amounts of the operation data to generate combined data in a state in which it is difficult to classify the training data and the operation data from anything other than the feature amounts (step S105). Thereafter, the combined data generation unit 12 outputs the generated combined data to the classification unit 13.
[0063] The classification unit 13 acquires training data from the training data database 31. Also, the classification unit 13 acquires operation data from the operation data database 32. Then, the classification unit 13 executes training of the detection model 103, which is a binary classifier, using the feature amounts of the training data and the feature amounts of the operation data (step S106).
[0064] The classification unit 13 also receives the combined data from the combined data generation unit 12. Then, the classification unit 13 inputs the combined data including the feature amounts of the training data and the feature amounts of the operational data to the trained detection model 103, and obtains an output from the detection model 103. As a result, the separation unit 13 estimates whether the original data of the input feature amounts is training data or operational data (step S107).
[0065] Then, the classification unit 13 obtains a tracking accuracy index indicating the degree of tracking of the detection model 103 from the estimation result, and measures the estimation accuracy of the detection model 103 (step S108). After that, the classification unit 13 outputs the measured estimation accuracy of the detection model 103 to the determination unit 14.
[0066] The determination unit 14 receives an input of the estimation accuracy of the detection model 103 from the classification unit 13. Then, when using, for example, an ROC / AUC score as the tracking accuracy index, the determination unit 14 determines whether or not the value of the tracking accuracy index, which is a measurement value of the estimation accuracy, is less than the accuracy threshold (step S109). If the tracking accuracy index is equal to or greater than the accuracy threshold (step S109: No), the determination unit 14 determines that tracking of the change in the operational data due to concept drift has been successful. In this case, the estimation accuracy evaluation process returns to step S102.
[0067] On the other hand, if the tracking accuracy index is less than the accuracy threshold (step S109: Yes), the determination unit 14 determines that tracking of the change in operational data due to concept drift has failed, and detects a failure of the automatic accuracy recovery. Then, the determination unit 14 notifies the notification unit 15 of the detection result of the failure of the automatic accuracy recovery. The notification unit 15 notifies the user, the administrator, or the like of the failure of the automatic accuracy recovery (step S110).
[0068] 6 is a graph showing the change in accuracy of the model fit against concept drift. Graph 210 shows the change in accuracy over time. The vertical axis of graph 220 shows accuracy, and the horizontal axis shows time.
[0069] Moreover, graphs 201 to 206 below graph 220 on the paper surface are graphs that represent the projection results onto the training data and operational data feature space. Graphs 201 to 206 represent the transition of classes 211 and 212 on the feature space over time in graph 210. As operational data changes over time, classes 211 and 212 change as shown in graphs 201 to 206. Graph 201 represents classes in the feature space at the start of operation, and the model is in a state immediately after learning is completed. Thereafter, the classes on the feature space change with operation as shown in graphs 201 to 206.
[0070] At this time, curve 221 represents the change in the estimation accuracy of the model when automatic accuracy recovery is not performed. Also, curve 222 represents the change in the estimation accuracy of the model when automatic accuracy recovery is performed. Region 223 is the start point of learning, and the estimation accuracy is high even without automatic accuracy recovery. After that, when automatic accuracy recovery is not performed, the estimation accuracy of the model deteriorates due to changes in the operational data in graphs 202 to 204 as shown in curve 221. At the time of graph 205, the estimation accuracy temporarily recovers because it happens to return to a state that is almost identical to the state of the original class. On the other hand, when automatic accuracy recovery is performed, as shown in curve 222, tracking of the change in the operational data is successful between graphs 202 to 205, and the estimation accuracy can be maintained at a high level. Furthermore, when the state of graph 206 is reached, the quality of classification differs from that during learning because the decision boundary is not linear, so the estimation accuracy deteriorates rapidly when automatic accuracy recovery is not performed. Furthermore, in the state of graph 206, even if automatic accuracy recovery is performed, the change in the operational data cannot be followed well, and the estimation accuracy deteriorates.
[0071] When such a transition in the estimation accuracy occurs, it has been difficult to measure the estimation accuracy in the past without using the correct label. In contrast, the accuracy evaluation device 1 according to the present embodiment can detect the deterioration of the estimation accuracy on the line 224, and can grasp that the tracking of the change in the operational data is not going well.
[0072] As described above, the accuracy evaluation device according to the present embodiment extracts the features of the training data and the features of the operational data using a model used for estimation, and classifies the extracted features of the training data and the features of the operational data using a binary classifier. If the accuracy of classification is high, the accuracy evaluation device according to the present embodiment determines that tracking of changes in the operational data is not going well and that automatic accuracy recovery has failed.
[0073] This makes it possible to determine whether changes in the operational data have been successfully tracked without using the correct answer label for the operational data. If it is determined that changes in the operational data have not been successfully tracked, the accuracy of the model can be quickly restored, making it possible to maintain the estimation accuracy of the machine learning model.
[0074] (Hardware configuration) 7 is a hardware configuration diagram of the accuracy evaluation device 1. The accuracy evaluation device 1 includes a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94, for example, as shown in FIG.
[0075] The network interface 94 is an interface for communication between the accuracy evaluation device 1 and an external device. The network interface 94 relays communication between the CPU 91 and the machine learning device 2, for example.
[0076] The hard disk 93 is an auxiliary storage device and stores various programs including a program for implementing the functions of the control unit 10 including the feature extraction unit 11, the combination data generation unit 12, the classification unit 13, the determination unit 14, and the notification unit 15 illustrated in FIG.
[0077] The memory 92 is a main storage device. For example, a direct random access memory (DRAM) can be used as the memory 92.
[0078] The CPU 91 reads out various programs from the hard disk 93, and deploys and executes them on the memory 92. In this way, the CPU 91 realizes the functions of the control unit 10 including the feature extraction unit 11, the combination data generation unit 12, the classification unit 13, the determination unit 14, and the notification unit 15 illustrated in FIG. [Explanation of symbols]
[0079] 1. Accuracy evaluation device 2 Machine learning device 10 Control section 11 Feature Extraction Unit 12. Combined Data Generation Unit 13 Classification section 14 Judgment section 15 Notification Department 21 Machine Learning Department 22 Estimation part 23 Auto Repair Department 31 Training Data Database 32 Operational Data Database
Claims
1. generating a second machine learning model by updating parameters of the first machine learning model based on a first feature amount obtained from the first data based on parameters of the first machine learning model; generating a third machine learning model based on first training data including a second feature obtained from second data based on parameters of the second machine learning model and a correct label indicating first information, and second training data including a third feature obtained from third data based on parameters of the second machine learning model and a correct label indicating the second information; Evaluating the second machine learning model based on a predictive accuracy of the generated third machine learning model. An evaluation program that causes a computer to execute a process.
2. The evaluation program according to claim 1, characterized in that the updating process includes a process of estimating a classification of the first data by clustering based on the first feature amount, and updating parameters of the first machine learning model based on third training data in which the classification is used as a correct label for the first data.
3. The evaluation program according to claim 1, characterized in that the third machine learning model is a binary classifier, and the evaluation program causes the computer to execute a process of calculating an accuracy index based on the prediction accuracy when the third machine learning model classifies the second feature and the third feature into a first class corresponding to the first information and a second class corresponding to the second information, and evaluating whether the estimation accuracy of the second machine learning model has deteriorated based on a relationship between the accuracy index and a threshold value.
4. The accuracy index is a receiver operating characteristic (ROC) / area under the curve (AUC) score of the prediction accuracy of the third machine learning model, The process of evaluating includes a process of evaluating that the estimation accuracy of the second machine learning model has deteriorated when the accuracy index is equal to or greater than a threshold.
4. The evaluation program according to claim 3.
5. generating a second machine learning model by updating parameters of the first machine learning model based on a first feature amount obtained from the first data based on parameters of the first machine learning model; generating a third machine learning model based on first training data including a second feature obtained from second data based on parameters of the second machine learning model and a correct label indicating first information, and second training data including a third feature obtained from third data based on parameters of the second machine learning model and a correct label indicating the second information; Evaluating the second machine learning model based on a predictive accuracy of the generated third machine learning model. An evaluation method characterized in that the processing is executed by a computer.
6. generating a second machine learning model by updating parameters of the first machine learning model based on a first feature amount obtained from the first data based on parameters of the first machine learning model; generating a third machine learning model based on first training data including a second feature obtained from second data based on parameters of the second machine learning model and a correct label indicating first information, and second training data including a third feature obtained from third data based on parameters of the second machine learning model and a correct label indicating the second information; a control unit that executes a process of evaluating the second machine learning model based on the prediction accuracy of the generated third machine learning model. An accuracy evaluation device comprising:
Citation Information
Patent Citations
Binary classification learning device, binary classification device, method, and program
JP2017126158A
Method, medium, and apparatus for eye detection
US20060110030A1
Determining validity of machine learning algorithms for datasets
US20200034665A1
Data processing device and method, recognition device, learning data storage device, machine learning device, and program
WO2019150813A1
Classifier evaluation device, classifier evaluation method, and program
WO2020036192A1
