Accuracy monitoring system, accuracy monitoring method, and accuracy monitoring program

Abstract

To stably detect a concept drift generated when a prediction model is operated, without depending on a know-how of a designer.SOLUTION: A prediction model monitor system 2 includes: a data collecting unit 3 for collecting learning data 32A acquired from each apparatus of an operation system 1, a prediction model 31A having learned the learning data 32A, and operational data 33B acquired when the operation system 1 is operated; and a drift detection logic construction unit 5 for constructing a drift detection unit 62 for detecting a concept drift between the prediction model 31A and the operation data 33B. The drift detection unit 62 notifies a user of generation of a concept drift in the operation system 1 in an operational state when the abnormality rate at the time of operation is continuously higher than the abnormality rate at the time of learning.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an accuracy monitoring system, an accuracy monitoring method, and an accuracy monitoring program. [Background technology]

[0002] Prediction models that output predicted values ​​for measurement data from control systems and the like are machine-learned from past measurement data. The accuracy of a predictive model (prediction accuracy) indicates the degree of match between the predicted value of the predictive model and the actual measured value that will be measured in the future, and the closer the predicted value is to the actual measured value, the higher the accuracy. In general, prediction accuracy is highest immediately after learning, and gradually deteriorates during operation.

[0003] Patent Document 1 proposes maintaining prediction accuracy in a prediction model, which is a type of machine learning model, by re-learning the prediction model using operational data when certain conditions are met. In order to maintain prediction accuracy during operation with as little calculation cost as possible, Patent Document 1 calculates the prediction accuracy when re-learning using data acquired during operation (operational data). Then, re-learning is performed when the prediction accuracy is expected to improve after re-learning, or when the prediction accuracy during operation is equal to or lower than a threshold.

[0004] The condition for re-learning the predictive model is to set a threshold for the prediction accuracy during operation, and the condition is used when the prediction accuracy exceeds the threshold (when a deterioration in the model accuracy is detected). However, this condition only captures momentary deterioration in accuracy, and cannot capture long-term trends in accuracy deterioration, and it also reacts to momentary deterioration in accuracy due to noise or outliers. Even if the machine learning model is retrained in response to momentary deterioration in accuracy, the long-term prediction accuracy will not change.

[0005] On the other hand, Non-Patent Document 1 describes a method for evaluating long-term model accuracy using concept drift detection technology for a classification model, which is a type of model different from the prediction model of a machine learning model. A classification model is a model that classifies input data into certain attributes. Concept drift is a change over time in the data input / output relationship of a system learned by a machine learning model. Non-Patent Document 1 detects data changes that are a decrease in model accuracy in a machine learning model, which is one type of concept drift. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2021-184139 A [Non-patent literature]

[0007] [Non-Patent Document 1] Joao Gama et al. "Learning with Drift Detection", Brazilian Symposium on Artificial Intelligence (SBIA 2004), Pages 286-295 Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, the occurrence of concept drift deteriorates the accuracy of the model, so it is important to appropriately detect the occurrence of concept drift.

[0009] FIG. 17 is an explanatory diagram of the prediction model 95 during learning. The production system 92 outputs measurement data such as production volume 93 according to input data 91 and an operation volume input by an operator 94 . The prediction model 95 is the result of learning the relationship between the input data 91, the operation amount input by the worker 94, and the production amount 93.

[0010] FIG. 18 is an explanatory diagram of the prediction model 95 during operation. The prediction model 95 inputs an operation amount to the production system 92 instead of an operator 94, and outputs a predicted production amount 96 that predicts the production amount 93 of the production system 92 for that operation amount. Therefore, the accuracy 97 of the prediction model 95 is calculated to be higher as the difference between the production volume 93 and the predicted value 96 of the production volume is smaller. Here, even if the prediction model 95 itself does not change from the time of learning to the time of operation, changes in the surroundings of the prediction model 95 may result in a deterioration in accuracy 97. Changes in the surroundings of the prediction model 95 include deterioration of equipment in the production system 92, the occurrence of new input data 91 that does not tend to exist during learning, etc.

[0011] In this way, the changes that occur from the time of learning to the time of operation are due to the occurrence of concept drift. In order to deal with the deterioration of the accuracy of the prediction model 95 due to the occurrence of concept drift, a mechanism for maintaining the accuracy of the prediction model 95 during operation is required. However, Non-Patent Document 1 targets a classification model, which is a type of machine learning model, and cannot be directly applied to the prediction model 95. Therefore, a designer must construct logic for converting the prediction result of the prediction model 95 into a format similar to the output result of the classification model according to the case, which is dependent on the know-how of the designer.

[0012] Therefore, a main objective of the present invention is to stably detect concept drift that occurs during operation of a predictive model, without relying on the know-how of designers. [Means for solving the problem]

[0013] In order to solve the above problems, the accuracy monitoring device of the present invention has the following features. The present invention relates to a data collection unit that collects learning data acquired from each device of an operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system; and a drift detection logic constructing unit that constructs a drift detection unit that detects a concept drift between the prediction model and the operational data, The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit; the anomaly detection model calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values ​​in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values ​​in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model; the concept drift detection unit is a processing unit that inputs an anomaly rate of the operational data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operational system during operation; The drift detection unit is characterized in that when the abnormality rate during operation continues to be higher than the abnormality rate during learning, it outputs a signal that a concept drift has occurred in the operational system during operation. Other means will be described later. Effect of the Invention

[0014] According to the present invention, it is possible to stably detect concept drift that occurs during operation of a predictive model without relying on the know-how of a designer. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is an explanatory diagram of each piece of data used when learning a prediction model according to the present embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of each piece of data used when operating a prediction model according to this embodiment. [Diagram 3] 1 is an overall configuration diagram of an accuracy monitoring system according to an embodiment of the present invention. [Figure 4]FIG. 2 is a hardware configuration diagram of the prediction model monitoring system according to the present embodiment. [Diagram 5] 5 is a flowchart showing a process of a data collection unit according to the present embodiment. [Figure 6] 13 is a flowchart showing a process of a drift detection logic construction unit according to the present embodiment. [Figure 7] 13 is a flowchart showing details of the processing of a learning-time accuracy index calculation unit according to the present embodiment. [Figure 8] FIG. 13 is a diagram showing the relationship between absolute error and frequency during learning in this embodiment. [Figure 9] FIG. 13 is a diagram showing the relationship between absolute error and frequency when the accuracy of a prediction model deteriorates during operation of this embodiment. [Figure 10] 10 is a flowchart showing details of a process of an anomaly detection model learning unit according to the present embodiment. [Figure 11] FIG. 2 is an explanatory diagram of a drift index and a drift determination criterion according to the embodiment. [Figure 12] 10 is a flowchart showing details of a process of a drift judgment criterion creation unit according to the present embodiment. [Figure 13] 10 is a flowchart showing a process of an accuracy monitoring unit according to the present embodiment. [Figure 14] 13 is a flowchart showing the process of an operational accuracy index calculation unit according to the present embodiment. [Figure 15] 5 is a flowchart showing a process of a drift detection unit according to the present embodiment. [Figure 16] FIG. 2 is an explanatory diagram of a system data input / output unit according to the embodiment. [Figure 17] FIG. 13 is an explanatory diagram of a prediction model during learning. [Figure 18] FIG. 13 is an explanatory diagram of a prediction model in operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

[0017] First, an overview of main data used in this embodiment will be described with reference to FIGS. FIG. 1 is an explanatory diagram of each piece of data used when learning the prediction model 31A. The data collection unit 3 collects measurement data from each device of the operation system 1 in FIG. 3 during learning of the prediction model 31A as learning data 32A, and also collects the prediction model 31A learned from the learning data 32A. The accuracy index 23A is an index for measuring the accuracy between the prediction model 31A and the learning data 32A. Any accuracy index 23A may be used as long as it can quantitatively evaluate the difference between the objective variable and the prediction result regardless of the positive or negative sign, such as the absolute error, squared value, L1 norm, or L2 norm shown in FIG. In this embodiment, as an example, assume an absolute error between a predicted value of the prediction model 31A and an actual value of the training data 32A. The larger the absolute error value, the lower the accuracy index 23A is evaluated to be (tending to deteriorate).

[0018] The anomaly detection model 24A is a model that classifies the learning data 32A into two categories, normal values ​​and abnormal values, in order to treat the output of the prediction model 31A as the output of a classification model (as a classification problem). By implementing the classification logic of the anomaly detection model 24A in this way, it is no longer necessary to use individual classification logic that has been used by designers according to projects and that depends on the designers' know-how. The anomaly detection model 24A can adopt any model such as Isolation Forest, One-Class SVM (Support Vector Machine), Local Outlier Factor, etc., as long as it can classify the learning data 32A into two categories, normal values ​​and abnormal values. In this embodiment, the anomaly detection model 24A is assumed to be Isolation Forest. In Isolation Forest, the proportion of abnormal values ​​contained in the learning data 32A is specified as an "anomaly rate" as a learning parameter. The anomaly detection model 24A calculates classification logic for classifying each of the learning data 32A and the operational data 33B into a normal value or an abnormal value according to the frequency of the accuracy index 23 defined between the learning data 32A and the predicted value output by the prediction model 31A. As a result, the anomaly detection model 24A is a model that calculates and outputs an anomaly rate at the time of learning indicating the proportion of abnormal values ​​in the learning data 32A and an anomaly rate at the time of operation indicating the proportion of abnormal values ​​in the operational data 33B (FIG. 2).

[0019] The error rate 25A is the probability that an incorrect classification has been made, and is, for example, the anomaly rate output by the anomaly detection model 24A. The error rate 25A is in a data format that can be input to the CD detection unit 28B in Fig. 2 as data for the classification model. In Non-Patent Document 1, the drift index 26A is statistical data indicating a combination of the average value of the error rate 25A and the standard deviation of the error rate 25A, and is in a data format that can be input to the CD detection unit 28B instead of the error rate 25A. Drift criterion 27A is the value of drift index 26A during learning, and indicates the prediction accuracy of prediction model 31A.

[0020] FIG. 2 is an explanatory diagram of each piece of data used when the prediction model 31A is operated. Among the data in Fig. 2, the prediction model 31A, the anomaly detection model 24A, and the drift judgment criterion 27A, which are data with the suffix "A", use the data calculated during the learning in Fig. 1. On the other hand, the operation data 33B, the accuracy index 23B, the error rate 25B, the drift index 26B, and the CD detection unit 28B, which are data with the suffix "B", use the data prepared during operation.

[0021] The data collection unit 3 collects measurement data from each device of the operation system 1 in FIG. 3 during operation of the prediction model 31A as operation data 33B. The accuracy index 23B is an index for measuring the accuracy between the prediction model 31A and the operation data 33B. The error rate 25B is an anomaly rate of the operation data 33B output by the anomaly detection model 24A. The drift index 26B is statistical data of the error rate 25B, similar to the drift index 26A.

[0022] The drift detection unit 62 has the anomaly detection model 24A and the CD detection unit 28B, and detects a concept drift between the prediction model 31A and the operation data 33B. That is, the drift detection unit 62 receives the input of the accuracy index 23B during operation, and outputs a drift detection result (the output result of the CD detection unit 28B) indicating whether or not a concept drift has been detected during operation. When the prediction accuracy of the prediction model 31A during operation is higher than the drift judgment criterion 27A, the drift detection unit 62 may correct the output indicating that a concept drift has occurred by the CD detection unit 28B to an output indicating that a concept drift has not occurred. Furthermore, the drift detection unit 62 may also output the output result of the anomaly detection model 24A. For example, when a state in which the abnormality rate during operation is higher than the abnormality rate during learning continues, the drift detection unit 62 outputs a message indicating that a concept drift has occurred in the operation system 1 during operation.

[0023] The CD detection unit 28B is configured as a means for detecting concept drift accompanied by accuracy degradation from the operation data 33B targeting the classification model as in Non-Patent Document 1. The CD detection unit 28B is a processing unit that inputs the anomaly rate of the operation data 33B output by the anomaly detection model 24A as an error rate 25 in which incorrect classification is performed, and outputs whether or not a concept drift has occurred in the operation system 1 during operation. The CD detection unit 28B outputs a drift detection result indicating whether or not a concept drift is detected during operation by calculating the error rate 25B or the drift index 26B. The CD detection unit 28B may also appropriately correct the drift detection result according to the comparison result between the drift judgment criterion 27A and the drift index 26B during operation. In this embodiment, the CD detection unit 28B is assumed to be the Drift Detection Method described in Non-Patent Document 1. Any method, such as the Early Drift Detection Method or HDDMs (drift diffusion models), can be adopted for the CD detection unit 28B as long as it is a technique for detecting concept drift accompanied by accuracy degradation.

[0024] Hereinafter, the reason why the model for inputting data to the CD detection unit 28B can be the anomaly detection model 24A that receives the output of the prediction model 31A, like the conventional classification model, will be described. The CD detection unit 28B receives data such as a binomial distribution indicating two classification results (correct or failed) as output values ​​of a conventional classification model. The CD detection unit 28B detects concept drift based on a drift index calculated from the failure probability (error rate) of the received classification result.

[0025] On the other hand, the output of the prediction model 31A is a continuous value that may include positive and negative signs and decimal points, such as floating point numbers, and the error rate and drift index that can be obtained using the binomial distribution do not exist in the case of continuous values. Therefore, the anomaly detection model 24A converts the continuous values ​​output by the prediction model 31A into data indicating a classification result such as a binomial distribution, and outputs the error rate 25B of the converted data. Note that the more the prediction accuracy deteriorates, the higher the value of the error rate 25B increases.

[0026] The CD detection unit 28B can detect concept drift from the error rate 25B or drift index 26B reflecting the classification result, in the same way as in the conventional classification model. The drift index 26B is, for example, a binomial distribution that outputs two results, normal and abnormal. In other words, the anomaly detection model 24A can generate data that the CD detection unit 28B can handle by classifying the output result of the prediction model 31A by the drift index 26B.

[0027] 3 is a diagram showing the overall configuration of the accuracy monitoring system. The accuracy monitoring system is configured by connecting an operation system 1 that is to be monitored and a prediction model monitoring system 2 that is to monitor the accuracy monitoring system via a network. The prediction model monitoring system 2 may be realized either in a local environment such as a general PC (accuracy monitoring device) or via a network such as a cloud service. Furthermore, the connection method between the prediction model monitoring system 2 and the operation system 1, and the connection method between the prediction model monitoring system 2 and the interface unit 7 may be established in a local environment or via a network. The prediction model monitoring system 2 includes a data collection unit 3, a storage unit 4, a drift detection logic construction unit 5, and an accuracy monitoring unit 6. The drift detection logic construction unit 5 has a learning-time accuracy index calculation unit 51, an anomaly detection model learning unit 52, and a drift judgment criterion creation unit 53. The accuracy monitoring unit 6 has an operation-time accuracy index calculation unit 61, and a drift detection unit 62.

[0028] The operation system 1 is a system that utilizes a prediction model 31A. The data collection unit 3 acquires the following necessary data from the operation system 1 and stores it in the storage unit 4. -Operational Prediction Model 31A Learning data 32A used to train the predictive model 31A Operational data acquired during operation of operational system 1 33B

[0029] The memory unit 4 stores data required by each component in the prediction model monitoring system 2 and data required for output to the interface unit 7. The drift detection logic construction unit 5 constructs a drift detection unit 62 using the data stored in the memory unit 4. The accuracy monitoring unit 6 detects whether or not the accuracy of the prediction model 31A during operation is degraded, using the data stored in the storage unit 4. That is, the accuracy monitoring unit 6 detects the degradation of the prediction accuracy of the prediction model 31A during operation of the operation system 1, using the drift detection unit 62 constructed by the drift detection logic construction unit 5. Specifically, the accuracy monitoring unit 6 calculates the accuracy index 23 during operation based on the operation data 33B and the prediction result of the prediction model 31A, and inputs the calculated accuracy index 23 to the anomaly detection model 24A, thereby causing the CD detection unit 28B to output whether or not a concept drift has occurred in the operation system 1 during operation. The interface unit 7 is an input / output means capable of inputting parameters for the drift detection logic construction unit 5 and displaying data from the prediction model monitoring system 2.

[0030] FIG. 4 is a hardware configuration diagram of the prediction model monitoring system 2. The prediction model monitoring system 2 is configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, a HDD 904, a communication I / F 905, an input / output I / F 906, and a media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or its abbreviation, an app) loaded into the RAM 902. This program can be distributed via a communication line, or can be recorded on a recording medium 917 such as a CD-ROM and distributed.

[0031] FIG. 5 is a flowchart showing the process of the data collection unit 3. The data collection unit 3 checks whether the prediction model 31A and the learning data 32A obtained when the prediction model 31A is learned are present in the storage unit 4 (S101). If the data are not present, the data collection unit 3 acquires the prediction model 31A and the learning data 32A from the operation system 1 (S102). Next, the data collection unit 3 always acquires the latest operation data 33B from the operation system 1 (S103). Finally, the data collection unit 3 stores the data acquired in S102 and S103 in the storage unit 4 (S104).

[0032] 6 is a flowchart showing the processing of the drift detection logic construction unit 5. This flowchart is started by an input from the interface unit 7. Note that the interface unit 7 may cooperate with another system to obtain a processing start signal for the drift detection logic construction unit 5. First, the training-time accuracy index calculation unit 51 calculates the accuracy index 23A of the prediction model 31A in the training data 32A (S201). Next, anomaly detection model learning unit 52 uses the calculated accuracy index 23A to learn anomaly detection model 24A (S202). Finally, drift judgment criterion creation unit 53 calculates drift index 26A during learning using learning data 32A and anomaly detection model 24A, and creates drift judgment criterion 27A using drift index 26A (S203).

[0033] FIG. 7 is a flowchart showing details of the process (S201) of the learning-time accuracy index calculation unit 51. First, the learning-time accuracy index calculation unit 51 acquires the prediction model 31A and the learning data 32A from the storage unit 4 (S301). Next, the learning-time accuracy index calculation unit 51 extracts explanatory variables at the time of learning from the learning data 32A and inputs the explanatory variables to the prediction model 31A to acquire a prediction result of the prediction model 31A at the time of learning (S302). Thereafter, the learning-time accuracy index calculation unit 51 extracts the learning-time objective variables from the learning data 32A, and calculates the accuracy index 23A using the objective variables and the prediction result of S302 (S303).

[0034] The role that the anomaly detection model learning unit 52 plays in constructing the drift detection unit 62 will be described with reference to FIGS. FIG. 8 is a graph showing the relationship between absolute error and frequency during learning. The anomaly detection model 24A learns how to classify the learning data 32A into an abnormal value DA5 that occurs infrequently and the other normal values ​​DA1-DA4. During learning in Fig. 8, the anomaly detection model 24A generally outputs a lower anomaly rate than during operation in Fig. 9. The anomaly rate is the ratio of the frequency of the abnormal value DA5 to the frequency of all data (normal values ​​DA1-DA4, abnormal value DA5). Moreover, the prediction model 31A is trained so as to minimize the absolute error between the predicted value output by the prediction model 31A and the actual measured value of the training data 32A. Therefore, in FIG. 8, the abnormal value DA5 is data with a larger absolute error than the normal values ​​DA1-DA4.

[0035] FIG. 9 is a graph showing the relationship between the absolute error and the frequency when the accuracy of the prediction model deteriorates during operation. 9, the anomaly detection model 24A classifies the operational data 33B into an abnormal value DB5 that occurs at a low frequency and other normal values ​​DB1-DB4. In other words, the anomaly detection model 24A uses the same classification logic for the learning data 32A and the classification logic for the operational data 33B. However, since the frequency of abnormal value DB5 is higher than the frequency of abnormal value DA5 in Fig. 8, during operation in Fig. 9, anomaly detection model 24A outputs an anomaly rate that is approximately twice as high as the anomaly rate in Fig. 8. The high anomaly rate indicates that prediction model 31A is incompatible with operational data 33B and that the accuracy of prediction model 31A has deteriorated compared to when it was learned.

[0036] In this way, the anomaly detection model 24A that evaluates the prediction model 31A that has been trained to fit the training data 32A outputs a high anomaly rate for the operational data 33B that has a different data tendency (concept drift) from that of the training data 32A. By using this anomaly rate as the error rate 25B, it is possible to apply a CD detection unit 28B such as that described in Non-Patent Document 1 to the prediction model 31A. In order to make the prediction model 31A applicable to the CD detection unit 28B as a classification model, the prediction model monitoring system 2 constructs the anomaly detection model 24A as a pre-processing unit of the CD detection unit 28B. This eliminates the need for customization of classification logic by designers, making it possible to detect deterioration in accuracy without relying on the know-how of designers, and achieving uniform prediction model monitoring performance.

[0037] FIG. 10 is a flowchart showing details of the process (S202) of the anomaly detection model learning unit 52. First, the anomaly detection model learning unit 52 acquires learning parameters of the anomaly detection model 24A to be used from the storage unit 4 (S401). Next, anomaly detection model learning unit 52 learns anomaly detection model 24A using accuracy index 23A calculated by learning-time accuracy index calculation unit 51 (S402). After that, anomaly detection model learning unit 52 stores anomaly detection model 24A in storage unit 4 (S403).

[0038] The role of the drift judgment criterion creation unit 53 in constructing the drift detection unit 62 will be described below. The CD detection unit 28B detects concept drift accompanied by deterioration in accuracy from the operation data 33B. Therefore, even if the prediction model 31A running in the operation system 1 shows higher accuracy than at the time of learning, if the accuracy is deteriorated in the operation data 33B, it is detected as concept drift. However, generally, when the accuracy at the time of operation is higher than the accuracy at the time of learning, it is often not appropriate to detect concept drift accompanied by deterioration in accuracy. Therefore, by creating drift judgment criterion 27A using drift judgment criterion creation section 53, it is possible to prevent drift detection section 62 from detecting concept drift when the accuracy is higher than that during learning.

[0039] In the CD detection unit 28B as in Non-Patent Document 1, the drift index 26B is calculated by processing the operation data 33B, and the transition of the drift index 26B is statistically processed to detect concept drift. For example, in Non-Patent Document 1, the average value and standard deviation of the error rate 25B are set as the drift index 26B. The prediction model monitoring system 2 calculates the drift index 26A during learning and sets it as the drift judgment criterion 27A in the processing of the drift detection unit 62, so that it is possible not to detect concept drift when the accuracy during operation is higher than that during learning.

[0040] FIG. 11 is an explanatory diagram of the drift index 26A and the drift determination criterion 27A. In FIG. 11, the average value of the drift index 26A is set as the drift judgment criterion 27A, but the maximum value, median value, etc. may also be used.

[0041] FIG. 12 is a flowchart showing details of the process (S203) of the drift judgment criterion generating unit 53. First, the drift judgment criterion creation unit 53 calculates the error rate 25A using the calculation result of the accuracy index 23A and the learned anomaly detection model 24A (S501). Next, the drift judgment criterion creation unit 53 calculates the drift index 26A that is matched to the CD detection unit 28B used in the drift detection unit 62 using the calculated error rate 25A (S502). Finally, the drift judgment criterion creation unit 53 calculates a numerical value (average value, maximum value, etc.) that becomes the drift judgment criterion 27A from the drift index 26A, and stores the calculation result in the storage unit 4 (S503).

[0042] FIG. 13 is a flowchart showing the process of the accuracy monitoring unit 6. First, the accuracy monitoring unit 6 calculates the accuracy index 23B of the prediction model 31A during operation (S601), and the drift detection unit 62 detects drift from the calculated accuracy index 23B and data acquired from the storage unit 4 (S602). Next, the accuracy monitoring unit 6 stores the following example data processed by the accuracy monitoring unit 6 in the storage unit 4 (S603). The operational accuracy index 23B calculated by the operational accuracy index calculation unit 61 - Concept drift detection result calculated by the drift detection unit 62

[0043] Next, the accuracy monitoring unit 6 checks whether or not a signal to end accuracy monitoring has been input (S604). The signal to end accuracy monitoring is input when the operation system 1 is stopped or by the interface unit 7. If a signal to end accuracy monitoring has been input, the accuracy monitoring unit 6 ends its processing. If there is no signal input for terminating accuracy monitoring, the accuracy monitoring unit 6 checks whether the operation data 33B stored in the storage unit 4 has been updated (S605). If the operation data 33B has not been updated, the process returns to S604, and if the operation data 33B has been updated, the process returns to S601.

[0044] FIG. 14 is a flowchart showing the process (S601) of the operational accuracy index calculation unit 61. 14, the processing of learning data 32A (S301 to S303) described in learning-time accuracy index calculation unit 51 shown in FIG. 7 is replaced with processing of operational data 33B (S701 to S703). The operation-time accuracy index calculation unit 61 acquires the prediction model 31A and the operation data 33B from the storage unit 4 (S701). The operation-time accuracy index calculation unit 61 inputs the explanatory variables during operation to the prediction model 31A and acquires the prediction result (S702). The operation-time accuracy index calculation unit 61 calculates the accuracy index 23B from the objective variables during operation and the prediction result (S703).

[0045] FIG. 15 is a flowchart showing the process of the drift detection unit 62. First, the drift detection unit 62 acquires information necessary for drift detection from the storage unit 4 (S801). The necessary information in S801 is the anomaly detection model 24A, the operation data 33B, and the drift determination criterion 27A. Next, the drift detection unit 62 inputs the calculation result of the accuracy index 23B calculated by the operational accuracy index calculation unit 61 to the anomaly detection model 24A, and calculates the error rate 25B (S802). Next, the drift detection unit 62 uses the error rate 25B to calculate the drift index 26B to be used in the CD detection unit 28B (S803).

[0046] Next, the drift detection unit 62 inputs the drift indicator 26B to the CD detection unit 28B (S804). Next, the drift detection unit 62 checks whether or not the CD detection unit 28B has detected a concept drift (S805). If it has been detected, the process proceeds to S806, and if it has not been detected, the process proceeds to S807.

[0047] If the process proceeds to S806, a comparison is made to see whether the drift index 26B is greater than the drift judgment criterion 27A (S806). If the drift index 26B is smaller than the drift judgment criterion 27AB, the prediction model accuracy 27B during operation is higher than the prediction model accuracy 27A at the time of learning, so the process proceeds to S807 and the detection result of the accuracy deterioration is set to "no detection". This allows the result of "detection" of a false positive detected by the concept drift technology to be correctly corrected to "no detection". On the other hand, if drift index 26B is greater than drift judgment criterion 27A, the accuracy of the prediction model during operation is lower than that during learning, so the process proceeds to S808, and the detection result of the deterioration in accuracy is set to "detected." The drift detection unit 62 outputs the detection result of the accuracy deterioration from the system data input / output unit 15, and ends the process.

[0048] FIG. 16 is an explanatory diagram of the system data input / output unit 15. As shown in FIG. The drift detection unit 62 displays the results of calculating the accuracy index 23 during operation via the system data input / output unit 15 on the screen as a time series graph as shown in Figure 16, and also displays information indicating the time when the concept drift occurred (time t3 below) on the time series graph. The system data input / output unit 15 can output data stored in the storage unit 4 and input parameters of the prediction model monitoring system 2 to the storage unit 4. As shown in the display tab, the data stored in the storage unit 4 includes the accuracy index 23B of the prediction model 31A, learning data 32A, operation data 33B, parameters such as error rate 25B, drift index 26B, error rate 25B, etc. In the configuration example of Fig. 16, the accuracy index 23B at the time of detection of accuracy deterioration is displayed. 16 shows the data as a graph, but the display method may be freely determined, such as a time-series table format. Also, the data of the prediction model monitoring system 2 may be processed and the results displayed. In the configuration example of FIG. 16, the slope of the accuracy index 23 is used to calculate the point where accuracy degradation begins, and the result is displayed on the graph. An example of parameter input is the learning parameter (anomaly rate) of the Isolation Forest assumed as the anomaly detection model 24A.

[0049] Below, we will explain the notable points on the graph in FIG. The time t1 indicates an outlier in the accuracy index. The drift detection unit 62 does not detect such outliers or momentary accuracy degradations such as noise, and therefore does not detect accuracy degradation at the time t1. Time t2 indicates the time when the accuracy of the prediction model starts to deteriorate. As described above, the method of obtaining time t2 may be a method of using the slope of accuracy index 23B. Time t3 indicates that the drift detection unit 62 detects deterioration in accuracy of the prediction model. The drift detection unit 62 is capable of detecting long-term deterioration in accuracy. When deterioration in accuracy has occurred continuously since time t2, the drift detection unit 62 detects deterioration in accuracy at time t3 and outputs the deterioration to the system data input / output unit 15.

[0050] In the prediction model monitoring system 2 of the present embodiment described above, the drift detection unit 62 for the prediction model 31A is constructed by the drift detection logic construction unit 5. The anomaly detection model 24A of the drift detection unit 62 enables application of the CD detection unit 28B for classification models that does not depend on the know-how of a designer. This makes it possible to detect long-term deterioration in accuracy and efficiently perform re-learning to maintain the accuracy of the predictive model. In addition, by clarifying the conditions for whether or not a predictive model needs to be re-learned, it is possible to improve the explainability and efficiency of maintenance work for predictive models.

[0051] The present invention is not limited to the above-mentioned embodiment, and various modified examples are included. For example, it is possible to link the predictive model monitoring system 2 of the present invention with other systems. The above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

[0052] Furthermore, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. Furthermore, each of the components of the above-described predictive model monitoring system 2 may be implemented in any hardware as long as the respective hardware can transmit and receive information to each other via a network. Furthermore, the processing executed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware. [Explanation of symbols]

[0053] 1. Operational System 2. Prediction model monitoring system (accuracy monitoring device) 3 Data Collection Section 4 Storage section 5. Drift detection logic construction section 6. Precision Monitoring Department 7 Interface section 15 System data input / output section 23 Accuracy index 24A Anomaly Detection Model 25 Error Rate 26 Drift Index 27A Drift Criteria 28B CD detector (concept drift detector) 31A Prediction Model 32A Training Data 33B Operational Data 51 Learning accuracy index calculation unit 52 Anomaly detection model learning unit 53 Drift Judgment Standard Creation Department 61 Operation accuracy index calculation unit 62 Drift detection unit

Claims

1. A data collection unit that collects learning data acquired from each device of an operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system; and a drift detection logic constructing unit that constructs a drift detection unit that detects a concept drift between the prediction model and the operational data, The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit; the anomaly detection model is a model that calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values ​​in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values ​​in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model; the concept drift detection unit is a processing unit that inputs an anomaly rate of the operation data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operation system during operation; The drift detection unit outputs a message indicating that a concept drift has occurred in the operational system during operation when the abnormality rate during operation continues to be higher than the abnormality rate during learning. Accuracy monitoring device.

2. The accuracy monitoring device further includes an accuracy monitoring unit that detects deterioration of prediction accuracy of the prediction model during operation of the operation system by using the drift detection unit constructed by the drift detection logic construction unit, The accuracy monitoring unit calculates the accuracy index during operation based on the operation data and the prediction result of the prediction model, and inputs the calculated accuracy index to the anomaly detection model, thereby causing the concept drift detection unit to output whether or not a concept drift has occurred in the operation system during operation. The accuracy monitoring device according to claim 1 .

3. the drift detection logic construction unit further creates a drift judgment criterion indicating a prediction accuracy of the prediction model during learning, and includes the drift judgment criterion in the drift detection unit; When the prediction accuracy of the prediction model during operation is higher than the drift judgment criterion, the drift detection unit corrects the output indicating that a concept drift has occurred by the concept drift detection unit to an output indicating that a concept drift has not occurred. The accuracy monitoring device according to claim 2.

4. The drift detection unit displays the results of calculating the accuracy index during operation as a time series graph on a screen, and also displays information indicating the time point when a concept drift occurred on the time series graph. The accuracy monitoring device according to claim 2.

5. A method for monitoring accuracy comprising the accuracy monitoring device according to any one of claims 1 to 4 and each device of the operation system. Accuracy monitoring system.

6. The accuracy monitoring device includes a data collection unit and a drift detection logic construction unit. The data collection unit collects learning data acquired from each device of the operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system; The drift detection logic construction unit constructs a drift detection unit that detects concept drift between the predictive model and the operational data; The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit; the anomaly detection model is a model that calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values ​​in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values ​​in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model; the concept drift detection unit is a processing unit that inputs an anomaly rate of the operation data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operation system during operation; The drift detection unit outputs a message indicating that a concept drift has occurred in the operational system during operation when the abnormality rate during operation continues to be higher than the abnormality rate during learning. Accuracy monitoring method.

7. 5. An accuracy monitoring program for causing a computer to function as the accuracy monitoring device according to claim 1.

Images