System and method for evaluating anomaly detection models

Abstract

To calculate the evaluation value for an abnormality detection model so as to reduce the waste of generation of the abnormality detection model.SOLUTION: An evaluating system includes: a model generating unit 1 that generates an abnormality detection model 2 that is a modelized relation between explanatory variable data which is time-series data and the normal range of objective variable data; an evaluation value calculating unit 3 that calculates the evaluation value for the abnormality detecting model 2 on the basis of the estimated result of the normal range of the objective variable output by the abnormality detection model 2 when an explanatory variable for test is input to the abnormality detection model 2 and of actual measured data on the objective variable corresponding to the explanatory variable for test; and an evaluating unit 4 that evaluates the abnormality detection model 2 on the basis of the evaluated value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an anomaly detection technology that estimates a normal range of an anomaly detection target, and more particularly to an evaluation system and evaluation method for an anomaly detection model that estimates the normal range. [Background technology]

[0002] In factories such as chemical plants, sensors are installed to measure the state of processes online, and time-series data acquired from these sensors is accumulated daily. There is a method of using this accumulated time-series data to construct a model that estimates the normal range of an anomaly detection target, and detect anomalies in the target (for example, see Patent Document 1).

[0003] FIG. 9 is a diagram for explaining a conventional anomaly detection method. A plurality of time series data that are input to the anomaly detection model 100 are called explanatory variables, and time series data that is the target of anomaly detection is called the objective variable. In the example of FIG. 9, the flow rate, pressure, temperature, etc. are explanatory variables, and the valve opening is the objective variable. Data used to create the anomaly detection model 100 are called learning data, and data for evaluating the performance of the anomaly detection model 100 are called verification data. As the learning data, data from a period when no abnormality occurs in the target of anomaly detection is used. There can be multiple such periods. As the verification data, data that was not used as the learning data is usually used, but the learning data may also be used.

[0004] The anomaly detection model 100 estimates the range of normal values ​​of the objective variable from the input explanatory variables. This range of normal values ​​has an upper limit and a lower limit. The upper limit and the lower limit are not limited to one set. For example, in Patent Document 1, as shown in FIG. 10, a reliability that takes a value in the range of 0 to 1 can be used to output a plurality of sets of upper and lower limits. In the example of FIG. 10, a reliability of 0 corresponds to the outermost upper and lower limit values, and a reliability of 1 corresponds to the innermost upper and lower limit values.

[0005] 9, anomalies can be detected based on whether the value of the objective variable is within the range of upper and lower limits output from anomaly detection model 100. However, more complicated anomaly detection methods may also be used (see Patent Document 1).

[0006] The conventional anomaly detection method described in Patent Document 1, which estimates the normal range (time series) of the dependent variable using time series data obtained from a sensor or the like, is a model that estimates the normal range, and since there are places with good accuracy and places with poor accuracy when viewed on the time axis, it is difficult to quantify the evaluation index of the anomaly detection model, and humans visually inspect the dependent variable and the shape of the estimated value to judge the quality of the anomaly detection model, resulting in the following issues: (I) Evaluation of the quality of an anomaly detection model becomes subjective and ambiguous. (II) The standard for how much an anomaly detection model should be improved becomes unclear.

[0007] Due to the issues such as (II) above, the person in charge repeatedly engaged in unnecessary trial and error, and not knowing how long they should continue placed a psychological burden on the person in charge. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2016-081352 A Summary of the Invention [Problem to be solved by the invention]

[0009] The present invention has been made to solve the above-mentioned problems, and aims to provide an anomaly detection model evaluation system and evaluation method that can calculate an evaluation value of an anomaly detection model and reduce waste in creating an anomaly detection model. [Means for solving the problem]

[0010] The anomaly detection model evaluation system of the present invention includes a model creation unit configured to create an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of dependent variable data; an evaluation value calculation unit configured to calculate an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimation result and the actual measurement data of the dependent variable corresponding to the explanatory variable for verification, based on an estimation result of the normal range of the dependent variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and actual measurement data of the dependent variable corresponding to the explanatory variable for verification; and an evaluation unit configured to evaluate the anomaly detection model based on the evaluation value. ,before The evaluation value calculation unit calculates three evaluation values: a coverage rate indicating the proportion of actual measurement data of the objective variable included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the actual measurement data of the objective variable and the movement in the normal range; and a width between the estimated upper limit value and the estimated lower limit value. The evaluation unit calculates a comprehensive evaluation value by integrating a plurality of the three evaluation values ​​or a single evaluation value. By comparing with a predetermined threshold value and evaluating the anomaly detection model.

[0011] In addition, the evaluation system for the anomaly detection model of the present invention the anomaly detection system includes a model creation unit configured to create an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of dependent variable data; an evaluation value calculation unit configured to calculate an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimation result of the normal range of the dependent variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and actual measurement data of the dependent variable corresponding to the explanatory variable for verification, and a time-series data evaluation unit configured to evaluate the anomaly detection model based on the evaluation value; The model creation unit creates a plurality of different anomaly detection models. ,before The evaluation value calculation unit calculates, as the evaluation value, three items: a coverage rate indicating a rate at which actual measurement data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the movement of the actual measurement data of the objective variable and the normal range; and a width between the estimated upper limit value and the estimated lower limit value. For each of the anomaly detection models The evaluation unit calculates a comprehensive evaluation value by integrating a plurality of the three evaluation values. for each of the anomaly detection models, and evaluating each of the anomaly detection models by comparing the overall evaluation values ​​for each of the anomaly detection models, or comparing one of the three evaluation values ​​between a plurality of anomaly detection models, The method is characterized in that each of the anomaly detection models is evaluated. In one configuration example of the anomaly detection model evaluation system of the present invention, the anomaly detection model is a neural network.

[0012] Moreover, the method for evaluating an anomaly detection model of the present invention includes a first step of creating an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of dependent variable data; a second step of calculating an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimated result of the normal range of the dependent variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model and actual measurement data of the dependent variable corresponding to the explanatory variable for verification, and a third step of evaluating the anomaly detection model based on the evaluation value. the second step includes a step of calculating, as the evaluation values, three items including a coverage rate indicating a rate at which actual measured data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model, a tracking degree indicating a similarity between the actual measured data of the objective variable and the movement in the normal range, and a width between the estimated upper limit value and the estimated lower limit value; and the third step includes a step of evaluating the anomaly detection model by comparing a comprehensive evaluation value obtained by integrating a plurality of evaluation values ​​out of the three evaluation values ​​or one evaluation value with a predetermined threshold value. It is characterized by the above. Further, a method for evaluating an anomaly detection model of the present invention includes a first step of creating an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of dependent variable data; a second step of calculating an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimation result of the normal range of the dependent variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model and actual measurement data of the dependent variable corresponding to the explanatory variable for verification; and a third step of evaluating the anomaly detection model based on the evaluation value, wherein the first step includes a step of creating a plurality of different anomaly detection models, and the second step includes a step of evaluating the dependent variable output from the anomaly detection model based on the estimation result of the normal range of the dependent variable output from the anomaly detection model and actual measurement data of the dependent variable corresponding to the explanatory variable for verification. The method includes a step of calculating, as the evaluation values ​​for each anomaly detection model, three items: a coverage rate indicating the proportion of actual measured data of the objective variable that falls within a normal range defined by an estimated upper limit value and an estimated lower limit value of a variable; a tracking degree indicating the similarity between the movement of the actual measured data of the objective variable and the normal range; and a width between the estimated upper limit value and the estimated lower limit value; and the third step includes a step of calculating an overall evaluation value for each anomaly detection model by integrating a plurality of evaluation values ​​out of the three evaluation values, and evaluating each of the anomaly detection models by comparing the overall evaluation values ​​for each anomaly detection model, or evaluating each of the anomaly detection models by comparing one evaluation value out of the three evaluation values ​​between a plurality of anomaly detection models. Effect of the Invention

[0013] According to the present invention, by providing an evaluation value calculation unit and an evaluation unit, a quantitative evaluation value of the anomaly detection model can be calculated to evaluate the anomaly detection model, thereby reducing waste in creating the anomaly detection model. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing a configuration of an anomaly detection model evaluation system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a flowchart illustrating the operation of the anomaly detection model evaluation system according to the first embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram for explaining the inclusion rate, which is an example of the evaluation value in the first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating the degree of tracking, which is an example of an evaluation value in the first embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram for explaining the range of an estimated value, which is an example of an evaluation value in the first embodiment of the present invention. [Figure 6] FIG. 6 is a block diagram showing a configuration of an anomaly detection model evaluation system according to the second embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart illustrating the operation of the anomaly detection model evaluation system according to the second embodiment of the present invention. [Figure 8] FIG. 8 is a block diagram showing an example of the configuration of a computer that realizes the anomaly detection model evaluation system according to the first and second embodiments of the present invention. [Figure 9] FIG. 9 is a diagram illustrating a conventional anomaly detection method. [Figure 10] FIG. 10 is a diagram showing estimated values ​​of a response variable according to a conventional anomaly detection model and an actual response variable. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] [Principle of the invention] In this invention, two types of anomaly detection model evaluation systems are assumed: a model correction type and a model selection type. In the model correction type, three evaluation values, which will be described later, are calculated for the anomaly detection model, and the anomaly detection model is evaluated based on a comprehensive evaluation value obtained by integrating multiple evaluation values ​​out of the three evaluation values, or a comparison result between one evaluation value and a corresponding threshold value. By evaluating the anomaly detection model, it is possible to determine whether to correct the anomaly detection model or to end model construction.

[0016] In the model selection type, multiple anomaly detection models are created in advance by changing the anomaly detection model parameters (explanatory variables, learning parameters, etc.), three evaluation values ​​are calculated for each anomaly detection model, and each anomaly detection model is evaluated based on either an overall evaluation value that combines multiple of the three evaluation values ​​or a single evaluation value.By evaluating multiple anomaly detection models, the best anomaly detection model can be selected.

[0017] [First Example] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an evaluation system for an anomaly detection model according to a first embodiment of the present invention. This embodiment is an example showing an example of a model correction type. The evaluation system of this embodiment includes a model creation unit 1 that creates an anomaly detection model 2 that models the relationship between explanatory variables and normal ranges of objective variables, an evaluation value calculation unit 3 that calculates an evaluation value of the anomaly detection model 2 based on an estimation result of the normal range of the objective variable output from the anomaly detection model 2 when explanatory variables for verification are input to the anomaly detection model 2 and actual measurement data of the objective variable corresponding to the explanatory variables for verification, an evaluation unit 4 that evaluates the anomaly detection model 2 based on the evaluation value, and a model parameter correction unit 5 that corrects anomaly detection model parameters based on the evaluation result.

[0018] Fig. 2 is a flowchart explaining the operation of the evaluation system. The anomaly detection model 2 is a mathematical model constructed by software. The model creation unit 1 creates the anomaly detection model 2 by a known modeling technique using a large amount of learning data including explanatory variables X for learning (step S100 in Fig. 2). The learning data includes actual measurement data of the objective variable Y corresponding to the explanatory variable X (actual measurement value of the objective variable Y with respect to the explanatory variable X when the plant to be subjected to anomaly detection is in a normal state).

[0019] Examples of the anomaly detection model 2 include a fuzzy neural network (FNN) and a regression model. As explained in FIG. 9, explanatory variables X that are input to the anomaly detection model 2 include, for example, flow rate, pressure, and temperature. Furthermore, objective variables Y that are output from the anomaly detection model 2 include, for example, a valve opening. It goes without saying that the explanatory variables X and objective variables Y are not limited to these examples. The method of creating the anomaly detection model 2 by the model creation unit 1 (the method of creating a regression model and the learning method of an FNN) is a well-known technique, so a detailed explanation will be omitted.

[0020] The evaluation value calculation unit 3 calculates an evaluation value of the anomaly detection model 2 by using verification data including explanatory variables X for verification (Step S101 in FIG. 2). The verification data includes actual measurement data of a target variable Y corresponding to the explanatory variable X.

[0021] The evaluation value calculation unit 3 calculates three evaluation values: the coverage rate, the degree of tracking, and the range of the estimated value. The coverage rate indicates the proportion of the actual measured data of the objective variable Y that is included in the normal range defined by the estimated upper limit and lower limit values ​​of the objective variable Y output from the anomaly detection model 2. The coverage rate is an evaluation value calculated based on the idea that it is desirable for the actual measured data of the objective variable Y to be included in the normal range of the objective variable Y estimated by the anomaly detection model 2.

[0022] The degree of tracking indicates the similarity between the actual measured data of the response variable Y and the normal range movement of the response variable Y estimated by the anomaly detection model 2. The degree of tracking is an evaluation value calculated based on the idea that it is desirable for the movement of the actual measured data of the response variable Y to be similar to the movement of the estimated value by the anomaly detection model 2.

[0023] Furthermore, it can be said that anomaly detection model 2 is more accurate when the range between the estimated upper limit and estimated lower limit of the objective variable Y output from anomaly detection model 2 is smaller. The range of estimated values ​​is an evaluation value calculated based on the idea that the narrower the range between the estimated upper and lower limits of anomaly detection model 2, the better (i.e., it is a highly accurate model).

[0024] First, a method for calculating the inclusion rate will be described. Fig. 3 shows an example in which a part of the measured data Ya of the objective variable Y (data in the portion of dashed lines 30, 31) deviates from the normal range defined by the estimated upper limit U and the estimated lower limit L. A more appropriate anomaly detection model is one in which the measured data Ya of the objective variable Y is included without deviating from the normal range as much as possible.

[0025] The evaluation value calculation unit 3 calculates the inclusion rate I [%] by the following formula, where n0 is the number of times that the actual measurement data Ya of the objective variable Y corresponding to the explanatory variable X for verification deviates from the range defined by the outermost upper and lower limit values ​​of the normal range of the objective variable Y output from the anomaly detection model 2 in response to the input of the explanatory variable X for verification, and N is the total number of verification data. I = (1 - n0 / N) × 100 (1)

[0026] When validation data (explanatory variable X) is input into anomaly detection model 2, estimates of the upper and lower limits of the normal range of objective variable Y are obtained for each validation data. Equation (1) indicates the proportion of actual measurement data Ya of objective variable Y that is included in the range defined by the outermost upper and lower limits (the range defined by the highest upper limit and the lowest lower limit). Note that the explanatory variable X may include multiple types of data, such as the flow rate, pressure, and temperature, and the total number of validation data N mentioned above is the number of validation data when these multiple types of data are counted as one data.

[0027] In addition, the evaluation value calculation unit 3 may calculate the inclusion rate I [%] by the following formula, where n1 is the number of times that the actual measurement data Ya of the objective variable Y deviates from the range defined by the innermost upper and lower limit values ​​of the normal range of the objective variable Y output from the anomaly detection model 2. I = (1 - n1 / N) × 100 (2)

[0028] Equation (2) indicates the proportion of the measured data Ya of the objective variable Y that is included in the range defined by the innermost upper and lower limit values ​​(the range defined by the lowest upper limit value and the highest lower limit value). According to equation (2), a stricter value is calculated as the evaluation value of the anomaly detection model 2 compared to the case of equation (1).

[0029] Next, a method for calculating the degree of tracking will be described. Fig. 4(A) shows an example in which the estimated value U of the upper limit and the estimated value L of the lower limit of the normal range of the objective variable Y do not track the movement of the actual measurement data Ya of the objective variable Y. Fig. 4(B) shows an example in which the estimated value U of the upper limit and the estimated value L of the lower limit track the movement of the actual measurement data Ya of the objective variable Y. A model in which the estimated value U of the upper limit and the estimated value L of the lower limit track the actual measurement data Ya of the objective variable Y is considered to be an appropriate anomaly detection model.

[0030] The evaluation value calculation unit 3 calculates the degree of tracking F by the following equation, where Ya(t) is the actual measurement data of the objective variable Y at time t and E(t) is the average value (U1(t)+L1(t)) / 2 of the innermost upper limit estimate U1(t) and the innermost lower limit estimate L1(t) of the upper limit estimate U and the lower limit estimate L of the normal range of the objective variable Y at time t. F = Corr(Ya(t),E(t)) (3)

[0031] Corr() in formula (3) is the correlation coefficient. Formula (3) uses the correlation coefficient to evaluate the degree of similarity between the waveform of the measured data Ya of the objective variable Y and the waveform of the normal range defined by the innermost upper and lower limits. Moreover, the evaluation value calculation unit 3 may calculate the degree of tracking F by the following formula. F = 1 / SUM((Ya(t)-E(t)) 2 ) ···(4)

[0032] The SUM() in formula (4) is the total value over a certain period of time. Formula (4) evaluates the degree of similarity between the waveform of the measured data Ya of the objective variable Y and the waveform of the normal range defined by the innermost upper and lower limits by the sum of squared errors.

[0033] Next, a method for calculating the range of the estimated values ​​will be described. Fig. 5(A) shows an example in which the range between the estimated value U of the upper limit and the estimated value L of the lower limit of the normal range of the objective variable Y is wide for the actual measured data Ya of the objective variable Y. Fig. 5(B) shows an example in which the range between the estimated value U of the upper limit and the estimated value L of the lower limit is narrow for the actual measured data Ya of the objective variable Y. A model with a narrower range between the estimated value U of the upper limit and the estimated value L of the lower limit is an appropriate anomaly detection model. In this case, it goes without saying that it is desirable for the actual measured data Ya to be between the estimated value U of the upper limit and the estimated value L of the lower limit.

[0034] The evaluation value calculation unit 3 calculates the width B of the estimates using the following equation, where Ya(t) is the actual measurement data of the objective variable Y at time t, U0(t) is the outermost estimate of the upper limit and L is the outermost estimate of the lower limit of the normal range of the objective variable Y at time t, and W is the range of the measuring instrument that measures the objective variable Y. B=Mean((|U0(t)-Ya(t)|+|Ya(t)-L0(t)|) / W) ···(5)

[0035] The Mean() in formula (5) is the average value over a certain period. Formula (5) shows the average width of the estimated upper and lower limits of the normal range of the objective variable Y when the actual measured data Ya of the objective variable is between the estimated upper limit U and the estimated lower limit L. Moreover, the evaluation value calculation unit 3 may calculate the width B of the estimated values ​​by the following formula. B=Max((|U0(t)-Ya(t)|+|Ya(t)-L0(t)|) / W) (6)

[0036] The (|U0(t)-Ya(t)|+|Ya(t)-L0(t)|) / W inside Mean() in equation (5) may be calculated simply by the following equation. The same applies to equation (6). (U0(t)-L0(t)) / W (7)

[0037] The Max() in Equation (6) represents the maximum value within a certain period. Equation (6) evaluates the part where the width of the estimated values of the upper and lower limits of the normal range of the target variable Y is the largest. In this way, the evaluation value calculation unit 3 calculates the inclusion rate I, the follow-up degree F, and the width B of the estimated value. In the above description, the verification data is described as being different from the learning data, but the learning data may be used as the verification data.

[0038] Next, the evaluation unit 4 evaluates the anomaly detection model 2 based on the evaluation values calculated by the evaluation value calculation unit 3 (step S102 in FIG. 2). Specifically, the evaluation unit 4 evaluates the anomaly detection model 2 by comparing each of the inclusion rate I, the follow-up degree F, and the width B of the estimated value with the corresponding threshold value.

[0039] In this embodiment, when the inclusion rate I is equal to or greater than the threshold value Ith, the evaluation unit 4 determines it as good (○), and when the inclusion rate I is less than the threshold value Ith, it determines it as slightly poor (△). The threshold value Ith is, for example, 100%.

[0040] Also, when the follow-up degree F is equal to or greater than the threshold value Fth1, the evaluation unit 4 determines it as good (○), when the follow-up degree F is less than the threshold value Fth1 and equal to or greater than the threshold value Fth2 (Fth1 > Fth2), it determines it as slightly poor (△), and when the follow-up degree F is less than the threshold value Fth2, it determines it as poor (×). The threshold value Fth1 is, for example, 0.8, and the threshold value Fth2 is, for example, 0.6.

[0041] In addition, when the width B of the estimated value is less than the threshold value Bth1, the evaluation unit 4 determines it as good (○), when the width B of the estimated value is equal to or greater than the threshold value Bth1 and less than the threshold value Bth2 (Bth1 < Bth2), it determines it as slightly poor (△), and when the width B of the estimated value is equal to or greater than the threshold value Bth2, it determines it as poor (×). The threshold value Bth1 is, for example, 0.05, and the threshold value Bth2 is, for example, 0.2.

[0042] Then, if the comparison result between the evaluation value and the threshold value satisfies a predetermined evaluation criterion, the evaluation unit 4 judges the anomaly detection model 2 to be good (no improvement required), and if the comparison result between the evaluation value and the threshold value does not satisfy the predetermined evaluation criterion, it judges the anomaly detection model 2 to be bad (improvement required) and outputs the evaluation result (Figure 2, step S103).

[0043] For example, the evaluation criterion may be that all comparison results between the evaluation value and the thresholds are good, or that the comparison results between the degree of tracking F and the thresholds Fth1 and Fth2 are good and that the comparison results between the range B of the estimated values ​​and the thresholds Bth1 and Bth2 are good. The evaluation criterion may also be that the comparison result between any one of the coverage rate I, the degree of tracking F, and the range B of the estimated values ​​and the corresponding threshold is good. The evaluation criterion may be set appropriately depending on the target of anomaly detection. Methods for outputting the evaluation result (pass / fail) include displaying the evaluation result and outputting a signal indicating the evaluation result.

[0044] If the evaluation result of the anomaly detection model 2 is poor (improvement is necessary) (NO in step S104 in FIG. 2), the model parameter correction unit 5 corrects the anomaly detection model parameters (step S105 in FIG. 2). Then, the model creation unit 1 redoes the creation of the anomaly detection model (step S100 in FIG. 2). One method for correcting the anomaly detection model parameters is to apply, for example, a Bayesian optimization algorithm.

[0045] In this manner, the processes of steps S100 to S105 are repeatedly performed until the evaluation result of anomaly detection model 2 becomes good (YES in step S104 in FIG. 2).

[0046] An anomaly detection system (not shown) that detects anomalies in a plant using anomaly detection model 2 can determine whether the object to be detected is normal or not by checking whether the value of the objective variable Y measured by a measuring instrument, as in the conventional case, is within a normal range defined by the estimated upper and lower limit values ​​output from the anomaly detection model 2.

[0047] As described above, in this embodiment, the criteria for the quality of anomaly detection model 2 are quantified, so that it is possible to eliminate personal dependence. Also, in this embodiment, a threshold value is set for the evaluation value and a judgment is made, so that it is possible to determine a criterion for discontinuing the correction of anomaly detection model 2.

[0048] Furthermore, in this embodiment, since the criteria for the quality of the anomaly detection model 2 are quantified, it is possible to automatically correct the anomaly detection model 2. However, automatic correction of the model is not an essential component of the present invention, and an operator may manually correct the model.

[0049] [Second Example] Next, a second embodiment of the present invention will be described. Fig. 6 is a block diagram showing the configuration of an anomaly detection model evaluation system according to the second embodiment of the present invention. This embodiment is an example showing a model selection type. The evaluation system of this embodiment includes a model creation unit 1a that creates a plurality of anomaly detection models 2a with different conditions, an evaluation value calculation unit 3a, an evaluation unit 4a, and a model selection unit 6 that selects the best anomaly detection model 2a based on the evaluation results.

[0050] Fig. 7 is a flowchart explaining the operation of the evaluation system of this embodiment. The model creation unit 1a creates an anomaly detection model 2a by a known modeling technique using a large amount of learning data including explanatory variables X for learning (step S200 in Fig. 7). At this time, the model creation unit 1a creates multiple anomaly detection models 2a with different conditions by using learning data with different types of explanatory variables X, using learning data with different selection ranges for explanatory variables X, or changing anomaly detection model parameters.

[0051] The evaluation value calculation unit 3a calculates an evaluation value for each anomaly detection model 2a using verification data including explanatory variable X for verification (step S201 in FIG. 7). The method of calculating the evaluation value is the same as in the first embodiment. However, in this embodiment, since an evaluation value is calculated for each anomaly detection model 2a with different conditions (type of explanatory variable X and range of explanatory variable X), it goes without saying that it is necessary to use verification data required by the anomaly detection model 2a.

[0052] Next, the evaluation unit 4a evaluates each anomaly detection model 2a based on the evaluation value calculated by the evaluation value calculation unit 3a (Step S202 in FIG. 7). Specifically, the evaluation unit 4a calculates an overall evaluation value (for example, a weighted sum of the coverage rate I, the tracking degree F, and the estimated value width B) by integrating the evaluation values ​​for each anomaly detection model 2a, and evaluates each anomaly detection model 2a by comparing the overall evaluation values.

[0053] If the predetermined weight of the coverage rate I is w1, the predetermined weight of the degree of tracking F is w2, and the predetermined weight of the width B of the estimated value is w3, the evaluation unit 4a calculates the weighted sum S by the following formula. S = I × w1 + F × w2 + B × w3 (8)

[0054] Then, the evaluation unit 4a determines that the anomaly detection model 2a having the largest weighted sum S is the best anomaly detection model 2a, and outputs the evaluation result (Step S203 in FIG. 7). When using the weighted sum S as the overall evaluation value and comparing the magnitudes of the weighted sum S, the weighted sum S can be calculated by setting the weights w1 and w2 as positive values ​​and the weight w3 as a negative value.

[0055] In the above example, the weighted sum of the three evaluation values ​​is used as the overall evaluation value, but the weighted sum of two evaluation values ​​of the inclusion rate I, the tracking degree F, and the range B of the estimated value may be used as the overall evaluation value. Furthermore, the evaluation unit 4a may evaluate each anomaly detection model 2a based on one evaluation value. When the inclusion rate I is used, the anomaly detection model 2a having the maximum inclusion rate I may be determined to be the best anomaly detection model 2a. Similarly, when the tracking degree F is used, the anomaly detection model 2a having the maximum tracking degree F may be determined to be the best anomaly detection model 2a. When the range B of the estimated value is used, the anomaly detection model 2a having the smallest range B of the estimated value may be determined to be the best anomaly detection model 2a.

[0056] The model selection unit 6 sets the anomaly detection model 2a with the best evaluation result as the anomaly detection model to be used in the anomaly detection system (Step S204 in FIG. 7).

[0057] In this way, in this embodiment, the same effect as in the first embodiment can be obtained. In this embodiment, the criteria for the quality of the anomaly detection model 2a are quantified, so that the best anomaly detection model 2a to be used in the anomaly detection system can be automatically selected. However, automatic model selection is not a required component of the present invention, and an operator may manually select a model.

[0058] In the first embodiment, the anomaly detection model 2 is evaluated based on a comparison result of at least one of the three evaluation values ​​with a corresponding threshold value, but the evaluation unit 4 in the first embodiment may calculate a comprehensive evaluation value by integrating a plurality of the three evaluation values ​​as in the present embodiment, and evaluate the anomaly detection model 2 based on a comparison result of this comprehensive evaluation value with a corresponding threshold value. For example, when the comprehensive evaluation value is a weighted sum of the three evaluation values, the evaluation unit 4 may determine the anomaly detection model 2 as good (◯) when the comprehensive evaluation value is equal to or greater than a threshold value Sth1, determine the anomaly detection model 2 as slightly poor (△) when the comprehensive evaluation value is less than the threshold value Sth1 and equal to or greater than a threshold value Sth2 (Sth1>Sth2), and determine the anomaly detection model 2 as poor (×) when the comprehensive evaluation value is less than the threshold value Sth2.

[0059] The evaluation system described in the first and second embodiments can be realized by a computer equipped with a CPU (Central Processing Unit), a storage device, and an interface, and a program that controls these hardware resources. An example of the configuration of this computer is shown in Figure 8.

[0060] The computer includes a CPU 200, a storage device 201, and an interface device (I / F) 202. An anomaly detection system or the like is connected to the I / F 202. In such a computer, a program for implementing the evaluation method of the present invention is stored in the storage device 201. The CPU 200 executes the processes described in the first and second embodiments in accordance with the program stored in the storage device 201. [Industrial Applicability]

[0061] The present invention can be applied to a technique for creating a model for detecting anomalies in a plant. [Explanation of symbols]

[0062] 1, 1a...model creation unit, 2, 2a...anomaly detection model, 3, 3a...evaluation value calculation unit, 4, 4a...evaluation unit, 5...model parameter correction unit, 6...model selection unit.

Claims

1. A model creation unit configured to create an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of objective variable data; an evaluation value calculation unit configured to calculate an evaluation value of the anomaly detection model by aggregating differences at each time point between time-series data of the estimation result and the actual measurement data of the objective variable corresponding to the explanatory variable for verification, based on an estimation result of a normal range of the objective variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and actual measurement data of the objective variable corresponding to the explanatory variable for verification; an evaluation unit configured to evaluate the anomaly detection model based on the evaluation value, the evaluation value calculation unit calculates, as the evaluation value, three items: a coverage rate indicating a rate at which actual measurement data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the actual measurement data of the objective variable and a movement in the normal range; and a width between the estimated upper limit value and the estimated lower limit value; The anomaly detection model evaluation system is characterized in that the evaluation unit evaluates the anomaly detection model by comparing an overall evaluation value that integrates multiple evaluation values ​​among the three evaluation values ​​or a single evaluation value with a predetermined threshold.

2. A model creation unit configured to create an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of objective variable data; an evaluation value calculation unit configured to calculate an evaluation value of the anomaly detection model by aggregating differences at each time point between time-series data of the estimation result and the actual measurement data of the objective variable corresponding to the explanatory variable for verification, based on an estimation result of a normal range of the objective variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and actual measurement data of the objective variable corresponding to the explanatory variable for verification; an evaluation unit configured to evaluate the anomaly detection model based on the evaluation value, The model creation unit creates a plurality of different anomaly detection models, the evaluation value calculation unit calculates, for each anomaly detection model, three evaluation values: a coverage rate indicating a rate at which actual measurement data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the actual measurement data of the objective variable and a movement in the normal range; and a width between the estimated upper limit value and the estimated lower limit value; The anomaly detection model evaluation system is characterized in that the evaluation unit calculates an overall evaluation value for each anomaly detection model by integrating multiple evaluation values ​​out of the three evaluation values, and evaluates each of the anomaly detection models by comparing the overall evaluation values ​​for each anomaly detection model, or evaluates each of the anomaly detection models by comparing one evaluation value out of the three evaluation values ​​between multiple anomaly detection models.

3. 3. The anomaly detection model evaluation system according to claim 1, The anomaly detection model evaluation system is characterized in that the anomaly detection model is a neural network.

4. A first step of creating an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of objective variable data; a second step of calculating an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimation result and the actual measurement data of the objective variable corresponding to the explanatory variable for verification, based on the estimation result of the normal range of the objective variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and the actual measurement data of the objective variable corresponding to the explanatory variable for verification; and a third step of evaluating the anomaly detection model based on the evaluation value; the second step includes a step of calculating, as the evaluation values, three items: a coverage rate indicating a rate at which actual measurement data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the actual measurement data of the objective variable and a movement in the normal range; and a width between the estimated upper limit value and the estimated lower limit value; The method for evaluating an anomaly detection model, wherein the third step includes a step of evaluating the anomaly detection model by comparing an overall evaluation value obtained by integrating multiple evaluation values ​​among the three evaluation values ​​or a single evaluation value with a predetermined threshold value.

5. A first step of creating an anomaly detection model that models the relationship between explanatory variable data, which is time-series data, and a normal range of objective variable data; a second step of calculating an evaluation value of the anomaly detection model by aggregating differences at each time point between the estimation result and the actual measurement data of the objective variable corresponding to the explanatory variable for verification, based on the estimation result of the normal range of the objective variable output from the anomaly detection model when an explanatory variable for verification is input to the anomaly detection model, and the actual measurement data of the objective variable corresponding to the explanatory variable for verification; and a third step of evaluating the anomaly detection model based on the evaluation value; The first step includes creating a plurality of different anomaly detection models; the second step includes a step of calculating, for each of the anomaly detection models, three evaluation values: a coverage rate indicating a rate at which actual measurement data of the objective variable is included in a normal range defined by an estimated upper limit value and an estimated lower limit value of the objective variable output from the anomaly detection model; a tracking degree indicating a similarity between the actual measurement data of the objective variable and a movement in the normal range; and a width between the estimated upper limit value and the estimated lower limit value; The third step includes a step of calculating an overall evaluation value for each anomaly detection model by integrating multiple evaluation values ​​out of the three evaluation values, and evaluating each of the anomaly detection models by comparing the overall evaluation values ​​for each anomaly detection model, or evaluating each of the anomaly detection models by comparing one evaluation value out of the three evaluation values ​​between multiple anomaly detection models.

6. The method for evaluating an anomaly detection model according to claim 4 or 5, A method for evaluating an anomaly detection model, wherein the anomaly detection model is a neural network.

Images