Evaluation system for abnormality detection algorithm
By calculating the ranking correlation coefficient between the exception level data output by the exception detection algorithm and the actual failure time, and generating evaluation values, the existing evaluation methods rely on experience and scenarios are solved, and an efficient and universal evaluation method is realized.
Patent Information
- Application Number
- JP2023145031
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-03-19
AI Technical Summary
The existing anomaly detection algorithm evaluation methods rely on experience and specific application scenarios, lack universality, and the objectivity of the evaluation value is affected by the definition of a specific range.
Evaluation values are generated to evaluate the performance of the algorithm by calculating the ranking correlation coefficient between the exception level data output by the exception detection algorithm and the actual failure time. This method does not rely on specific experience and application scenarios, and can objectively evaluate the performance of the algorithm.
It realizes efficient and universal evaluation of anomaly detection algorithm, and can provide an objective and reliable evaluation method independently of specific application scenarios and experience.
Smart Images
Figure 2025038412000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an evaluation system and method for evaluating anomaly detection algorithms. [Background technology]
[0002] Conventionally, "anomaly detection algorithms" are used to predict anomalies in specific machines. For example, if a machine tool used in a factory suddenly breaks down, in the worst case scenario, it may be necessary to stop production in the entire factory. For this reason, some kind of "anomaly detection algorithm" is used to monitor the machine tool and to predict the failure of the machine tool based on the degree of anomaly detected.
[0003] Conventionally, there are two main methods for verifying whether an anomaly detection algorithm is functioning normally:
[0004] The first evaluation method is to evaluate based on practical knowledge.
[0005] For example, if the machine targeted for failure prediction is a water purification device, an anomaly detection algorithm may be used to detect the flow rate of water flowing into the water purification device using a sensor installed upstream, and predict when the water purification device will fail based on fluctuations in that flow rate.
[0006] The first evaluation method for evaluating the anomaly detection algorithm is to evaluate it against the practical knowledge and experience of a person with knowledge of the failure of the water purification device. This method is to judge, based on experience, whether the flow velocity value detected by the upstream sensor and the anomaly value calculated based on that flow velocity are appropriate in light of empirical values.
[0007] The second evaluation method involves evaluating whether the time from the date and time when the degree of anomaly, calculated using only the detection value, exceeded a specified threshold to the date and time when the failure actually occurred falls within a specified range.
[0008] In the case of the water purification device mentioned above, for example, if the time between the detection of an abnormality in the flow rate by the sensor and the actual occurrence of a breakdown is within one day, the prediction is deemed to have been correct.The anomaly detection algorithm is evaluated as being better the higher the proportion of correctly predicted breakdowns (recall rate). Summary of the Invention [Problem to be solved by the invention]
[0009] All of the above-mentioned evaluation methods have the following problems.
[0010] First, the first method has a problem of lack of versatility. In other words, the evaluation method may change depending on the type of work. In addition, it cannot be applied in the first place when the practical knowledge that enables the evaluation does not exist.
[0011] Next, in the second method, the evaluation value changes depending on how the specific range is determined. If the specific range cannot be objectively determined, the evaluation value will also lack objectivity.
[0012] The present invention has been made in consideration of the above circumstances, and aims to provide a highly versatile method for evaluating anomaly detection algorithms that is not dependent on experience or the type of work. [Means for solving the problem]
[0013] In order to solve the above problems, the inventors of the present invention investigated a method for evaluating an anomaly detection algorithm that is highly versatile and not affected by experience or type of practice, and discovered that when predicting general machine failures using an anomaly detection algorithm, regardless of the type of experience or type of practice, the shorter the number of days from a given point in time to the next failure, the higher the level of anomaly related to the machine failure at that point in time will be calculated. Based on this finding, the inventors of the present invention have arrived at the present invention.
[0014] That is, according to the present invention, the following system, method, and computer software inventions are provided.
[0015] (1) An evaluation system for evaluating anomaly detection algorithms, comprising: An evaluation unit in which the computer outputs an evaluation value of the anomaly detection algorithm based on the observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and the data on the failure date and time. having The evaluation unit is a rank correlation coefficient generating unit for calculating a rank correlation function between the failure date and time data and an observed anomaly degree; an evaluation value output unit that calculates an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; An evaluation system comprising:
[0016] (2) In the evaluation system described in (1) above, moreover, An evaluation target storage unit that stores the anomaly detection algorithm of the evaluation target; a failure date and time data storage unit for storing failure date and time data of a device for which failure is to be predicted; An evaluation system comprising:
[0017] (3) In the evaluation system described in (2) above, moreover, a sensor data storage unit for storing sensor data of a device for which a failure is to be predicted; an observation anomaly degree calculation unit that calculates an observation anomaly degree by the anomaly detection algorithm; The evaluation system according to claim 2, further comprising:
[0018] (4) In the evaluation system according to (1), The observed abnormality degree is an index indicating the possibility of failure of a device for which a failure is to be predicted. 13. An evaluation system comprising:
[0019] (5) An evaluation method for evaluating an anomaly detection algorithm, comprising: an evaluation system configured by a computer executes an evaluation step of outputting an evaluation value of the anomaly detection algorithm based on observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and data on a failure date and time; The evaluation step includes: a rank correlation coefficient generating step of calculating a rank correlation function between the failure date and time data and an observed anomaly degree in the evaluation system; an evaluation value output step of an evaluation system for determining an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; The evaluation method according to claim 1,
[0020] (6) A computer software program for evaluating anomaly detection algorithms, comprising: The method causes the computer to execute an evaluation step of outputting an evaluation value of the anomaly detection algorithm based on observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and data on a failure date and time, The evaluation step includes: a rank correlation coefficient generating step of calculating a rank correlation function between the failure date and time data and an observed anomaly degree in the evaluation system; an evaluation value output step of an evaluation system for determining an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; A computer software program comprising:
[0021] With this configuration, anomaly detection algorithms can be evaluated based only on sensor data and failure data, providing a highly versatile method for evaluating anomaly detection algorithms that is not dependent on experience or type of work.
[0022] Other aspects of the present invention and other configurations will be described in the following embodiments and drawings. [Brief description of the drawings]
[0023] [Figure 1]FIG. 1 is a system configuration diagram of an evaluation system according to an embodiment of the present invention.
[0024] [Diagram 2] Figure 2 is also a flowchart. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0026] (System Configuration) FIG. 1 is a system configuration diagram showing an anomaly detection algorithm evaluation system 1 according to this embodiment, and FIG.
[0027] As shown in FIG. 1, this evaluation system 1 is a computer system in which a data storage unit 6 and a program storage unit 7 are connected to a bus 5 to which a CPU 2, a RAM 3, and an input / output interface 4 are connected.
[0028] The CPU 2, RAM 3, and input / output interface 4 may be general-purpose devices. Of these, the input / output interface 4 may communicate with a failure prediction target machine 8, a sensor 9, and an anomaly detection algorithm 10, and may directly acquire data required for evaluating the anomaly detection algorithm 10 from itself. In addition, the input / output interface 4 may be provided with a user interface for accepting data and commands uploaded by a user and displaying output such as evaluation values to the user.
[0029] In addition, the data storage unit 6 and the program storage unit 7 are actually storage media such as a hard disk or SSD, and each program stored in the program storage unit 7 is configured to function as each component of the present invention by being called up and executed on the RAM 3 by the CPU 2.
[0030] The data storage unit 6 stores an anomaly detection algorithm 10 to be evaluated (corresponding to the anomaly detection algorithm storage unit of the present invention), detection value data 12 of the sensor 9 (corresponding to the sensor detection value data storage unit), and actual failure date and time data 13 of the machine whose failure is predicted (corresponding to the failure date and time data storage unit).The data storage unit 6 also stores an observed anomaly degree 14 and an evaluation value 15 calculated and output by the evaluation system 1.
[0031] The program storage unit 7 stores an observed anomaly degree calculation unit 16 that calls the anomaly detection algorithm 10 and calculates an observed anomaly degree 14 using the sensor detection value data 12, and an evaluation unit 17 that outputs an evaluation value 15 of the anomaly detection algorithm 10 based on the calculated observed anomaly degree 14 and failure date and time data 13. The evaluation unit 17 stores a rank correlation coefficient generation unit 18 that calculates a rank correlation function between the failure date and time data 13 and the observed anomaly degree 14, and an evaluation value output unit 19 that outputs the evaluation value 15 based on the calculated rank correlation coefficient.
[0032] (Data storage section) (Evaluation target algorithm storage section) The anomaly detection algorithm 10 is set by a user of the system 1 by storing the anomaly detection algorithm 10 that the user wishes to evaluate in the data storage unit 6 via the input / output interface 4 .
[0033] The anomaly detection algorithm 10 to be evaluated in this embodiment may be any algorithm that outputs the observed anomaly degree of the failure prediction target machine 8. It is also possible to store a plurality of different algorithms, in which case the evaluation values of the algorithms can be compared.
[0034] (Sensor data storage) The sensor detection value data 12 is a detection value by a sensor 9 attached to the failure prediction target machine 8, and is data to be processed by the abnormality detection algorithm 10.
[0035] This sensor detection value data 12 is preferably time-series data including the sensor detection value and the detection date and time, but may also include other values (statistical values, etc.) This sensor detection value data is preferably acquired by operating the sensor 9 for a certain test period, but the same data prepared by some method may also be read.
[0036] (Failure date and time data storage section) The failure date and time data 13 is data indicating the past date and time when the failure prediction target machine 8 actually failed. The failure referred to here is preferably related to the sensor detection value data 12, but the relationship may not necessarily be clear.
[0037] This failure date and time data 13 is also preferably obtained by operating the device subject to anomaly detection during a test period that overlaps with the period during which the sensor 9 is operated, but the same data prepared by some other method may also be used. Note that if the failure date and time data 13 contains too few failure dates and times, there is a risk that accurate judgment cannot be made, so it is preferable that there are at least 10 or more pieces of data.
[0038] (Program storage section) (Observation Anomaly Calculation Unit) The observed anomaly degree calculation unit 16 stored in the program storage unit 7 calls the anomaly detection algorithm 10 from the data storage unit 6, and calculates the observed anomaly degree 14 using the sensor detection value data 12 (step S1 in the flowchart in FIG. 2). It is preferable that the observed anomaly degree 14 is output as time-series data.
[0039] The observed anomaly degree 14 in this embodiment does not indicate an abnormality in the machine 8 itself that is the subject of failure prediction, but is calculated by the anomaly detection algorithm 10 solely from data observed at the target time, and is not guaranteed to be related to future failures.
[0040] It is preferable that the observed anomaly degree 14 is an index which indicates that the larger the degree, the higher the possibility of failure of the failure prediction target machine 8, but the unit and format of the index are not important. For example, when the Hotelling method is used as the anomaly detection algorithm, the observed anomaly degree can be output as an index which indicates that the larger the degree, the higher the possibility of failure, as described above.
[0041] The evaluation unit 17 outputs an evaluation value of the anomaly detection algorithm 10 based on the observed anomaly degree data calculated by the observed anomaly degree calculation unit 16 and the failure date and time data 13, and has a rank correlation coefficient generation unit 18 and an evaluation value output unit 19.
[0042] The rank correlation coefficient generating unit 18 calculates the rank correlation coefficient between the failure date and time data 13 and the observed anomaly data (step S2 in the flow chart of FIG. 2). Here, the rank correlation coefficient is one of the correlation coefficients in statistics, and in this example, it represents the strength of correlation between two ranks, the rank of the time from the previous failure date to the current failure date, and the rank of the observed anomaly calculated by the algorithm at that time. In this embodiment, the rank is considered as a quantity in the usual sense, and Spearman's rank correlation coefficient ρ is used to calculate the usual correlation coefficient between two ranks. (Kendall's rank correlation coefficient τ, which determines whether the hierarchical relationship between the two ranks is the same or reversed and tabulates the result for each pair, may also be used.) The value of this rank correlation coefficient ρ is between 1 and -1.
[0043] Therefore, for a specific failure, the rank correlation coefficient generating unit 18 calculates the rank of the failure by determining the elapsed time from the previous failure date and time based on the failure date and time data 13, and also calculates the rank of the observed anomaly degree at that time.Then, the correlation between the two ranks is obtained by a specific method (Spearman's rank correlation coefficient ρ in this example), and the value is output.
number
[0044] In this embodiment, the coefficient is multiplied by (-1) and the result is output as the evaluation value H.
[0045] H = -1 × rank correlation coefficient ρ This evaluation value output unit 19 can also output and display the evaluation values 15 for a plurality of anomaly detection algorithms 10 in a comparative manner. Also, a method can be adopted in which the accuracy of the algorithm 10 is evaluated on a five-point scale or output graphically.
[0046] (Effects of this embodiment) According to the embodiment of the present invention described above, the system has been constructed based on the knowledge that when predicting general machine failures using an anomaly detection algorithm, the shorter the number of days between one failure and the next, the higher the degree of anomaly related to the machine failure is calculated to be. This makes it possible to provide a highly versatile evaluation system for anomaly detection algorithm that is not affected by experience or the type of work.
[0047] It should be noted that the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit and scope of the present invention. [Explanation of symbols]
[0048] 1. Anomaly detection algorithm evaluation system 2. CPU 3. RAM 4... Input / Output Interface 5. Bus 6. Data storage section 7…Program storage section 8. Machines subject to failure prediction 9…Sensor 10. Anomaly detection algorithm 12...Sensor detection value data 13…Failure date and time data 14…Anomaly level 15…Evaluation value 16…Observation anomaly calculation section 17…Evaluation section 18…Rank correlation coefficient generator 19...Evaluation value output section
Claims
1. An evaluation system for evaluating an anomaly detection algorithm, comprising: An evaluation unit in which the computer outputs an evaluation value of the anomaly detection algorithm based on the observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and the data on the failure date and time. having The evaluation unit is a rank correlation coefficient generating unit for calculating a rank correlation function between the failure date and time data and an observed anomaly degree; an evaluation value output unit that calculates an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; An evaluation system comprising:
2. 2. The evaluation system according to claim 1, moreover, An evaluation target storage unit that stores the anomaly detection algorithm of the evaluation target; a failure date and time data storage unit for storing failure date and time data of a device for which failure is to be predicted; An evaluation system comprising:
3. 3. The evaluation system according to claim 2, moreover, a sensor data storage unit for storing sensor data of a device for which a failure is to be predicted; an observation anomaly degree calculation unit that calculates an observation anomaly degree by the anomaly detection algorithm; An evaluation system comprising:
4. 2. The evaluation system according to claim 1, The observed abnormality degree is an index indicating the possibility of failure of a device for which a failure is to be predicted.
13. An evaluation system comprising:
5. 1. An evaluation method for evaluating an anomaly detection algorithm, comprising: an evaluation system configured by a computer executes an evaluation step of outputting an evaluation value of the anomaly detection algorithm based on observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and data on a failure date and time; The evaluation step includes: a rank correlation coefficient generating step of calculating a rank correlation function between the failure date and time data and an observed anomaly degree in the evaluation system; an evaluation value output step of an evaluation system for determining an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; The evaluation method according to claim 1,
6. The evaluation method according to claim 5, moreover, an evaluation target storage step of storing the anomaly detection algorithm of the evaluation target; A failure date and time data storage step for storing failure date and time data of a device for which a failure is to be predicted; The evaluation method according to claim 1,
7. The evaluation method according to claim 6, moreover, A sensor data storage step of storing sensor data of a device for which a failure is to be predicted; an observation anomaly degree calculation step of calculating an observation anomaly degree by the anomaly detection algorithm; The evaluation method according to claim 1,
8. The evaluation method according to claim 5, The observed abnormality degree is an index indicating the possibility of failure of a device for which a failure is to be predicted.
13. An evaluation method comprising:
9. 1. A computer software program for evaluating an anomaly detection algorithm, comprising: The method causes the computer to execute an evaluation step of outputting an evaluation value of the anomaly detection algorithm based on observed anomaly degree data calculated by the anomaly detection algorithm to be evaluated and data on a failure date and time, The evaluation step includes: a rank correlation coefficient generating step of calculating a rank correlation function between the failure date and time data and an observed anomaly degree in the evaluation system; an evaluation value output step of an evaluation system for determining an evaluation value of the anomaly detection algorithm based on the rank correlation coefficient; A computer software program comprising:
10. 10. The computer software program of claim 9, further comprising: The evaluation method further comprises: an evaluation target storage step of storing the anomaly detection algorithm of the evaluation target; A failure date and time data storage step for storing failure date and time data of a device for which a failure is to be predicted; A computer software program comprising:
11. 11. The computer software program of claim 10, The evaluation method further comprises: A sensor data storage step of storing sensor data of a device for which a failure is to be predicted; an observation anomaly degree calculation step of calculating an observation anomaly degree by the anomaly detection algorithm; A computer software program comprising:
12. 10. The computer software program of claim 9, further comprising: The observed abnormality degree is an index indicating the possibility of failure of a device for which a failure is to be predicted. A computer software program comprising: