Performance evaluation device and method for anomaly detection model
By generating normal and abnormal distributions, calculating index values and combining them with threshold correction, the accuracy problem of performance evaluation of anomaly detection models is solved, and the effectiveness and reliability of the models in real-world environments are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LG ENERGY SOLUTION LTD
- Filing Date
- 2025-09-02
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies struggle to accurately evaluate the performance of anomaly detection models, impacting their effectiveness and false detection rate in real-world environments.
By generating normal and abnormal distributions, calculating normality index and area index, and combining them with threshold correction index, the performance of the anomaly detection model is quantified.
This enables quantitative evaluation of the anomaly detection model's performance, improving the model's accuracy and reliability in real-world environments.
Smart Images
Figure CN122497960A_ABST
Abstract
Description
Technical Field
[0001] This application is based on and claims priority to Korean Patent Application No. 10-2024-0119501, filed with the Korean Intellectual Property Office on September 3, 2024, the entire disclosure of which is incorporated herein by reference.
[0002] This disclosure relates to an apparatus and method for performance evaluation of anomaly detection models, and more specifically, to an apparatus and method for evaluating the classification performance of anomaly detection models. Background Technology
[0003] Anomaly detection is a method used to identify anomalous data that deviates from normal data patterns. For example, anomaly detection aims to use techniques such as statistical analysis, machine learning, and deep learning to detect outliers in data and plays a critical role in various industries including security, finance, manufacturing, and healthcare.
[0004] Anomaly detection methods are broadly categorized into supervised learning, unsupervised learning, and semi-supervised learning. Supervised learning relies on a model trained with both normal and anomalous data to predict whether new data is anomalous. Unsupervised learning is used when labeled data is unavailable; it learns normal patterns and then considers data deviating from those patterns as anomalous. Semi-supervised learning uses a limited amount of anomalous data to train the model.
[0005] Furthermore, various algorithms are used for anomaly detection. Statistical methods detect outliers based on data distribution, while machine learning methods include SVM (Support Vector Machine), K-means, and Random Forest. Deep learning-based methods utilize neural network structures such as autoencoders.
[0006] Evaluating the performance of anomaly detection models is a crucial process for determining how effectively they work in real-world environments. This evaluation is necessary not only to assess the model's ability to accurately detect anomalous data but also to reduce false detections. Therefore, since evaluating the performance of anomaly detection models is essential for a comprehensive assessment of their accuracy, reliability, real-time processing capabilities, and generalization abilities, a more accurate evaluation of their performance is required. Summary of the Invention
[0007] Technical issues
[0008] This disclosure aims to provide an apparatus and method for evaluating the performance of anomaly detection models, which more accurately assesses the performance of anomaly detection models.
[0009] These and other objects and advantages of this disclosure will be understood from the following detailed description and will become even more apparent from exemplary embodiments of the disclosure. Likewise, it will be readily understood that the objects and advantages of this disclosure can be achieved by the means shown in the appended claims and combinations thereof.
[0010] Technical solution
[0011] According to one aspect of this disclosure, a performance evaluation device for an anomaly detection model may include: a distribution calculation unit configured to use the anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset abnormal training data; an index value calculation unit configured to calculate a first index value for the normal distribution and the anomaly distribution based on the normality of the normal distribution and the anomaly distribution, and to calculate a second index value for the area of the normal distribution and the anomaly distribution; and a performance evaluation unit configured to evaluate the performance of the anomaly detection model based on the first index value and the second index value.
[0012] The index value calculation unit can be configured to: calculate a first normality index value for the normal distribution; calculate a second normality index value for the abnormal distribution; and calculate the first index value based on the first normality index value and the second normality index value.
[0013] The index value calculation unit can be configured to: calculate a first correction index value based on the result of comparing a preset threshold with the first normality index value; calculate a second correction index value based on the result of comparing the threshold with the second normality index value; and calculate the first index value based on the first correction index value and the second correction index value.
[0014] The index value calculation unit can be configured to set the first correction index value to a preset reference value when the first normality index value is greater than or equal to the threshold.
[0015] The index value calculation unit can be configured to calculate the first correction index value as a composite value of the first normality index value and the threshold when the first normality index value is less than the threshold.
[0016] The index value calculation unit can be configured to set the second correction index value as a preset reference value when the second normality index value is greater than or equal to the threshold.
[0017] The index value calculation unit can be configured to calculate the second correction index value as a composite value of the second normality index value and the threshold when the second normality index value is less than the threshold.
[0018] The index value calculation unit can be configured to calculate the first index value as the average of the first correction index value and the second correction index value.
[0019] The index value calculation unit can be configured to calculate a first area of the normal distribution, a second area of the abnormal distribution, and a third area of the overlapping region of the normal distribution and the abnormal distribution, and to calculate the second index value based on the first area, the second area, and the third area.
[0020] The index value calculation unit can be configured to calculate the second index value as a value obtained by subtracting the third area from the sum of the first area and the second area.
[0021] The performance evaluation unit can be configured to calculate the performance value of the anomaly detection model based on the first index value and the second index value.
[0022] The performance evaluation unit can be configured to evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.
[0023] The anomaly detection model can be provided in multiple ways.
[0024] The performance evaluation unit can be configured to calculate a performance value for each of a plurality of anomaly detection models, and to evaluate the relative performance of the plurality of anomaly detection models based on the calculated performance values.
[0025] According to another aspect of this disclosure, the server may include a performance evaluation device for an anomaly detection model according to another aspect of this disclosure.
[0026] A performance evaluation method for an anomaly detection model according to another aspect of this disclosure may include the following steps: a distribution calculation step of using the anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset abnormal training data; an index value calculation step of calculating a first index value for the normal distribution and the anomaly distribution based on the normality of the normal distribution and the anomaly distribution, and calculating a second index value for the area of the normal distribution and the anomaly distribution; and a performance evaluation step of evaluating the performance of the anomaly detection model based on the first index value and the second index value.
[0027] According to another aspect of this disclosure, a computer-readable storage medium can store a computer program for executing a performance evaluation method for an anomaly detection model, the performance evaluation method for an anomaly detection model comprising the following steps: a distribution calculation step of using an anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset anomalous training data; an index value calculation step of calculating a first index value for the normal distribution and the anomaly distribution based on the normality of the normal distribution and the anomaly distribution, and calculating a second index value for the area of the normal distribution and the anomaly distribution; and a performance evaluation step of evaluating the performance of the anomaly detection model based on the first index value and the second index value.
[0028] Beneficial effects
[0029] According to one aspect of this disclosure, there is an advantage in being able to quantitatively evaluate the performance of anomaly detection models.
[0030] The effects of this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art based on the description of the claims. Attached Figure Description
[0031] The accompanying drawings illustrate preferred embodiments of the present disclosure and, together with the foregoing disclosure, serve to provide a further understanding of the technical features of the present disclosure; therefore, the present disclosure is not to be construed as limited to the drawings.
[0032] Figure 1 This is a schematic diagram illustrating a performance evaluation apparatus for an anomaly detection model according to an embodiment of the present disclosure.
[0033] Figure 2 This is a schematic diagram illustrating normal and abnormal distributions according to embodiments of the present disclosure.
[0034] Figures 3 to 7 This is a schematic diagram illustrating the operational configuration of a performance evaluation device for an anomaly detection model according to an embodiment of the present disclosure.
[0035] Figure 8 This is a schematic diagram illustrating a performance evaluation method for an anomaly detection model according to another embodiment of the present disclosure. Detailed Implementation
[0036] It should be understood that the terms used in the specification and appended claims should not be construed as limited to their general or dictionary meanings, but rather are based on the principle of allowing the inventors to appropriately define the terms for the best interpretation, and are interpreted based on their meanings and concepts corresponding to the technical aspects of this disclosure.
[0037] Therefore, the description presented herein is merely a preferred example for illustrative purposes and is not intended to limit the scope of this disclosure. It should be understood that other equivalents and modifications may be made thereto without departing from the scope of this disclosure.
[0038] In addition, when describing this disclosure, detailed descriptions of relevant known elements or functions are omitted here if they are deemed to obscure the key subject matter of this disclosure.
[0039] Ordinal terms such as “first” and “second” can be used to distinguish one element from another among various elements, but are not intended to limit these elements by these terms.
[0040] Throughout this specification, when a section is referred to as “containing” or “including” any element, it means that the section may further include other elements, but does not exclude other elements, unless otherwise specifically stated.
[0041] Furthermore, throughout the specification, when one part is referred to as "connected" to another part, it is not limited to the case where they are "directly connected," but also includes the case where they are "indirectly connected" to another element inserted between them.
[0042] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
[0043] Figure 1 This is a schematic diagram illustrating a performance evaluation device 100 for an anomaly detection model according to an embodiment of the present disclosure.
[0044] Reference Figure 1 The performance evaluation device 100 for anomaly detection models may include a distributed calculation unit 110, an index value calculation unit 120, and a performance evaluation unit 130.
[0045] The distribution calculation unit 110 can be configured to use an anomaly detection model to generate a normal distribution for preset normal training data.
[0046] Specifically, the anomaly detection model can be a supervised learning-based model. That is, the anomaly detection model can calculate anomaly scores for labeled normal training data. The distribution calculation unit 110 can simulate a Gaussian distribution for each anomaly score calculated by the anomaly detection model for normal training data using Equation 1 below.
[0047] [Formula 1]
[0048] Here, p(x) is the probability density function, x is a random variable, σ is the standard deviation, and μ is the mean.
[0049] The distribution calculation unit 110 can be configured to generate anomaly distributions for preset anomalous training data.
[0050] Similarly, the anomaly detection model can calculate anomaly scores for the labeled anomalous training data. The distribution calculation unit 110 can use Equation 1 to model the anomaly score of each anomalous training data calculated by the anomaly detection model as a Gaussian distribution.
[0051] Figure 2 This is a schematic diagram illustrating a normal distribution (nb) and an anomaly distribution (ab) according to an embodiment of the present disclosure. Specifically, the normal distribution (nb) and the anomaly distribution (ab) can be represented as an XY plot, where the X-axis is the anomaly score and the Y-axis is set as a probability density function.
[0052] The index value calculation unit 120 can be configured to calculate the first index value of the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution.
[0053] Here, normality refers to the degree to which normal and outlier distributions follow a Gaussian distribution. In other words, the higher the normality, the more the distribution shape corresponds to a Gaussian distribution, and the lower the normality, the more likely the distribution shape deviates from a Gaussian distribution. In other words, the index value calculation unit 120 can calculate the first index value by considering the degree to which normal and outlier distributions follow a Gaussian distribution.
[0054] Specifically, the index value calculation unit 120 can be configured to calculate the first normality index value of the normal distribution.
[0055] The index value calculation unit 120 can use conventional techniques for verifying the normality of a distribution to calculate the first normality index value of the normal distribution. For example, the index value calculation unit 120 can calculate the first normality index value of the normal distribution by verifying the p-value of the Gaussian distribution using the KS test (Kolmogorov-Smirnov test). Here, the p-value of the normal distribution to be calculated corresponds to the first normality index value. That is, the first normality index value is a probability value used to verify whether the normal distribution follows a Gaussian distribution.
[0056] For example, in Figure 2 In one embodiment, the index value calculation unit 120 can calculate the p-value of the normal distribution (nb) to calculate the first normality index value of the normal distribution (nb).
[0057] In addition, the index value calculation unit 120 can be configured to calculate a second normality index value for abnormal distributions.
[0058] The index value calculation unit 120 can calculate a second normality index value for anomaly distributions in response to the method used to calculate a first normality index value. For example, the index value calculation unit 120 can also use the KS test to calculate the second normality index value for anomaly distributions. Here, the calculated p-value of the anomaly distribution corresponds to the second normality index value. In other words, the second normality index value is a probability value used to test whether anomaly distributions follow a Gaussian distribution.
[0059] For example, in Figure 2 In one embodiment, the index value calculation unit 120 can calculate the p-value of the outlier distribution (ab) to calculate the second normality index value of the outlier distribution (ab).
[0060] The index value calculation unit 120 can be configured to calculate the first index value based on the first normality index value and the second normality index value.
[0061] Specifically, the first index value calculated by the index value calculation unit 120 is the value of both a first normality index value considering a normal distribution and a second normality index value considering an anomaly distribution. That is, normality is used as the first index value to evaluate the output results of the anomaly detection model for normal training data and anomaly training data. Therefore, the index value calculation unit 120 can calculate the first index value by considering both the first normality index value and the second normality index value in order to evaluate the performance of the anomaly detection model.
[0062] The index value calculation unit 120 can be configured to calculate a second index value for regions with normal and abnormal distributions.
[0063] Specifically, the index value calculation unit 120 can also calculate a second index value independently of the first index value to evaluate the performance of the anomaly detection model from various aspects. The second index value is based on the area of the normal distribution and the area of the anomaly distribution, and is a value related to the probability of the normal distribution and the probability of the anomaly distribution. For example, the index value calculation unit 120 can calculate the second index value based on the area of the normal distribution and the area of the anomaly distribution.
[0064] That is, the index value calculation unit 120 can calculate the first index value based on the normality of the normal distribution and the abnormal distribution, and calculate the second index value based on the area (probability) of the normal distribution and the abnormal distribution.
[0065] The performance evaluation unit 130 can be configured to evaluate the performance of the anomaly detection model based on a first index value and a second index value.
[0066] Specifically, the performance evaluation unit 130 can evaluate the performance of the anomaly detection model by considering a first metric value and a second metric value that are independent of each other. More specifically, the performance evaluation unit 130 can quantify the performance of the anomaly detection model based on the first metric value and the second metric value.
[0067] For example, the performance evaluation unit 130 can calculate the performance value of the anomaly detection model based on the first index value and the second index value. Furthermore, the performance evaluation unit 130 can evaluate the performance of the corresponding anomaly detection model based on the calculated performance value.
[0068] The performance evaluation device 100 for anomaly detection models according to embodiments of the present disclosure has the advantage of being able to quantitatively evaluate the performance of anomaly detection models by considering both a first index value related to the normality of normal and anomaly distributions and a second index value related to area.
[0069] Meanwhile, the distributed computing unit 110, index value calculation unit 120, and performance evaluation unit 130 provided in the performance evaluation device 100 for the anomaly detection model can optionally include processors, application-specific integrated circuits (ASICs), other chipsets, logic circuits, registers, communication modems, data processing devices, etc., known in the art, to execute various control logics performed in this disclosure. Furthermore, when the control logic is implemented in software, the distributed computing unit 110, index value calculation unit 120, and performance evaluation unit 130 can be implemented as a set of program modules. In this case, the program modules can be stored in memory and executed by the distributed computing unit 110, index value calculation unit 120, and performance evaluation unit 130. The memory can be located inside or outside the distributed computing unit 110, index value calculation unit 120, and performance evaluation unit 130, or can be connected to the distributed computing unit 110, index value calculation unit 120, and performance evaluation unit 130 through various known methods.
[0070] Furthermore, the performance evaluation device 100 for the anomaly detection model may also include a storage unit 140. The storage unit 140 may store data required for the operation and function of each component of the performance evaluation device 100 for the anomaly detection model, data generated during the execution of operations or functions, etc. The type of storage unit 140 is not particularly limited, as long as it is a known information storage device capable of recording, erasing, updating, and retrieving data. As an example, the information storage device may include RAM, flash memory, ROM, EEPROM, registers, etc. In addition, the storage unit 140 may store program code that defines the processes that can be executed by each component of the performance evaluation device 100 for the anomaly detection model.
[0071] See below. Figures 3 to 5 The implementation method for the index value calculation unit 120 to calculate the first index value will be described in detail here. Figure 3 Steps S30 to S34 and Figure 4 Steps S40 to S44 are independent steps.
[0072] In step S30, the distribution calculation unit 110 can use an anomaly detection model to calculate a normal distribution based on normal training data.
[0073] In step S31, the index value calculation unit 120 can calculate a first normality index value indicating whether the normal distribution calculated by the distribution calculation unit 110 follows a Gaussian distribution.
[0074] In step S32, the index value calculation unit 120 compares a preset threshold with a first normality index value. If the result of step S32 is "yes", then step S33 is executed; if it is "no", then step S34 is executed.
[0075] The index value calculation unit 120 can be configured to calculate a first correction index value based on the result of comparing a preset threshold with a first normality index value.
[0076] Specifically, the index value calculation unit 120 can directly compare the preset threshold with the first normality index value. That is, the index value calculation unit 120 can compare the preset threshold with the first normality index value.
[0077] Here, the threshold is a preset value that serves as a criterion for determining whether a normal distribution follows a Gaussian distribution based on the first normality index value. Specifically, the smaller the first normality index value, the greater the likelihood that the null hypothesis will be rejected, and the normal distribution will be determined to be statistically non-Gaussian. Conversely, the larger the first normality index value, the smaller the likelihood that the null hypothesis will be rejected, and the normal distribution will be determined to be statistically Gaussian. In other words, the threshold can be a preset value that serves as a criterion for determining whether to reject the null hypothesis (or conversely, whether to accept the alternative hypothesis). For example, the threshold could be set to 0.05.
[0078] Alternatively, the index value calculation unit 120 may also calculate the first correction index value based on the comparison result between the threshold and the first normality index value.
[0079] In step S33, the index value calculation unit 120 can set the first correction index value as a reference value. Specifically, if the first normality index value is greater than or equal to a threshold, the index value calculation unit 120 can be configured to set the first correction index value as a preset reference value.
[0080] In step S34, the index value calculation unit 120 can set the first correction index value as a composite value of the first normality index value and the threshold. Specifically, if the first normality index value is less than the threshold, the index value calculation unit 120 can be configured to calculate the first correction index value as a composite value of the first normality index value and the threshold.
[0081] In steps S33 and S34, the index value calculation unit 120 can calculate the first correction index value from the first normality index value using the following formula 2.
[0082] [Equation 2]
[0083] Here, p1' is the first correction index value, p1 is the first normality index value, pr is the preset reference value, and TH is the preset threshold.
[0084] For example, suppose the threshold (TH) is set to 0.05 and the reference value (pr) is set to 1. If the first normality index value (p1) is greater than or equal to 0.05, then the first correction index value (p1') can be calculated as 1. Conversely, if the first normality index value (p1) is less than 0.05, then the first correction index value (p1') can be calculated as a composite value obtained by dividing the first normality index value (p1) by the threshold (TH). That is, when the first normality index value (p1) is less than the threshold (TH), the first correction index value (p1') is calculated as a value less than 1.
[0085] In step S40, the distribution calculation unit 110 can use an anomaly detection model to calculate the anomaly distribution based on the anomaly training data.
[0086] In step S41, the index value calculation unit 120 can calculate a second normality index value indicating whether the abnormal distribution calculated by the distribution calculation unit 110 follows a Gaussian distribution.
[0087] In step S42, the index value calculation unit 120 can compare a preset threshold with a second normality index value. If the result of step S42 is "yes", then step S43 is executed; if it is "no", then step S44 is executed.
[0088] The index value calculation unit 120 can be configured to calculate a second correction index value based on the result of comparing a threshold with a second normality index value.
[0089] Specifically, the index value calculation unit 120 can directly compare the preset threshold with the second normality index value. That is, the index value calculation unit 120 can compare the preset threshold with the second normality index value. Here, the threshold is the same as the threshold (TH) mentioned above.
[0090] In addition, the index value calculation unit 120 can calculate the second correction index value based on the comparison result between the threshold and the second normality index value.
[0091] In step S43, the index value calculation unit 120 can set the second correction index value as a reference value. Specifically, if the second normality index value is greater than or equal to a threshold, the index value calculation unit 120 can be configured to set the second correction index value as a preset reference value.
[0092] In step S44, the index value calculation unit 120 can set the second correction index value as a composite value of the second normality index value and the threshold. Specifically, if the second normality index value is less than the threshold, the index value calculation unit 120 can be configured to calculate the second correction index value as a composite value of the second normality index value and the threshold.
[0093] In steps S43 and S44, the index value calculation unit 120 can use the following formula 3 to calculate the second correction index value based on the second normality index value.
[0094] [Formula 3]
[0095] Here, p2' is the second correction index value, p2 is the second normality index value, pr is the preset reference value, and TH is the preset threshold.
[0096] For example, similar to the previous implementation, assume the threshold (TH) is set to 0.05 and the reference value (pr) is set to 1. If the second normality index value (p2) is 0.05 or greater, the second correction index value (p2') can be calculated as 1. Conversely, if the second normality index value (p2) is less than 0.05, the second correction index value (p2') can be calculated as the value obtained by dividing the second normality index value (p2) by the threshold (TH). That is, if the second normality index value (p2) is less than the threshold (TH), the second correction index value (p2') is calculated as a value less than 1.
[0097] In step S50, the index value calculation unit 120 can be configured to calculate the first index value based on the first correction index value calculated in step S34 or step S35 and the second correction index value calculated in step S44 or step S45.
[0098] The index value calculation unit 120 can calculate the first index value by synthesizing the first corrected index value and the second corrected index value. More specifically, the index value calculation unit 120 can calculate a representative value of the first corrected index value and the second corrected index value as the first index value.
[0099] In one embodiment, the index value calculation unit 120 can be configured to calculate the first index value as the average of the first correction index value and the second correction index value. For example, assuming the first correction index value is 1 and the second correction index value is calculated to be 0.98, the index value calculation unit 120 can calculate the formula "(1 + 0.98) ÷ 2" to calculate the first index value as 0.99.
[0100] Specifically, the index value calculation unit 120 can obtain the statistical test result of the normality of the normal distribution by calculating the first normality index value. In addition, the index value calculation unit 120 can quantify the statistical test result of the normality of the normal distribution by calculating the first correction index value based on the first normality index value and a threshold.
[0101] Similarly, the index value calculation unit 120 can obtain the statistical test result of the normality of the abnormal distribution by calculating the second normality index value. In addition, the index value calculation unit 120 can quantify the statistical test result of the normality of the abnormal distribution by calculating the second correction index value based on the second normality index value and the threshold.
[0102] Finally, the index value calculation unit 120 can quantify the performance of the anomaly detection model in terms of normality by calculating the first index value based on the first correction index value and the second correction index value.
[0103] See below Figure 6 The specific implementation method of the indicator value calculation unit 120 calculating the second indicator value will be described.
[0104] After calculating the normal distribution and the abnormal distribution through steps S30 and S40, the index value calculation unit 120 can calculate the first area, the second area, and the third area in steps S60 to S62, respectively. Here, steps S60 to S62 are independent steps.
[0105] In step S60, the index value calculation unit 120 can calculate the first area of the normal distribution.
[0106] Specifically, the index value calculation unit 120 can calculate the first area by integrating the interior area of the normal distribution. For example, in Figure 2 In one embodiment, the index value calculation unit 120 can calculate the first area by integrating the normal distribution (nb) from ni to nf.
[0107] In step S61, the index value calculation unit 120 can calculate the second area of the abnormal distribution.
[0108] Specifically, the index value calculation unit 120 can calculate the second area by integrating the internal area of the abnormal distribution. For example, in Figure 2In one implementation, the index value calculation unit 120 can calculate the second area by integrating the abnormal distribution (ab) from ai to af.
[0109] In step S62, the index value calculation unit 120 can calculate the third area of the overlapping region of the normal distribution and the abnormal distribution.
[0110] Specifically, the index value calculation unit 120 can first determine the overlapping region where the normal distribution and the abnormal distribution intersect. Then, the index value calculation unit 120 can calculate the third area by integrating the internal area of the overlapping region.
[0111] For example, in Figure 2 In this embodiment, the index value calculation unit 120 can determine the overlapping area (ov) as the area corresponding to the abnormal distribution (ab) from ai to k and the area corresponding to the normal distribution (nb) from k to af. The index value calculation unit 120 can calculate the third area (ov) of the overlapping region by integrating the abnormal distribution (ab) from ai to k and integrating the normal distribution (nb) from k to af.
[0112] In step S63, the index value calculation unit 120 can be configured to calculate the second index value based on the first area calculated in step S60, the second area calculated in step S61, and the third area calculated in step S62.
[0113] Specifically, the third area is the area that overlaps with the first area and the second area. That is, if the third area is greater than 0, then the third area overlaps with the value obtained by adding the first area and the second area. Therefore, the index value calculation unit 120 can be configured to calculate the second index value by subtracting the third area from the value obtained by adding the first area and the second area.
[0114] Finally, refer to Figure 7 This section describes in detail the implementation method of the performance evaluation unit 130 for evaluating the performance of the anomaly detection model.
[0115] In step S70, the performance evaluation unit 130 can be configured to calculate the performance value of the anomaly detection model based on the first index value calculated in step S50 and the second index value calculated in step S63.
[0116] Specifically, the first and second indicator values can be calculated within the range of 0 to 1. The performance evaluation unit 130 can calculate the performance value of the anomaly detection model within the range of 0 to 1 based on the first and second indicator values.
[0117] For example, the performance evaluation unit 130 can calculate the performance value of the anomaly detection model by multiplying the first index value by the second index value.
[0118] As another example, the performance evaluation unit 130 can calculate the average of the first index value and the second index value as the performance value of the anomaly detection model.
[0119] As described above, the second index value corresponds to the area under the normal and anomalous distributions. Here, since the area under the distribution represents probability, the second index value can be considered as the total probability based on the normal and anomalous distributions. Furthermore, the first index value is a value that quantifies the normality of the normal and anomalous distributions. In other words, the performance evaluation unit 130 can quantify the performance value of the anomaly detection model by correcting the total probability based on the normality of the normal and anomalous distributions.
[0120] In step S71, the performance evaluation unit 130 can evaluate the performance of the anomaly detection model based on the calculated performance value.
[0121] In one embodiment, the performance evaluation unit 130 can be configured to absolutely evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.
[0122] Specifically, the performance reference value is a standard value used to absolutely evaluate the performance of the anomaly detection model. The performance evaluation unit 130 can evaluate and classify the performance of the anomaly detection model by comparing the performance value with the performance reference value.
[0123] For example, the performance reference value can be set to a value greater than or equal to 0.5 and less than or equal to 1. More preferably, the performance reference value can be set to a value greater than or equal to 0.8 and less than or equal to 1. If the performance reference value is assumed to be set to 0.8, the performance evaluation unit 130 can evaluate the performance of an anomaly detection model with a calculated performance value greater than or equal to 0.8 as level 1, and can evaluate the performance of an anomaly detection model with a calculated performance value not greater than 0.8 as level 2.
[0124] As another example, multiple performance reference values can be preset. That is, based on multiple performance reference values, the range from 0 or greater to 1 or less can be divided into multiple segment intervals. The performance evaluation unit 130 can evaluate the performance of the anomaly detection model as a level based on the segment interval to which the calculated performance value belongs.
[0125] In another embodiment, multiple anomaly detection models are provided, and the performance evaluation unit 130 can be configured to compare and evaluate the performance of the multiple anomaly detection models.
[0126] Specifically, the performance evaluation unit 130 can be configured to calculate the performance value of each of the multiple anomaly detection models. Furthermore, the performance evaluation unit 130 can be configured to evaluate the relative performance of the multiple anomaly detection models based on the calculated performance values.
[0127] For example, the performance evaluation unit 130 can evaluate the anomaly detection model with relatively superior performance based on the calculated performance values from high to low.
[0128] That is, the performance evaluation device 100 for anomaly detection models according to the embodiments of the present disclosure can evaluate the performance of anomaly detection models in various ways by calculating performance values representing the performance of the anomaly detection models.
[0129] The server according to another embodiment of the present disclosure may include a performance evaluation device 100 for an anomaly detection model according to an embodiment of the present disclosure.
[0130] For example, the server can connect to the outside via wired and / or wireless communication. Furthermore, the server can receive and store normal and abnormal training data from the outside. Additionally, the server can update the normal and abnormal training data periodically or irregularly.
[0131] The server can evaluate the performance of the anomaly detection model based on both normal and abnormal training data, and store the performance evaluation results. Furthermore, the server can provide the performance evaluation results of the anomaly detection model upon external request or when a specific event occurs.
[0132] In other words, since the performance evaluation results stored on the server can be referenced by various external devices that can access the server, anomaly detection models with excellent performance evaluation results can be preferentially adopted in various devices or industrial environments that require anomaly detection.
[0133] Figure 8 This is a schematic diagram illustrating a performance evaluation method for an anomaly detection model according to another embodiment of the present disclosure.
[0134] Reference Figure 8 The performance evaluation method for anomaly detection models may include a distributed calculation step (S100), an index value calculation step (S200), and a performance evaluation step (S300).
[0135] Preferably, each step of the performance evaluation method for the anomaly detection model can be performed by the performance evaluation device 100 for the anomaly detection model. In the following text, for ease of explanation, content overlapping with the previously described content will be omitted or briefly described.
[0136] The distribution calculation step (S100) is a step of generating a normal distribution for preset normal training data and generating an abnormal distribution for preset abnormal training data using an anomaly detection model, and can be executed by the distribution calculation unit 110.
[0137] For example, the distributed computing unit 110 can simulate the anomaly score calculated by the anomaly detection model for each normal training data as a Gaussian distribution. Furthermore, the distributed computing unit 110 can simulate the anomaly score calculated by the anomaly detection model for each anomalous training data as a Gaussian distribution.
[0138] The index value calculation step (S200) is a step of calculating the first index value of the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and calculating the second index value of the area of the normal distribution and the abnormal distribution, and can be executed by the index value calculation unit 120.
[0139] Specifically, the index value calculation unit 120 can calculate a first normality index value for a normally distributed system and a second normality index value for an outlier system. Furthermore, the index value calculation unit 120 can calculate a first index value based on the first and second normality index values.
[0140] Furthermore, the index value calculation unit 120 can calculate the first area of the normal distribution, the second area of the abnormal distribution, and the third area of the overlapping region of the normal and abnormal distributions. Additionally, the index value calculation unit 120 can calculate the second index value by subtracting the third area from the sum of the first and second areas.
[0141] The performance evaluation step (S300) is a step for evaluating the performance of the anomaly detection model based on the first index value and the second index value, and can be performed by the performance evaluation unit 130.
[0142] For example, the performance evaluation unit 130 can absolutely evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.
[0143] As another example, the performance evaluation unit 130 can evaluate the performance of multiple anomaly detection models in a relative manner.
[0144] The embodiments of this disclosure described above can be implemented not only by devices and methods, but also by a program that implements functions corresponding to the configuration of the embodiments of this disclosure, or by a recording medium on which the program is recorded. Based on the description of the embodiments described above, those skilled in the art to which this disclosure pertains can easily implement such implementations.
[0145] Another embodiment of this disclosure may provide a computer-readable storage medium on which programs for executing the various embodiments described above are recorded.
[0146] The program can be implemented as hardware components, software components, and / or a combination of hardware and software components. The program can be executed by any system capable of executing computer-readable instructions.
[0147] Software may include computer programs, code, instructions, or combinations thereof, which may configure processing equipment to perform desired operations or may independently or jointly command processing equipment.
[0148] Software can be implemented as a computer program comprising instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disks, hard disks, etc.) and optical storage media (e.g., CD-ROMs, DVDs, etc.). Computer-readable storage media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The storage medium can be computer-readable, stored in memory, and executed by a processor.
[0149] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently on the storage medium and cases where data is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored.
[0150] In addition, the program can be provided as part of a computer program product. The computer program product can then be traded as a commodity between sellers and buyers.
[0151] Computer program products may include software programs and computer-readable storage media storing the software programs. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of electronic devices or through an electronic marketplace. For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be the storage medium of a server belonging to the manufacturer of the electronic devices, a server of an electronic marketplace, or a relay server temporarily storing the software program.
[0152] The present disclosure has been described in detail. However, although preferred embodiments of the present disclosure have been pointed out, the detailed description and specific examples have been given by way of illustration only, as various changes and modifications within the scope of the present disclosure will become apparent to those skilled in the art based on this detailed description.
[0153] Furthermore, without departing from the technical aspects of this disclosure, those skilled in the art can make many substitutions, modifications and changes to the disclosure described above, and this disclosure is not limited to the above embodiments and drawings, and each embodiment can be selectively combined in part or in whole to allow for various modifications.
[0154] (Explanation of reference numerals in the attached image)
[0155] 100: Performance evaluation equipment for anomaly detection models
[0156] 110: Distributed Computing Department
[0157] 120: Indicator Value Calculation Department
[0158] 130: Performance Evaluation Department
[0159] 140: Storage Department
Claims
1. A performance evaluation device for an anomaly detection model, the performance evaluation device for the anomaly detection model comprising: A distributed computing unit is configured to use an anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset abnormal training data. The index value calculation unit is configured to calculate a first index value for the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and to calculate a second index value for the area of the normal distribution and the abnormal distribution. as well as A performance evaluation unit is configured to evaluate the performance of the anomaly detection model based on the first index value and the second index value.
2. The performance evaluation device for anomaly detection models according to claim 1, wherein, The index value calculation unit is also configured to: Calculate the first normality index value for the normal distribution; Calculate the second normality index value for the said anomalous distribution; and The first index value is calculated based on the first normality index value and the second normality index value.
3. The performance evaluation device for anomaly detection models according to claim 2, wherein The index value calculation unit is also configured to: The first correction index value is calculated based on the result of comparing the preset threshold with the first normality index value. The second correction index value is calculated based on the result of comparing the threshold with the second normality index value; as well as The first index value is calculated based on the first correction index value and the second correction index value.
4. The performance evaluation device for anomaly detection models according to claim 3, wherein The index value calculation unit is also configured to: When the first normality index value is greater than or equal to the threshold, the first correction index value is set to a preset reference value; as well as When the first normality index value is less than the threshold, the first correction index value is calculated as a composite value of the first normality index value and the threshold.
5. The performance evaluation device for anomaly detection models according to claim 3, wherein, The index value calculation unit is also configured to: When the value of the second normality index is greater than or equal to the threshold, the value of the second correction index is set to a preset reference value; as well as When the second normality index value is less than the threshold, the second correction index value is calculated as a composite value of the second normality index value and the threshold.
6. The performance evaluation device for anomaly detection models according to claim 3, in, The index value calculation unit is configured to calculate the first index value as the average of the first correction index value and the second correction index value.
7. The performance evaluation device for anomaly detection models according to claim 1, in, The index value calculation unit is configured to calculate a first area of the normal distribution, a second area of the abnormal distribution, and a third area of the overlapping region of the normal distribution and the abnormal distribution, and to calculate the second index value based on the first area, the second area, and the third area.
8. The performance evaluation device for anomaly detection models according to claim 7, in, The index value calculation unit is configured to calculate the second index value as a value obtained by subtracting the third area from the sum of the first area and the second area.
9. The performance evaluation device for anomaly detection models according to claim 1, in, The performance evaluation unit is configured to calculate the performance value of the anomaly detection model based on the first index value and the second index value.
10. The performance evaluation device for anomaly detection models according to claim 9, in, The performance evaluation unit is configured to evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.
11. The performance evaluation device for anomaly detection models according to claim 9, in, The anomaly detection model is configured as multiple, and The performance evaluation unit is configured to calculate a performance value for each of a plurality of anomaly detection models, and to evaluate the relative performance of the plurality of anomaly detection models based on the calculated performance values.
12. A server comprising a performance evaluation device for an anomaly detection model according to any one of claims 1 to 11.
13. A performance evaluation method for an anomaly detection model, the performance evaluation method comprising the following steps: The distribution calculation step uses an anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset abnormal training data. The index value calculation steps involve calculating a first index value for the normal distribution and the abnormal distribution based on their normality, and calculating a second index value for the area of the normal distribution and the abnormal distribution. as well as The performance evaluation step evaluates the performance of the anomaly detection model based on the first indicator value and the second indicator value.
14. A computer-readable storage medium storing a computer program for executing a performance evaluation method for an anomaly detection model, the performance evaluation method comprising the following steps: The distribution calculation step uses an anomaly detection model to generate a normal distribution for preset normal training data and an anomaly distribution for preset abnormal training data. The index value calculation steps involve calculating a first index value for the normal distribution and the abnormal distribution based on their normality, and calculating a second index value for the area of the normal distribution and the abnormal distribution. as well as The performance evaluation step evaluates the performance of the anomaly detection model based on the first indicator value and the second indicator value.