Apparatus and method for evaluating performance of anomaly detection model

Abstract

An apparatus for evaluating the performance of an anomaly detection model, according to one embodiment of the present invention, comprises: a distribution calculation unit which uses an anomaly detection model so as to generate a normal distribution for preset normal learning data and which generates an abnormal distribution for preset abnormal learning data; an index value calculation unit which calculates a first index value for the normal distribution and the abnormal distribution on the basis of normality for the normal distribution and the abnormal distribution, and which calculates a second index value for areas of the normal distribution and the abnormal distribution; and a performance evaluation unit for evaluating the performance of the anomaly detection model on the basis of the first index value and the second index value.

Description

Performance evaluation device and method for anomaly detection model

[0001] This application claims priority to Korean Patent Application No. 10-2024-0119501, filed on September 3, 2024, the entire contents of which are disclosed in the specification and drawings of the said application are incorporated herein by reference.

[0002] The present invention relates to a device and method for evaluating the performance of an anomaly detection model, and more particularly, to a device and method for evaluating the classification performance of an anomaly detection model.

[0003] Anomaly detection is a method for identifying abnormal data that deviates from normal data patterns. For example, anomaly detection aims to detect outliers in data using techniques such as statistical analysis, machine learning, and deep learning, and plays a crucial role in various industries, including security, finance, manufacturing, and healthcare.

[0004] Anomaly detection methodologies are broadly categorized into supervised learning, unsupervised learning, and semi-supervised learning. Supervised learning predicts whether new data is abnormal based on a model trained on normal and abnormal data. Unsupervised learning is used when labeled data is unavailable. It learns normal patterns and then considers any data that deviates from these patterns as abnormal. Semi-supervised learning trains a model using limited anomalous data.

[0005] Additionally, anomaly detection is implemented using a variety of algorithms. Statistical methods detect outliers based on data distribution, while machine learning methods include support vector machines (SVMs), k-means, and random forests. Deep learning-based methods utilize neural network structures such as autoencoders.

[0006] Evaluating the performance of an anomaly detection model is a crucial step in determining how effectively it performs in real-world environments. This evaluation is essential not only to determine the model's ability to accurately detect anomalous data but also to reduce false positives. Consequently, evaluating the performance of anomaly detection models is crucial for comprehensively assessing their accuracy, reliability, real-time processing capabilities, and generalization capabilities. Therefore, more accurate evaluation of the performance of anomaly detection models is crucial.

[0007] The purpose of the present invention is to provide a performance evaluation device and method for an anomaly detection model that more accurately evaluates the performance of the anomaly detection model.

[0008] Other objects and advantages of the present invention can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0009] A performance evaluation device for an anomaly detection model according to one aspect of the present invention may include a distribution generating unit configured to generate a normal distribution for preset normal learning data and an abnormal distribution for preset abnormal learning data using an anomaly detection model; an indicator value generating unit configured to generate a first indicator value for the normal distribution and the abnormal distribution based on normality of the normal distribution and the abnormal distribution, and a second indicator value for the area of ​​the normal distribution and the abnormal distribution; and a performance evaluating unit configured to evaluate the performance of the anomaly detection model based on the first indicator value and the second indicator value.

[0010] The above indicator value calculation unit may be configured to calculate a first normality indicator value for the normal distribution, calculate a second normality indicator value for the abnormal distribution, and calculate the first indicator value based on the first normality indicator value and the second normality indicator value.

[0011] The above-mentioned index value calculation unit may be configured to calculate a first modified index value based on a result of comparing a preset threshold value with the first normality index value, calculate a second modified index value based on a result of comparing the threshold value with the second normality index value, and calculate the first index value based on the first modified index value and the second modified index value.

[0012] The above indicator value calculation unit may be configured to set the first modified indicator value to a preset reference value if the first normality indicator value is greater than or equal to the threshold value.

[0013] The above indicator value calculation unit may be configured to calculate the first modified indicator value as a composite value of the first normality indicator value and the threshold value when the first normality indicator value is less than the threshold value.

[0014] The above indicator value calculation unit may be configured to set the second modified indicator value to a preset reference value if the second normality indicator value is greater than or equal to the threshold value.

[0015] The above indicator value calculation unit may be configured to calculate the second modified indicator value as a composite value of the second normality indicator value and the threshold value when the second normality indicator value is less than the threshold value.

[0016] The above indicator value calculation unit may be configured to calculate the first indicator value as an average value of the first modified indicator value and the second modified indicator value.

[0017] The above indicator value calculation unit may be configured to calculate a first area of ​​the normal distribution, a second area of ​​the abnormal distribution, and a third area of ​​an overlapping region of the normal distribution and the abnormal distribution, and to calculate the second indicator value based on the first area, the second area, and the third area.

[0018] The above index value calculation unit may be configured to calculate a value obtained by subtracting the third area from the sum of the first area and the second area as the second index value.

[0019] The above performance evaluation unit may be configured to calculate a performance value of the anomaly detection model based on the first indicator value and the second indicator value.

[0020] The above performance evaluation unit may be configured to evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.

[0021] The above anomaly detection model may be provided in multiple forms.

[0022] The above performance evaluation unit may be configured to calculate a performance value for each of a plurality of anomaly detection models and to evaluate the relative performance of the plurality of anomaly detection models based on the calculated plurality of performance values.

[0023] A server according to another aspect of the present invention may include a performance evaluation device of an anomaly detection model according to one aspect of the present invention.

[0024] A performance evaluation method of an anomaly detection model according to another aspect of the present invention may include a distribution generation step of generating a normal distribution for preset normal learning data using an anomaly detection model and an abnormal distribution for preset abnormal learning data; an index value generation step of generating a first index value for the normal distribution and the abnormal distribution based on normality of the normal distribution and the abnormal distribution, and a second index value for the area of ​​the normal distribution and the abnormal distribution; and a performance evaluation step of evaluating the performance of the anomaly detection model based on the first index value and the second index value.

[0025] According to another aspect of the present invention, a computer-readable recording medium may store a computer program for executing a performance evaluation method of an anomaly detection model, the method comprising: a distribution generation step of generating a normal distribution for preset normal learning data using an anomaly detection model and generating an abnormal distribution for preset abnormal learning data; an index value generation step of generating a first index value for the normal distribution and the abnormal distribution based on normality of the normal distribution and the abnormal distribution, and a second index value for the area of ​​the normal distribution and the abnormal distribution; and a performance evaluation step of evaluating the performance of the anomaly detection model based on the first index value and the second index value.

[0026] According to one aspect of the present invention, there is an advantage in that the performance of an anomaly detection model can be quantitatively evaluated.

[0027] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0028] The following drawings attached to this specification serve to further understand the technical idea of ​​the present invention together with the detailed description of the invention described below, and therefore the present invention should not be interpreted as being limited to the matters described in such drawings.

[0029] FIG. 1 is a schematic diagram illustrating a performance evaluation device of an anomaly detection model according to one embodiment of the present invention.

[0030] FIG. 2 is a diagram schematically illustrating a normal distribution and an abnormal distribution according to one embodiment of the present invention.

[0031] FIGS. 3 to 7 are schematic diagrams illustrating the operation configuration of a performance evaluation device for an anomaly detection model according to one embodiment of the present invention.

[0032] FIG. 8 is a diagram schematically illustrating a performance evaluation method of an anomaly detection model according to another embodiment of the present invention.

[0033] Terms or words used in this specification and claims should not be interpreted as limited to their usual or dictionary meanings, but should be interpreted as meanings and concepts that conform to the technical spirit of the present invention, based on the principle that the inventor can appropriately define the concept of the term to explain his or her own invention in the best possible manner.

[0034] Accordingly, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the present invention and do not represent all of the technical ideas of the present invention. Therefore, it should be understood that there may be various equivalents and modified examples that can replace them at the time of filing this application.

[0035] In addition, when describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description is omitted.

[0036] Terms that include ordinal numbers, such as first, second, etc., are used to distinguish one of the various components from the rest, and are not used to limit the components by such terms.

[0037] Throughout the specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0038] Additionally, throughout the specification, when we say that a part is "connected" to another part, this includes not only cases where it is "directly connected" but also cases where it is "indirectly connected" with other elements in between.

[0039]

[0040] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

[0041] FIG. 1 is a schematic diagram illustrating a performance evaluation device (100) of an anomaly detection model according to one embodiment of the present invention.

[0042] Referring to FIG. 1, the performance evaluation device (100) of the anomaly detection model may include a distribution calculation unit (110), an indicator value calculation unit (120), and a performance evaluation unit (130).

[0043] The distribution generating unit (110) can be configured to generate a normal distribution for preset normal learning data using an anomaly detection model.

[0044] Specifically, the anomaly detection model may be a supervised learning-based model. That is, the anomaly detection model can calculate anomaly scores for labeled normal training data. The distribution calculation unit (110) can model the anomaly scores for each normal training data produced by the anomaly detection model into a normal distribution using Equation 1 below.

[0045] [Formula 1]

[0046]

[0047] Here, p(x) is the probability density function, x is a random variable, σ is the standard deviation, and μ is the mean.

[0048] The distribution generating unit (110) can be configured to generate an abnormal distribution for preset abnormal learning data.

[0049] Similarly, the anomaly detection model can calculate anomaly scores for labeled abnormal learning data. The distribution calculation unit (110) can simulate the anomaly scores for each abnormal learning data produced by the anomaly detection model into a normal distribution using Equation 1.

[0050] FIG. 2 is a diagram schematically illustrating a normal distribution (nb) and an abnormal distribution (ab) according to one embodiment of the present invention. Specifically, the normal distribution (nb) and the abnormal distribution (ab) can be expressed as an XY graph in which the X-axis represents an abnormality score and the Y-axis is set as a probability density function.

[0051] The indicator value calculation unit (120) can be configured to calculate the first indicator value for the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution.

[0052] Here, normality refers to the degree to which a normal distribution and an abnormal distribution follow a normal distribution. In other words, the higher the normality, the more likely the distribution shape is to conform to a normal distribution. The lower the normality, the more likely the distribution shape is to deviate from a normal distribution. In other words, the index value calculation unit (120) can calculate the first index value by considering the degree to which the normal distribution and the abnormal distribution follow a normal distribution.

[0053] Specifically, the indicator value calculation unit (120) can be configured to calculate a first normality indicator value for a normal distribution.

[0054] The index value calculation unit (120) can calculate the first normality index value for the normal distribution using a conventional technique for verifying the normality of the distribution. For example, the index value calculation unit (120) can calculate the first normality index value for the normal distribution by verifying the p-value of the normal distribution using the KS test (Kolmogorov-Smirnov Test). Here, the calculated p-value of the normal distribution corresponds to the first normality index value. In other words, the first normality index value is a probability value used to test whether the normal distribution follows a normal distribution.

[0055] For example, in the embodiment of FIG. 2, the indicator value calculation unit (120) can calculate the p-value of the normal distribution (nb) to calculate the first normality indicator value for the normal distribution (nb).

[0056] In addition, the indicator value calculation unit (120) can be configured to calculate a second normality indicator value for an abnormal distribution.

[0057] The index value calculation unit (120) can calculate a second normality index value for an abnormal distribution in accordance with the method for calculating the first normality index value. For example, the index value calculation unit (120) can calculate a second normality index value for an abnormal distribution using the KS test. Here, the p-value of the abnormal distribution thus calculated corresponds to the second normality index value. In other words, the second normality index value is a probability value used to test whether the abnormal distribution follows a normal distribution.

[0058] For example, in the embodiment of FIG. 2, the indicator value calculation unit (120) can calculate the p-value of the abnormal distribution (ab) to calculate the second normality indicator value for the abnormal distribution (ab).

[0059] The indicator value calculation unit (120) can be configured to calculate the first indicator value based on the first normality indicator value and the second normality indicator value.

[0060] Specifically, the first index value calculated by the index value calculation unit (120) is a value that takes into account both the first normality index value of the normal distribution and the second normality index value of the abnormal distribution. In other words, the value in which the output result of the anomaly detection model for normal learning data and abnormal learning data is evaluated in terms of normality corresponds to the first index value. Therefore, the index value calculation unit (120) can calculate the first index value by taking into account both the first normality index value and the second normality index value in order to evaluate the performance of the anomaly detection model.

[0061] The index value calculation unit (120) can be configured to calculate a second index value for the area of ​​a normal distribution and an abnormal distribution.

[0062] Specifically, the indicator value calculation unit (120) can further calculate a second indicator value independent of the first indicator value in order to evaluate the performance of the anomaly detection model from various aspects. The second indicator value is based on the area of ​​the normal distribution and the area of ​​the abnormal distribution, and is a value related to the probability of the normal distribution and the probability of the abnormal distribution. For example, the indicator value calculation unit (120) can calculate the second indicator value based on the area of ​​the normal distribution and the area of ​​the abnormal distribution.

[0063] That is, the indicator value calculation unit (120) can calculate the first indicator value based on the normality of the normal distribution and the abnormal distribution, and can calculate the second indicator value based on the area (probability) of the normal distribution and the abnormal distribution.

[0064] The performance evaluation unit (130) can be configured to evaluate the performance of the anomaly detection model based on the first indicator value and the second indicator value.

[0065] Specifically, the performance evaluation unit (130) can evaluate the performance of the anomaly detection model by considering first and second indicator values ​​that are independent of each other. More specifically, the performance evaluation unit (130) can quantify the performance of the anomaly detection model based on the first and second indicator values.

[0066] For example, the performance evaluation unit (130) can calculate the performance value of the anomaly detection model from the first indicator value and the second indicator value. Then, the performance evaluation unit (130) can evaluate the performance of the corresponding anomaly detection model based on the calculated performance value.

[0067] The performance evaluation device (100) of an anomaly detection model according to one embodiment of the present invention has the advantage of being able to quantitatively evaluate the performance of the anomaly detection model by considering both the first indicator value related to the normality of the normal distribution and the abnormal distribution and the second indicator value related to the area.

[0068]

[0069] Meanwhile, the distribution calculation unit (110), the indicator value calculation unit (120), and the performance evaluation unit (130) provided in the performance evaluation device (100) of the anomaly detection model may optionally include a processor, an application-specific integrated circuit (ASIC), another chipset, a logic circuit, a register, a communication modem, a data processing device, etc. known in the art to execute various control logics performed in the present invention. In addition, when the control logic is implemented in software, the distribution calculation unit (110), the indicator value calculation unit (120), and the performance evaluation unit (130) may be implemented as a set of program modules. At this time, the program modules may be stored in a memory and executed by the distribution calculation unit (110), the indicator value calculation unit (120), and the performance evaluation unit (130). The above memory may be located inside or outside the distribution calculation unit (110), the indicator value calculation unit (120), and the performance evaluation unit (130), and may be connected to the distribution calculation unit (110), the indicator value calculation unit (120), and the performance evaluation unit (130) by various well-known means.

[0070] In addition, the performance evaluation device (100) of the anomaly detection model may further include a storage unit (140). The storage unit (140) may store data or programs required for each component of the performance evaluation device (100) of the anomaly detection model to perform operations and functions, or data generated in the process of performing operations and functions. The storage unit (140) is not particularly limited in type as long as it is a known information storage means known to be able to record, erase, update, and read data. As an example, the information storage means may include RAM, flash memory, ROM, EEPROM, registers, etc. In addition, the storage unit (140) may store program codes defining processes executable by each component of the performance evaluation device (100) of the anomaly detection model.

[0071]

[0072] Hereinafter, with reference to FIGS. 3 to 5, an embodiment in which the index value calculation unit (120) calculates the first index value will be specifically described. Here, steps S30 to S34 of FIG. 3 and steps S40 to S44 of FIG. 4 are independent steps.

[0073] In step S30, the distribution generating unit (110) can generate a normal distribution from normal learning data using an anomaly detection model.

[0074] In step S31, the indicator value calculation unit (120) can calculate a first normality indicator value indicating whether the normal distribution calculated by the distribution calculation unit (110) follows a normal distribution.

[0075] In step S32, the indicator value calculation unit (120) can compare the first normality indicator value with a preset threshold value. If the result of step S32 is "YES", step S33 is performed, and if "NO", step S34 is performed.

[0076] The indicator value calculation unit (120) can be configured to calculate the first modified indicator value based on the result of comparing the first normality indicator value with a preset threshold value.

[0077] Specifically, the indicator value calculation unit (120) can directly compare the size between a preset threshold value and the first normality indicator value. That is, the indicator value calculation unit (120) can compare the size of the preset threshold value and the first normality indicator value.

[0078] Here, the threshold value is a preset value that serves as a criterion for determining whether a normal distribution follows a normal distribution, based on the first normality index value. Specifically, a smaller first normality index value indicates a higher likelihood of rejecting the null hypothesis and determining that the normal distribution statistically does not follow a normal distribution. Conversely, a larger first normality index value indicates a higher likelihood of failing to reject the null hypothesis and determining that the normal distribution statistically follows a normal distribution. In other words, the threshold value can be preset as a value that serves as a criterion for determining whether to reject the null hypothesis (or, conversely, whether to accept the alternative hypothesis). For example, the threshold value can be set to 0.05.

[0079] And, the index value calculation unit (120) can calculate the first modified index value from the first normality index value based on the comparison result between the threshold value and the first normality index value.

[0080] In step S33, the indicator value calculation unit (120) may set the first modified indicator value as a reference value. Specifically, if the first normality indicator value is greater than or equal to a threshold value, the indicator value calculation unit (120) may be configured to set the first modified indicator value as a preset reference value.

[0081] In step S34, the indicator value calculation unit (120) may set the first modified indicator value as a composite value of the first normality indicator value and the threshold value. Specifically, if the first normality indicator value is less than the threshold value, the indicator value calculation unit (120) may be configured to calculate the first modified indicator value as a composite value of the first normality indicator value and the threshold value.

[0082] In steps S33 and S34, the indicator value calculation unit (120) can calculate the first modified indicator value from the first normality indicator value using Equation 2 below.

[0083] [Formula 2]

[0084]

[0085] Here, p1' is the first correction index value, p1 is the first normality index value, pr is a preset reference value, and TH is a preset threshold value.

[0086] For example, it is assumed that the threshold value (TH) is set to 0.05 and the reference value (pr) is set to 1. If the first normality index value (p1) is 0.05 or greater, the first correction index value (p1') can be calculated as 1. Conversely, if the first normality index value (p1) is less than 0.05, the first correction index value (p1') can be calculated as a composite value obtained by dividing the first normality index value (p1) by the threshold value (TH). That is, if the first normality index value (p1) is less than the threshold value (TH), the first correction index value (p1') is calculated as a value less than 1.

[0087] In step S40, the distribution generating unit (110) can generate an abnormal distribution from abnormal learning data using an anomaly detection model.

[0088] In step S41, the indicator value calculation unit (120) can calculate a second normality indicator value indicating whether the abnormal distribution calculated by the distribution calculation unit (110) follows a normal distribution.

[0089] In step S42, the indicator value calculation unit (120) can compare a preset threshold value with the second normality indicator value. If the result of step S42 is "YES", step S43 is performed, and if the result is "NO", step S44 is performed.

[0090] The indicator value calculation unit (120) can be configured to calculate a second modified indicator value based on the result of comparing the threshold value and the second normality indicator value.

[0091] Specifically, the indicator value calculation unit (120) can directly compare the size between a preset threshold value and the second normality indicator value. That is, the indicator value calculation unit (120) can compare the size of the preset threshold value and the second normality indicator value. Here, the threshold value is identical to the threshold value (TH) described above.

[0092] In addition, the index value calculation unit (120) can calculate a second modified index value from the second normality index value based on the comparison result between the threshold value and the second normality index value.

[0093] In step S43, the indicator value calculation unit (120) may set the first modified indicator value as a reference value. Specifically, if the second normality indicator value is greater than or equal to a threshold value, the indicator value calculation unit (120) may be configured to set the second modified indicator value as a preset reference value.

[0094] In step S44, the indicator value calculation unit (120) may set the second modified indicator value as a composite value of the second normality indicator value and the threshold value. Specifically, if the second normality indicator value is less than the threshold value, the indicator value calculation unit (120) may be configured to calculate the second modified indicator value as a composite value of the second normality indicator value and the threshold value.

[0095] In steps S43 and S44, the indicator value calculation unit (120) can calculate the second modified indicator value from the second normality indicator value using Equation 3 below.

[0096] [Formula 3]

[0097]

[0098] Here, p2' is the second correction index value, p2 is the second normality index value, pr is a preset reference value, and TH is a preset threshold value.

[0099] As in the previous example, for example, it is assumed that the threshold value (TH) is set to 0.05 and the reference value (pr) is set to 1. If the second normality index value (p2) is 0.05 or greater, the second correction index value (p2') can be calculated as 1. Conversely, if the second normality index value (p2) is less than 0.05, the second correction index value (p2') can be calculated as the value obtained by dividing the second normality index value (p2) by the threshold value (TH). That is, if the second normality index value (p2) is less than the threshold value (TH), the second correction index value (p2') is calculated as a value less than 1.

[0100] In step S50, the index value calculation unit (120) can be configured to calculate the first index value based on the first modified index value calculated in step S34 or step S35 and the second modified index value calculated in step S44 or step S45.

[0101] The index value calculation unit (120) can calculate the first index value by synthesizing the first modified index value and the second modified index value. More specifically, the index value calculation unit (120) can calculate the representative value of the first modified index value and the second modified index value as the first index value.

[0102] In one embodiment, the indicator value calculation unit (120) may be configured to calculate the first indicator value as the average of the first modified indicator value and the second modified indicator value. For example, assuming that the first modified indicator value is 1 and the second modified indicator value is calculated as 0.98, the indicator value calculation unit (120) may calculate the formula "(1+0.98)÷2" to calculate the first indicator value as 0.99.

[0103] Specifically, the indicator value calculation unit (120) can derive a statistical test result for the normality of a normal distribution by calculating a first normality indicator value. In addition, the indicator value calculation unit (120) can quantify the statistical test result for the normality of a normal distribution by calculating a first modified indicator value based on the first normality indicator value and a threshold value.

[0104] Likewise, the indicator value calculation unit (120) can derive statistical test results for the normality of an abnormal distribution by calculating a second normality index value. In addition, the indicator value calculation unit (120) can quantify the statistical test results for the normality of an abnormal distribution by calculating a second modified index value based on the second normality index value and a threshold value.

[0105] Lastly, the indicator value calculation unit (120) can quantify the performance of the anomaly detection model in terms of normality by calculating the first indicator value based on the first modified indicator value and the second modified indicator value.

[0106]

[0107] Next, with reference to FIG. 6, an embodiment in which the index value calculation unit (120) calculates the second index value will be specifically described.

[0108] After the normal distribution and the abnormal distribution are calculated through steps S30 and S40, the indicator value calculation unit (120) can calculate the first area, the second area, and the third area, respectively, in steps S60 to S62. Here, steps S60 to S62 are independent steps.

[0109] In step S60, the indicator value calculation unit (120) can calculate the first area of ​​the normal distribution.

[0110] Specifically, the index value calculation unit (120) can calculate the first area by integrating the internal area of ​​the normal distribution. For example, in the embodiment of FIG. 2, the index value calculation unit (120) can calculate the first area by integrating the normal distribution (nb) from ni to nf.

[0111] In step S61, the indicator value calculation unit (120) can calculate the second area of ​​the abnormal distribution.

[0112] Specifically, the index value calculation unit (120) can calculate the second area by integrating the internal area of ​​the non-stationary distribution. For example, in the embodiment of FIG. 2, the index value calculation unit (120) can calculate the second area by integrating the non-stationary distribution (ab) from ai to af.

[0113] In step S62, the indicator value calculation unit (120) can calculate the third area of ​​the overlapping region of the normal distribution and the abnormal distribution.

[0114] Specifically, the index value calculation unit (120) can first determine an overlapping area where a normal distribution and an abnormal distribution overlap. Then, the index value calculation unit (120) can calculate a third area by integrating the internal area of ​​the overlapping area.

[0115] For example, in the embodiment of FIG. 2, the index value calculation unit (120) can determine an area corresponding to an abnormal distribution (ab) from ai to k and a normal distribution (nb) from k to af as an overlapping area (ov). The index value calculation unit (120) can calculate a third area of ​​the overlapping area (ov) by integrating the abnormal distribution (ab) from ai to k and integrating the normal distribution (nb) from k to af.

[0116] In step S63, the index value calculation unit (120) can be configured to calculate a second index value based on the first area calculated in step S60, the second area calculated in step S61, and the third area calculated in step S62.

[0117] Specifically, the third area is an area that overlaps the first and second areas. That is, if the third area is greater than 0, the third area is overlapped in the sum of the first and second areas. Therefore, the index value calculation unit (120) may be configured to calculate the second index value by subtracting the third area from the sum of the first and second areas.

[0118]

[0119] Finally, referring to FIG. 7, an embodiment in which the performance evaluation unit (130) evaluates the performance of the anomaly detection model is specifically described.

[0120] In step S70, the performance evaluation unit (130) can be configured to calculate the performance value of the anomaly detection model based on the first indicator value calculated in step S50 and the second indicator value calculated in step S63.

[0121] Specifically, the first indicator value and the second indicator value can be calculated in a range of 0 or more and 1 or less. The performance evaluation unit (130) can calculate the performance value of the anomaly detection model within a range of 0 or more and 1 or less based on the first indicator value and the second indicator value.

[0122] For example, the performance evaluation unit (130) can calculate the performance value of the anomaly detection model by multiplying the first indicator value and the second indicator value.

[0123] As another example, the performance evaluation unit (130) can calculate the average value of the first indicator value and the second indicator value as the performance value of the anomaly detection model.

[0124] As previously explained, the second indicator value corresponds to the area of ​​the normal distribution and the abnormal distribution. Here, since the area of ​​the distribution represents probability, the second indicator value can be said to be the overall probability based on the normal distribution and the abnormal distribution. In addition, the first indicator value is a value that quantifies the normality of the normal distribution and the abnormal distribution. In other words, the performance evaluation unit (130) can quantify the performance value of the anomaly detection model by correcting the overall probability based on the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution.

[0125] In step S71, the performance evaluation unit (130) can evaluate the performance of the anomaly detection model based on the calculated performance value.

[0126] In one embodiment, the performance evaluation unit (130) may be configured to absolutely evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.

[0127] Specifically, the performance benchmark is a value that serves as a standard for absolutely evaluating the performance of an anomaly detection model. The performance evaluation unit (130) can evaluate the performance of an anomaly detection model by comparing the performance value and the performance benchmark value to grade the performance.

[0128] For example, the performance criterion may be set to a value greater than or equal to 0.5 and less than or equal to 1. More preferably, the performance criterion may be set to a value greater than or equal to 0.8 and less than or equal to 1. If the performance criterion is assumed to be set to 0.8, the performance evaluation unit (130) may evaluate the performance of an anomaly detection model whose calculated performance value is greater than or equal to 0.8 as grade 1, and may evaluate the performance of an anomaly detection model whose calculated performance value is less than or equal to 0.8 as grade 2.

[0129] As another example, the performance benchmark value may be preset to multiple values. That is, based on the multiple performance benchmark values, a range of 0 or more and 1 or less may be divided into multiple intervals. The performance evaluation unit (130) may evaluate the performance of the anomaly detection model as a grade based on the interval to which the calculated performance value belongs.

[0130] In another embodiment, multiple anomaly detection models may be provided, and the performance evaluation unit (130) may be configured to relatively evaluate the performance of the multiple anomaly detection models.

[0131] Specifically, the performance evaluation unit (130) may be configured to calculate a performance value for each of a plurality of anomaly detection models. Furthermore, the performance evaluation unit (130) may be configured to evaluate the relative performance of the plurality of anomaly detection models based on the calculated performance values.

[0132] For example, the performance evaluation unit (130) can evaluate an anomaly detection model with superior relative performance in order of high calculated performance values.

[0133] That is, the performance evaluation device (100) of an anomaly detection model according to one embodiment of the present invention can evaluate the performance of an anomaly detection model in various ways by calculating a performance value representing the performance of the anomaly detection model.

[0134]

[0135] A server according to another embodiment of the present invention may include a performance evaluation device (100) of an anomaly detection model according to one embodiment of the present invention.

[0136] For example, the server can be connected to the outside world via wired and / or wireless communication. Furthermore, the server can receive normal and abnormal learning data from the outside world and store the received normal and abnormal learning data. Furthermore, the server can update the normal and abnormal learning data periodically or aperiodically.

[0137] The server can evaluate the performance of an anomaly detection model based on normal and abnormal training data and store the performance evaluation results. Furthermore, upon external request or when a specific event occurs, the server can provide the performance evaluation results of the anomaly detection model.

[0138] That is, since the performance evaluation results stored in the server can be referenced by various external devices that can access the server, an anomaly detection model with excellent performance evaluation results can be preferentially adopted in various devices or industrial environments where anomaly detection is required.

[0139]

[0140] FIG. 8 is a diagram schematically illustrating a performance evaluation method of an anomaly detection model according to another embodiment of the present invention.

[0141] Referring to Fig. 8, a performance evaluation method of an anomaly detection model may include a distribution calculation step (S100), an indicator value calculation step (S200), and a performance evaluation step (S300).

[0142] Preferably, each step of the performance evaluation method of the anomaly detection model can be performed by the performance evaluation device (100) of the anomaly detection model. In the following, for convenience of explanation, any content that overlaps with the previously described content will be omitted or briefly described.

[0143] The distribution generation step (S100) is a step of generating a normal distribution for preset normal learning data and generating an abnormal distribution for preset abnormal learning data using an anomaly detection model, and can be performed by the distribution generation unit (110).

[0144] For example, the distribution generating unit (110) can simulate the abnormality scores for each normal learning data from which the anomaly detection model is generated as a normal distribution. In addition, the distribution generating unit (110) can simulate the abnormality scores for each abnormal learning data from which the anomaly detection model is generated as a normal distribution.

[0145] The index value calculation step (S200) is a step of calculating a first index value for a normal distribution and an abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and calculating a second index value for the area of ​​the normal distribution and the abnormal distribution, and can be performed by the index value calculation unit (120).

[0146] Specifically, the indicator value calculation unit (120) can calculate a first normality indicator value for a normal distribution and a second normality indicator value for an abnormal distribution. In addition, the indicator value calculation unit (120) can calculate the first indicator value based on the first normality indicator value and the second normality indicator value.

[0147] In addition, the index value calculation unit (120) can calculate the first area of ​​the normal distribution, the second area of ​​the abnormal distribution, and the third area of ​​the overlapping region of the normal distribution and the abnormal distribution. In addition, the index value calculation unit (120) can calculate the value obtained by subtracting the third area from the sum of the first and second areas as the second index value.

[0148] The performance evaluation step (S300) is a step for evaluating the performance of the anomaly detection model based on the first indicator value and the second indicator value, and can be performed by the performance evaluation unit (130).

[0149] For example, the performance evaluation unit (130) can absolutely evaluate the performance of the anomaly detection model by comparing the calculated performance value with a preset performance reference value.

[0150] As another example, the performance evaluation unit (130) can relatively evaluate the performance of multiple anomaly detection models.

[0151]

[0152] The embodiments of the present invention described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which the program is recorded, and such implementation can be easily implemented by an expert in the technical field to which the present invention belongs based on the description of the embodiments described above.

[0153] Another embodiment of the present invention can provide a computer-readable recording medium having recorded thereon a program for performing the various embodiments described above on a computer.

[0154] The program may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0155] Software may include computer programs, codes, instructions, or any combination thereof, which may configure a processing device to perform a desired operation or may independently or collectively command a processing device.

[0156] Software may be implemented as a computer program comprising instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, Digital Versatile Discs (DVDs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The storage media may be readable by a computer, stored in a memory, and executed by a processor.

[0157] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored on the storage media and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0158] Additionally, the program may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0159] A computer program product may include a software program or a computer-readable storage medium storing the software program. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by an electronic device manufacturer or through an electronic marketplace. For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of the electronic device manufacturer's server, an electronic marketplace server, or an intermediary server that temporarily stores the software program.

[0160] Although the present invention has been described above with reference to limited embodiments and drawings, the present invention is not limited thereto, and it is obvious that various modifications and variations are possible within the scope of the technical idea of ​​the present invention and the equivalent scope of the patent claims to be described below by a person having ordinary skill in the art to which the present invention pertains.

[0161] In addition, the present invention described above is not limited to the above-described embodiments and the attached drawings, and all or part of each embodiment may be selectively combined and configured so that various modifications can be made, as those skilled in the art can make various substitutions, modifications, and changes within the scope of the technical idea of ​​the present invention.

[0162]

[0163] (Explanation of symbols)

[0164] 100: Performance Evaluation Device for Anomaly Detection Models

[0165] 110: Distribution Output Section

[0166] 120: Index value calculation section

[0167] 130: Performance Evaluation Department

[0168] 140: Storage

Claims

1. A distribution generating unit configured to generate a normal distribution for preset normal learning data using an anomaly detection model and to generate an abnormal distribution for preset abnormal learning data; An indicator value calculation unit configured to calculate a first indicator value for the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and to calculate a second indicator value for the area of ​​the normal distribution and the abnormal distribution; and A performance evaluation device for an anomaly detection model, comprising a performance evaluation unit configured to evaluate the performance of the anomaly detection model based on the first indicator value and the second indicator value.

2. In paragraph 1, The above indicator value calculation section, Calculate the first normality index value for the above normal distribution, Calculate the second normality index value for the above non-normal distribution, A performance evaluation device for an anomaly detection model configured to calculate the first indicator value based on the first normality indicator value and the second normality indicator value.

3. In paragraph 2, The above indicator value calculation section, A first correction index value is calculated based on the result of comparing the preset threshold value with the first normality index value, A second modified index value is calculated based on the result of comparing the above threshold value and the second normality index value, A performance evaluation device for an anomaly detection model configured to calculate the first indicator value based on the first modified indicator value and the second modified indicator value.

4. In paragraph 3, The above indicator value calculation section, If the first normality index value is greater than or equal to the threshold value, the first correction index value is set to a preset reference value, A performance evaluation device for an anomaly detection model configured to calculate the first correction index value as a composite value of the first normality index value and the threshold value when the first normality index value is less than the threshold value.

5. In paragraph 3, The above indicator value calculation section, If the second normality index value is greater than or equal to the threshold value, the second correction index value is set to a preset reference value, A performance evaluation device for an anomaly detection model configured to calculate the second correction index value as a composite value of the second normality index value and the threshold value when the second normality index value is less than the threshold value.

6. In paragraph 3, The above indicator value calculation section, A performance evaluation device for an anomaly detection model configured to calculate the first indicator value as an average value of the first modified indicator value and the second modified indicator value.

7. In paragraph 1, The above indicator value calculation section, A performance evaluation device for an anomaly detection model configured to calculate a first area of ​​the normal distribution, a second area of ​​the abnormal distribution, and a third area of ​​an overlapping region of the normal distribution and the abnormal distribution, and to calculate the second indicator value based on the first area, the second area, and the third area.

8. In paragraph 7, The above indicator value calculation section, A performance evaluation device for an anomaly detection model configured to calculate a value obtained by subtracting the third area from the sum of the first area and the second area as the second indicator value.

9. In paragraph 1, The above performance evaluation section, A performance evaluation device for an anomaly detection model configured to calculate a performance value of the anomaly detection model based on the first indicator value and the second indicator value.

10. In paragraph 9, The above performance evaluation section, A performance evaluation device for an anomaly detection model configured to evaluate the performance of the anomaly detection model by comparing the above-determined performance value with a preset performance reference value.

11. In paragraph 9, The above anomaly detection model is provided in multiple forms, The above performance evaluation section, A performance evaluation device for an anomaly detection model configured to calculate a performance value for each of a plurality of anomaly detection models and to evaluate the relative performance of the plurality of anomaly detection models based on the calculated plurality of performance values.

12. A server including a performance evaluation device of an anomaly detection model according to any one of clauses 1 to 11.

13. A distribution generation step for generating a normal distribution for preset normal learning data and an abnormal distribution for preset abnormal learning data using an anomaly detection model; An indicator value calculation step for calculating a first indicator value for the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and calculating a second indicator value for the area of ​​the normal distribution and the abnormal distribution; and A performance evaluation method of an anomaly detection model, comprising a performance evaluation step of evaluating the performance of the anomaly detection model based on the first indicator value and the second indicator value.

14. A distribution generation step for generating a normal distribution for preset normal learning data and an abnormal distribution for preset abnormal learning data using an anomaly detection model; An indicator value calculation step for calculating a first indicator value for the normal distribution and the abnormal distribution based on the normality of the normal distribution and the abnormal distribution, and calculating a second indicator value for the area of ​​the normal distribution and the abnormal distribution; and A computer-readable recording medium storing a computer program for executing a performance evaluation method of an anomaly detection model, including a performance evaluation step of evaluating the performance of the anomaly detection model based on the first indicator value and the second indicator value.

Images