Abnormal detection model evaluation method and device, equipment, storage medium and product
By evaluating multiple evaluation metrics of the anomaly detection model, the problem of low reliability caused by single evaluation basis in the prior art is solved, and a more comprehensive and reliable model evaluation is achieved.
Patent Information
- Application Number
- CN202411137828.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-26
AI Technical Summary
When evaluating anomaly detection models, the evaluation basis is relatively single, and it is difficult to comprehensively evaluate the shortcomings of the model, resulting in low reliability of the evaluation results.
By obtaining the key performance indicators within the preset time range, input them into the abnormality detection model to be evaluated, obtain the upper limit, lower limit and abnormal output of the dynamic threshold value, and calculate the abnormality ratio measurement, periodic matching measurement, and repeatable false alarm measurement. , smoothness measurement and underlying distribution measurement of abnormal scores to comprehensively evaluate the performance of the model.
Multi-faceted quantitative analysis of the anomaly detection model is realized, the evaluation effect is optimized, and the reliability of the evaluation results is improved.
Smart Images

Figure CN119025425A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of model evaluation technology, and in particular, relates to an anomaly detection model evaluation method, device, equipment, storage medium and product. Background Art
[0002] With the continuous development and expansion of mobile Internet and digital systems, maintaining the stability and efficient operation of the system and improving user experience have become important goals for business providers. In recent years, with the development of intelligent operation and maintenance, more and more companies have adopted automation, big data, and artificial intelligence to replace traditional manual methods to monitor and maintain the system. In intelligent operation and maintenance technology, the monitoring and anomaly detection of key performance indicators of the system play an important role.
[0003] Anomaly detection of key performance indicators can be achieved through fixed rules, statistical methods, machine learning, and deep learning. In order to select a suitable anomaly detection model for a specific key performance indicator and compare the detection effects of different anomaly detection models, the output results of the anomaly detection model can be evaluated. However, the evaluation basis currently considered in the evaluation is relatively single, and it is difficult to fully evaluate the shortcomings of the anomaly detection model, resulting in low reliability of the evaluation results. Summary of the invention
[0004] The embodiments of the present application provide an anomaly detection model evaluation method, apparatus, device, storage medium and computer program product, which can improve the reliability of the evaluation results of the anomaly detection model.
[0005] In a first aspect, an embodiment of the present application provides an anomaly detection model evaluation method, the method comprising:
[0006] Obtain key performance indicators within a preset time range at preset time intervals;
[0007] Inputting the key performance indicator into the anomaly detection model to be evaluated, and obtaining an output result of the anomaly detection model output, wherein the output result includes at least one of a dynamic threshold upper limit value, a dynamic threshold lower limit value, and an abnormal output;
[0008] Determine an evaluation metric based on the output result, the evaluation metric including at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric;
[0009] Based on the evaluation metric, the evaluation result of the anomaly detection model is determined.
[0010] In a second aspect, an embodiment of the present application provides an anomaly detection model evaluation device, the device comprising:
[0011] An acquisition module is used to acquire key performance indicators within a preset time range at preset time intervals;
[0012] A detection module, used for inputting the key performance indicator into the anomaly detection model to be evaluated, and obtaining an output result of the anomaly detection model output, wherein the output result includes at least one of a dynamic threshold upper limit value, a dynamic threshold lower limit value and an abnormal output;
[0013] A first determination module is used to determine an evaluation metric based on the output result, where the evaluation metric includes at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric;
[0014] The second determination module is used to determine the evaluation result of the anomaly detection model according to the evaluation metric.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising:
[0016] A processor and a memory storing programs or instructions;
[0017] The above method is implemented when the processor executes the program or instruction.
[0018] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, on which a program or instruction is stored, and the program or instruction implements the above method when executed by a processor.
[0019] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the above method.
[0020] The embodiment of the present application provides an anomaly detection model evaluation method that can obtain key performance indicators within a preset time range according to a preset time interval; input the key performance indicators into the anomaly detection model to be evaluated, and obtain the output result of the anomaly detection model output, the output result includes at least one of the dynamic threshold upper limit value, the dynamic threshold lower limit value and the abnormal output; according to the output result, determine the evaluation metric, the evaluation metric includes at least two of the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric and the anomaly score underlying distribution metric; according to the evaluation metric, determine the evaluation result of the anomaly detection model. In this way, by separately evaluating multiple matching metrics including the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric and the anomaly score underlying distribution metric between the anomaly detection model and the key performance indicator data, a comprehensive quantitative analysis of the characteristics and defects of the anomaly detection model is achieved, and the evaluation results affected by multiple factors are obtained, which greatly optimizes the effect of the anomaly detection model evaluation and improves the reliability of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0022] Figure 1 It is a flowchart of an anomaly detection model evaluation method provided by an embodiment of the present application;
[0023] Figure 2 This is a flowchart of a specific scenario implementation of the anomaly detection model evaluation method provided in the embodiment of the present application;
[0024] Figure 3 is a structural diagram of an anomaly detection model evaluation device provided by another embodiment of the present application;
[0025] Figure 4 It is a structural diagram of an electronic device provided by yet another embodiment of the present application. DETAILED DESCRIPTION
[0026] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0028] In addition, it should be noted that the acquisition, storage, use, and processing of data in the embodiments of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.
[0029] In order to solve the problems of the prior art, the embodiments of the present application provide an anomaly detection model evaluation method, device, equipment, storage medium and product. The anomaly detection model evaluation method is first introduced below.
[0030] like Figure 1 As shown, the anomaly detection model evaluation method may include:
[0031] Step 101: obtaining key performance indicators within a preset time range at preset time intervals.
[0032] In step 101, the key performance indicators may include business-related key performance indicators, such as business call volume, number of successful requests, number of failed requests, average request duration, etc. It may also include machine-related key performance indicators, such as central processing unit (CPU) occupancy, memory occupancy, disk usage, system input and output (I / O), etc.
[0033] The key performance indicators within the preset time range can be obtained at the preset time interval. The time interval and the preset time range can be designed according to actual needs and are not specifically limited here. For example, the time interval can be 1 minute, and the preset time range can be the time period from the system starting to obtain the key performance indicators to the current moment.
[0034] It is understandable that the key performance indicator may include multiple data according to time division, and each data may carry a timestamp. The T data of the key performance indicator within the preset time range can be expressed as x={x0,x1,...,x T-1}, the corresponding timestamp list is ts = {ts0, ts1, ..., ts T-1}, where the timestamp can be expressed as a ten-digit timestamp in seconds.
[0035] Step 102, input the key performance indicator into the anomaly detection model to be evaluated, and obtain the output result of the anomaly detection model output, where the output result includes at least one of the dynamic threshold upper limit value, the dynamic threshold lower limit value and the abnormal output.
[0036] In step 102, the key performance indicator can be input into the anomaly detection model to be evaluated, and the anomaly detection model can be output in the form of a dynamic threshold to obtain at least one of the dynamic threshold upper limit value, the dynamic threshold lower limit value and the abnormal output output by the anomaly detection model.
[0037] The dynamic threshold upper limit value and the dynamic threshold lower limit value can be the thresholds used in the anomaly detection model to determine whether it is an abnormal value, and the threshold changes dynamically according to the input key performance indicators, rather than a fixed threshold. For example, if the value of a certain key performance indicator is within the range of its corresponding dynamic threshold upper limit value and dynamic threshold lower limit value, the value of the key performance indicator can be considered to be a normal value. If the value of a certain key performance indicator exceeds its corresponding dynamic threshold upper limit value, or exceeds its corresponding dynamic threshold lower limit value, the value of the key performance indicator can be considered to be an abnormal value.
[0038] The abnormal output can be used to reflect the abnormal detection results of the dynamic key performance indicators. It is understandable that for T data in the key performance indicators within the preset time range, the corresponding abnormal output may include T sub-outputs, and the sub-output may be 0 or 1, where 0 represents normal and 1 represents abnormal.
[0039] For example, for T data in the key performance indicator, in the output result of the corresponding anomaly detection model, the upper limit value of the dynamic threshold can be expressed as ub = {ub0, ub1, ..., ub T-1}, the lower limit of the dynamic threshold can be expressed as lb={lb0,lb1,...,lb T-1}, the abnormal output can be expressed as y = {y0,y1,...,y T-1}.
[0040] Step 103, determining an evaluation metric based on the output result, the evaluation metric including at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric.
[0041] In step 103, evaluation metrics including anomaly ratio metrics, periodicity matching metrics, repeatability false alarm metrics, smoothness metrics, and anomaly score underlying distribution metrics can be calculated based on the dynamic threshold upper limit value, the dynamic threshold lower limit value, and the abnormal output. The type and data of the evaluation metric can be selected according to actual needs. In order to facilitate understanding of the scheme of the embodiment of the present application, the following will be described by taking the evaluation metric including anomaly ratio metrics, periodicity matching metrics, repeatability false alarm metrics, smoothness metrics, and anomaly score underlying distribution metrics as an example.
[0042] It is understandable that the anomaly ratio metric may be a percentage system, and the higher the value of the anomaly ratio metric is, the more reasonable and appropriate the anomaly detection model is in this metric, and this factor is not likely to affect the detection effect of the model.
[0043] The periodic matching metric calculates the periodic matching degree between the upper and lower dynamic threshold values output by the anomaly detection model and the key performance indicators. Appropriate upper and lower dynamic threshold values should have similar periodicity to the key performance indicators. If the periodicity does not match, there may be more missed reports, that is, the detection performance of the model is poor.
[0044] The repetitive false alarm metric calculates whether there are periodic repetitive false alarms in the anomaly output of the anomaly detection model. The repetitive false alarm metric estimates the detection performance of the anomaly detection model through the repetitive false alarm data in the anomaly detection model.
[0045] The smoothness metric estimates the detection performance of the anomaly detection model by calculating the smoothness of the dynamic upper threshold value and the dynamic lower threshold value output by the anomaly detection model. The dynamic upper threshold value and the dynamic lower threshold value of the anomaly detection model generally need to be smoothed to a certain extent. In particular, the output of the dynamic upper threshold value and the dynamic lower threshold value that are not smooth may cause the output results of the model to be greatly affected by noise.
[0046] The underlying distribution metric of the anomaly score can convert the upper and lower bounds of the dynamic threshold output by the anomaly detection model into an anomaly score. The difference between the underlying distribution of the anomaly score and the ideal distribution, as well as the changing trend of the underlying distribution, can be used to determine whether the underlying score of the anomaly score is reasonable. The ideal distribution can be pre-set by empirical values.
[0047] Step 104: Determine an evaluation result of the anomaly detection model according to the evaluation metric.
[0048] In step 104, an anomaly detection evaluation data may be generated according to different types of matching metrics, so that the evaluation data may be determined to generate an evaluation report or to calculate an evaluation score, thereby obtaining a more comprehensive and accurate evaluation result.
[0049] In an embodiment of the present application, the anomaly detection model evaluation method can obtain key performance indicators within a preset time range at preset time intervals; input the key performance indicators into the anomaly detection model to be evaluated to obtain the dynamic threshold upper limit value, the dynamic threshold lower limit value and the abnormal output output by the anomaly detection model; determine the evaluation metric based on the dynamic threshold upper limit value, the dynamic threshold lower limit value and the abnormal output, and the evaluation metric includes at least two of the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric and the anomaly score underlying distribution metric; determine the evaluation result of the anomaly detection model based on the evaluation metric.
[0050] In this way, by separately evaluating multiple matching metrics between the anomaly detection model and key performance indicator data, including anomaly ratio metric, periodic matching metric, repeatability false alarm metric, smoothness metric, and anomaly score underlying distribution metric, a comprehensive quantitative analysis of the characteristics and defects of the anomaly detection model is achieved, and evaluation results influenced by multiple factors are obtained, which greatly optimizes the effect of anomaly detection model evaluation and improves the reliability of evaluation results.
[0051] In some embodiments, when the evaluation metric includes an abnormality ratio metric, the output result includes an abnormal output; the above step 103 may include:
[0052] According to the abnormal output, determine the abnormal point ratio and abnormal hour ratio, wherein the abnormal output includes multiple sub-outputs, and each sub-output carries time information, the abnormal point ratio is used to indicate the ratio of the number of the first sub-output to the total number of sub-outputs in the abnormal output, and the abnormal hour ratio is used to indicate the ratio of the number of hours including the first sub-output in each hour of the sub-output to the total hours, and the output result of the first sub-output is abnormal;
[0053] Determine the abnormal ratio metric based on the abnormal point ratio and abnormal hour ratio.
[0054] In this embodiment, as mentioned above, the T data in the key performance indicator can be expressed as x={x0, x1, ..., x T-1}, the corresponding timestamp list is ts = {ts0, ts1, ..., ts T-1}, the abnormal output can be expressed as y = {y0,y1,...,y T-1}.
[0055] The abnormal point ratio can be determined based on the abnormal output. a The calculation formula is shown in formula (1):
[0056]
[0057] Among them, count(·) means to find the total number of elements in the set.
[0058] The abnormal hours ratio can also be determined based on the abnormal output. For example, the timestamp in seconds can be converted to the timestamp in hours. The conversion formula is shown in formula (2):
[0059]
[0060] in, This is the rounding operation.
[0061] Merge the same elements in the list ts′ in sequence to obtain a non-repeated hourly timestamp. The expression of the hourly timestamp is shown in formula (3):
[0062] tsh={tsh0,tsh1,...,tsh K-1} (3)
[0063] The total number of hours K is calculated using the formula shown in formula (4):
[0064]
[0065] Then the abnormal output can be converted into hourly abnormal output. The conversion formula is shown in formula (5):
[0066]
[0067] The abnormal hours ratio Rh can be calculated based on the total hours and abnormal output per hour. a , the calculation formula is shown in formula (6):
[0068]
[0069] The anomaly ratio metric can be calculated based on the anomaly point ratio and the anomaly hour ratio. R The calculation formula is shown in formula (7):
[0070]
[0071] in, ω(·) is the ratio score function, which is a monotonically decreasing function, indicating that the higher the anomaly ratio, the worse the anomaly detection model may perform. and They are the thresholds for the abnormal point ratio and abnormal hour ratio, respectively. When the corresponding abnormal ratio is higher than the threshold, the abnormal ratio measurement will decrease. The thresholds for the abnormal point ratio and abnormal hour ratio can be set according to actual needs. For example, the default value of the parameter is
[0072] In this way, the anomaly ratio metric can be calculated by the proportion of abnormal points and the proportion of abnormal hours, so that the anomaly ratio metric can be used to analyze the characteristics and defects of the anomaly detection model later.
[0073] In some embodiments, the evaluation metric includes a periodic matching metric, and the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value; the above step 103 may include:
[0074] Extract m data segments from the key performance indicators, the upper bound value of the dynamic threshold, and the lower bound value of the dynamic threshold, respectively, each data segment includes a preset number of data of multiple consecutive cycles, and m is an integer greater than 1;
[0075] For each data segment, cyclic autocorrelation processing is performed, and the processed data is subjected to Fourier transform processing to obtain the periodicity score corresponding to the data segment;
[0076] Determining a first periodicity score of the key performance indicator according to the m periodicity scores corresponding to the m data segments of the key performance indicator;
[0077] Determine a second periodicity fraction of the upper limit value of the dynamic threshold according to the m periodicity fractions corresponding to the m data segments of the upper limit value of the dynamic threshold;
[0078] Determine a third periodicity fraction of the dynamic threshold lower limit value according to the m periodicity fractions corresponding to the m data segments of the dynamic threshold lower limit value;
[0079] A periodicity matching metric is determined based on the first periodicity score, the second periodicity score, and the third periodicity score.
[0080] In this embodiment, since the calculation process of the first periodicity score, the second periodicity score and the third periodicity score is the same, the following will take the calculation of the first periodicity score of m data segments of the key performance indicator as an example for detailed description.
[0081] It is understandable that common key performance indicator data may have a daily periodicity with one day as the unit, so the period in this embodiment may refer to a daily period. The value of m and the number of consecutive periods included in each data segment can be set according to actual needs and are not specifically limited here. For example, m can be 3 and the number of consecutive periods can also be 3 as an example for explanation.
[0082] For example, three random samples can be drawn from the key performance indicator x, each time a continuous data segment with a time span of 3 days is drawn, and they are set as x (1) , x (2) , x (3) As mentioned above, the key performance indicators are obtained at 1-minute time intervals, so the number of sample points in each data segment is N=4320.
[0083] For each data segment x (m) , find the corresponding periodic fraction The method for calculating the periodicity score can be: first perform three cyclic autocorrelation processes to enhance its periodicity, then perform Fourier transform, calculate the proportion of the energy value of the periodic component to the total energy value, and obtain the periodicity score.
[0084] The expression of cyclic autocorrelation processing is shown in formula (8):
[0085]
[0086]
[0087] Among them, when t<i, ti is a negative number, Undefined, assign value here Can make It is defined when t<i, and a cyclic autocorrelation result is obtained.
[0088] The frequency domain sequence can be obtained by Fourier transform processing, and the expression is shown in formula (9):
[0089]
[0090] Each data segment x (m) The corresponding periodic fraction The calculation formula can be shown as formula (10):
[0091]
[0092] Here, i=3 is because the time span of each data segment is 3 days.
[0093] The three periodic scores corresponding to the three data segments of the key performance indicators can be and Calculate the first periodicity score P of the key performance indicator x , the calculation formula can be shown as formula (11):
[0094]
[0095] The second periodic fraction P of the upper limit of the dynamic threshold can be calculated by the same method. ub and the third periodic fraction P of the lower limit of the dynamic threshold lb
[0096] Based on the first periodicity score, the second periodicity score and the third periodicity score, calculate the periodicity matching metric M P , the calculation formula is shown in formula (12):
[0097] M P =50·Sgn(|P ub -P x |≤th P )+50·Sgn(|P lb -P x |≤thP )(12)
[0098] Among them, Sgn(·) represents the sign function, th P is the periodic fractional difference threshold, the default value can be th P =0.3. When the first periodicity score differs too much from the second periodicity score or the third periodicity score, the periodicity match metric decreases, indicating that the anomaly detection model may not be able to learn the periodic characteristics of the key performance indicators well, so its detection results may be affected and deteriorate. The periodicity match metric can be a percentage system, and its possible value is 0, 50 or 100 according to the number of period mismatch thresholds.
[0099] In this way, the periodic matching degree between the upper and lower bounds of the dynamic threshold of the anomaly detection model and the key performance indicators can be calculated through the periodic matching metric to analyze the characteristics and defects of the anomaly detection model.
[0100] In some embodiments, when the evaluation metric includes a repetitive false alarm metric, the output result includes an abnormal output; the above step 103 may include:
[0101] Initialize the abnormal hours list and cumulative hours list within the preset period;
[0102] According to the abnormal output, determine each abnormal hour in the abnormal hours list and each cumulative hour in the cumulative hours list;
[0103] Determine multiple repeated abnormality ratios within a preset period based on each abnormal hour and each cumulative hour;
[0104] The maximum value among multiple repeated abnormal ratios is determined as the repeatability false alarm rate;
[0105] Based on the repeatability false alarm rate, determine the repeatability false alarm measure.
[0106] In this embodiment, according to the calculation method disclosed above, the hourly timestamp tsh={tsh0, tsh1, ..., tsh K-1} and hourly abnormal output yh = {yh0, yh1, ..., yh K-1}.
[0107] Initialize the abnormal hours list of 24 hours Ha = {Ha0, Ha1, ...Ha 23}={0,0,...,0}. And initialize the 24-hour cumulative hour list Hn={Hn0,Hn1,...Hn 23}={0,0,...,0}.
[0108] By traversing the hourly timestamp tsh and the hourly abnormal output yh, the abnormal hours list Ha and the cumulative hours list Hn can be calculated.
[0109] For example, for k=0, 1, ..., K-1, calculate h=tsh k If yh k =1, then assign value Ha k =Ha k +1, Hn k =Hn k +1.
[0110] Multiple repeated abnormality ratios within a preset period can be determined based on each abnormal hour and each cumulative hour. For example, the calculation formula for the repeated abnormality ratio of 24 hours is shown in formula (13):
[0111]
[0112] The maximum value of the repeated abnormal ratio in 24 hours can be determined as the repeatability false alarm rate R h =max(Reph), and the repeatability false alarm metric can be calculated based on the repeatability false alarm rate. The calculation formula of the repeatability false alarm metric can be shown as formula (14):
[0113] M H =100·Sgn(R h ≤th h ) (14)
[0114] Among them, Sgn(·) represents the sign function, th h is the repetitive false alarm threshold, the default value can be th P =0.25. When the anomaly detection model frequently outputs anomalies in the same time period every day, it is considered that the anomaly detection model may not be able to capture the fixed fluctuations of the key performance indicators, so its detection performance may be affected and deteriorate. The repeatability false alarm metric is a percentage system, and its possible values are 0 or 100.
[0115] As mentioned above, due to the daily periodicity of key performance indicators, key performance indicators may have fixed fluctuations in the same time period of each day, such as jumps, spikes, and 0 values. Fixed fluctuations are normal changes in key performance indicators. Due to the non-smoothness of fixed fluctuations, fixed fluctuations are difficult to detect by anomaly detection models. Many anomaly detection models will generate abnormal inputs at fixed fluctuations, resulting in inaccurate output results.
[0116] In this embodiment, the repetitive false alarm metric can estimate the detection performance of the anomaly detection model for fixed fluctuations through the repetitive false alarm data in the anomaly detection model, thereby analyzing whether the output results of the anomaly detection model are accurate, providing a basis for improving the reliability of the evaluation results.
[0117] In some embodiments, when the evaluation metric includes a smoothness metric, the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value; the above step 103 may include:
[0118] According to the upper limit value of the dynamic threshold, determine the sliding average sequence of the upper limit value, and according to the lower limit value of the dynamic threshold, determine the sliding average sequence of the lower limit value;
[0119] Determine the first variance of the difference between the upper limit value sliding average sequence and the upper limit value of the dynamic threshold;
[0120] Determine the upper limit value smoothness value based on the first variance and the upper limit value sliding average sequence;
[0121] Determine the second variance of the difference between the lower limit value sliding average sequence and the lower limit value of the dynamic threshold;
[0122] Determine the smoothness value of the lower limit value according to the second variance and the lower limit value sliding average sequence;
[0123] A smoothness measure is determined based on the upper bound smoothness value and the lower bound smoothness value.
[0124] In this embodiment, since the calculation process of the upper boundary value smoothness value and the lower boundary value sliding average sequence is the same, the calculation of the upper boundary value smoothness value will be described in detail below as an example.
[0125] The upper limit value sliding average sequence mva can be determined according to the upper limit value of the dynamic threshold. The expression of the upper limit value sliding average sequence mva is shown in formula (15):
[0126]
[0127] Among them, since the length of the upper limit sliding average sequence is W-1 shorter than the length of the dynamic threshold upper limit, W-1 are added to the front of mva. So that its length is T-1.
[0128] The first variance of the difference between the upper limit value sliding average sequence and the upper limit value of the dynamic threshold can be determined. For example, the difference between the key performance indicator and the upper limit value sliding average sequence can be calculated first, and the calculation formula can be shown as formula (16):
[0129] diff=mva-ub (16)
[0130] Then the first variance of the difference can be calculated, and the calculation formula can be shown as formula (17):
[0131]
[0132] The upper limit smoothness value S can be calculated based on the first variance and the upper limit sliding average sequence: ub , the calculation formula can be shown as formula (18):
[0133]
[0134] The same calculation method can be used to calculate the lower limit smoothness value S lb .
[0135] The smoothness measure can be calculated based on the upper boundary smoothness value and the lower boundary smoothness value. The calculation formula can be shown as formula (19):
[0136] M S =100·min(S ub ,S lb )(19)
[0137] It is understandable that when the upper or lower bound of the dynamic threshold of the anomaly detection model is less smooth, the smoothness metric will decrease. The smoothness metric can be a percentage system, and the higher the metric value, the more reasonable the anomaly detection is in this metric, and this factor is not likely to affect the detection effect of the anomaly detection model.
[0138] In this embodiment, the non-smooth output of the dynamic threshold upper limit value and the dynamic threshold lower limit value may cause the output result of the anomaly detection model to be greatly affected by noise. The smoothness metric estimates the detection performance of the anomaly detection model by calculating the smoothness of the dynamic threshold upper limit value and the dynamic threshold lower limit value of the output result of the anomaly detection model, providing a basis for improving the reliability of the evaluation results.
[0139] In some embodiments, when the evaluation metric includes an underlying distribution metric of anomaly scores, the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value; the above step 103 may include:
[0140] Determine an upper threshold abnormality score after the dynamic threshold upper threshold value is transformed, and a lower threshold abnormality score after the dynamic threshold lower threshold value is transformed;
[0141] Obtaining a preset number of thresholds within the value range of the upper boundary value anomaly score to obtain an upper boundary value threshold list, and obtaining a preset number of thresholds within the value range of the lower boundary value anomaly score to obtain a lower boundary value threshold list;
[0142] Traverse the upper limit value threshold list and the lower limit value threshold list to determine their corresponding upper limit value abnormal probability list and lower limit value abnormal probability list;
[0143] Determine the underlying distribution score of the abnormal score of the upper boundary value according to the abnormal probability list of the upper boundary value, and determine the underlying distribution score of the abnormal score of the lower boundary value according to the abnormal probability list of the lower boundary value;
[0144] The anomaly score underlying distribution metric is determined according to the anomaly score underlying distribution score of the upper boundary value and the anomaly score underlying distribution score of the lower boundary value.
[0145] In this embodiment, since the calculation process of the abnormal score underlying distribution score of the upper boundary value and the abnormal score underlying distribution score of the lower boundary value are the same, the following will take the calculation of the abnormal score underlying distribution score of the upper boundary value as an example for detailed description.
[0146] The upper limit abnormal score after the conversion of the dynamic threshold upper limit can be calculated first. The expression of the upper limit abnormal score can be shown in formula (19):
[0147] as=x-ub(19)
[0148] A preset number of thresholds can be obtained within the value range of the upper limit abnormality score to obtain an upper limit threshold list. The preset number can be set according to actual needs. For example, in this embodiment, 51 thresholds from small to large can be obtained within the value range of the upper limit abnormality score to form an upper limit threshold list th. The expression of the upper limit threshold list th can be shown as formula (20):
[0149]
[0150] You can traverse the upper threshold value list th and calculate the corresponding upper abnormal probability list ap = {ap0, ap1, ..., ap 50}. For example, for each i in i=0,1,...,50, calculate its abnormal probability ap i As shown in formula (21):
[0151]
[0152] The upper limit abnormal probability list can be used to determine the upper limit abnormal score underlying distribution score. For example, the probability distribution curve of the upper limit abnormal score can be obtained with th as the horizontal axis and ap as the vertical axis, and compared with the ideal probability distribution curve apid = {apid0,apid1,...,apid 50}, and calculate the underlying distribution score of the upper limit of the abnormal score based on the difference value.
[0153] Among them, the expression of the ideal probability distribution curve can be shown as formula (22):
[0154]
[0155] The calculation formula of the upper limit value of the abnormal score underlying distribution score can be shown as formula (23):
[0156]
[0157] The same calculation method can be used to calculate the lower bound of the abnormal score underlying distribution score PS lb The difference is that the expression of the lower bound anomaly score can be shown as formula (24):
[0158] as=lb-x(24)
[0159] The anomaly score underlying distribution metric M can be calculated based on the anomaly score underlying distribution score of the upper boundary value and the anomaly score underlying distribution score of the lower boundary value. U , the calculation formula can be shown as formula (25):
[0160] M U =100·min(PS ub ,PS lb )(25)
[0161] It can be understood that the underlying distribution metric of the anomaly score is a percentage system. The smaller the difference between the anomaly score probability distribution curve of the anomaly detection model and the ideal probability distribution curve, the higher the underlying distribution metric of the anomaly score, which means that the anomaly detection is more reasonable in this metric, and this factor is not likely to affect the detection effect of the anomaly detection model.
[0162] It is also understandable that the underlying distribution metric of the anomaly distribution is affected by the characteristics of the key performance indicator itself. Therefore, when the same key performance indicator is detected using different anomaly detection models, the underlying distribution metrics of the anomaly scores obtained can be compared with each other, and the anomaly detection model with a higher underlying distribution metric of the anomaly score performs better. However, the underlying distribution metrics of the anomaly scores obtained by different key performance indicators are not suitable for comparison with each other, and the results of this metric can be ignored at this time.
[0163] In this way, due to the unstable or unreasonable underlying distribution model, when the model parameters change slightly, the abnormal output will change significantly, thus affecting the robustness of the model itself. Therefore, the underlying distribution metric of the abnormal distribution can be used to reflect the difference between the underlying distribution of the upper and lower bounds of the dynamic threshold and the ideal distribution, as well as the changing trend of the underlying distribution, to determine whether the underlying score of the abnormal score is reasonable.
[0164] In some embodiments, determining the abnormal score underlying distribution score of the upper boundary value according to the upper boundary value abnormal probability list, and determining the abnormal score underlying distribution score of the lower boundary value according to the lower boundary value abnormal probability list may include:
[0165] When the probability in the upper bound abnormal probability list does not satisfy monotonically decreasing, the upper bound abnormal score underlying distribution score is determined to be 0, and,
[0166] When the probability in the lower bound abnormal probability list does not satisfy monotonically decreasing, the lower bound abnormal score underlying distribution score is determined to be 0.
[0167] In this embodiment, the stability of the upper limit value abnormal probability list and the lower limit value abnormal probability list can be determined first. If the probabilities in the upper limit value abnormal probability list or the lower limit value abnormal probability list do not satisfy monotonically decreasing, it can be considered that the underlying distribution that does not satisfy monotonically decreasing is unstable, and at this time, the underlying distribution score of the abnormal score can be directly determined to be 0. In this way, the efficiency of calculating the underlying distribution score of the abnormal score can be improved, and computing power can be saved.
[0168] In some embodiments, determining the abnormal score underlying distribution score of the upper boundary value according to the upper boundary value abnormal probability list, and determining the abnormal score underlying distribution score of the lower boundary value according to the lower boundary value abnormal probability list may also include:
[0169] When the probability in the upper limit abnormal probability list satisfies monotonically decreasing, the upper limit abnormal probability list is compared with the ideal probability distribution curve to obtain the underlying distribution score of the upper limit abnormal score, and,
[0170] When the probability in the lower bound abnormal probability list satisfies monotonically decreasing, the lower bound abnormal probability list is compared with the ideal probability distribution curve to obtain the underlying distribution score of the lower bound abnormal score.
[0171] In this embodiment, if the probabilities in the upper limit abnormal probability list or the lower limit abnormal probability list satisfy monotonically decreasing, the corresponding abnormal score underlying distribution score can be calculated in the manner described above, that is, the corresponding abnormal probability list can be compared with the ideal probability distribution curve to calculate the relative difference, thereby calculating the abnormal score underlying distribution score.
[0172] In some embodiments, the above step 104 may include:
[0173] Based on the evaluation metrics, an evaluation report of the anomaly detection model is generated, and the evaluation results include the evaluation report.
[0174] In this embodiment, after calculating the anomaly ratio metric, periodic matching metric, repeatability false alarm metric, smoothness metric, and anomaly score underlying distribution metric of the anomaly detection model, an evaluation report for the anomaly detection model can be generated based on the results of each metric. Through the evaluation report, possible deficiencies of the model can be intuitively analyzed to assist operation and maintenance personnel in selecting a suitable anomaly detection model.
[0175] The anomaly detection model evaluation report is shown in the following table:
[0176]
[0177]
[0178] In some embodiments, the evaluation metrics include an anomaly ratio metric, a periodicity matching metric, a repetitive false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric.
[0179] The above step 104 may include:
[0180] An evaluation score of the anomaly detection model is determined based on the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric, and the anomaly score underlying distribution metric, and their respective corresponding weights. The evaluation result includes the evaluation score.
[0181] In this embodiment, a weighted calculation can be performed based on the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric, and the anomaly score underlying distribution metric to obtain a weighted anomaly detection model evaluation score M. The calculation formula for the anomaly detection model evaluation score is shown in formula (26):
[0182]
[0183] Among them, the weights corresponding to the anomaly ratio metric, periodic matching metric, repeatability false alarm metric, smoothness metric, and anomaly score underlying distribution metric can be set according to actual needs. For example, the default value can be W R =1, W P =2, W H =2, W S =1, W U =1.
[0184] In order to facilitate understanding of the anomaly detection model evaluation method provided in the above embodiment, the above anomaly detection model evaluation method is described below using a specific scenario embodiment. Figure 2 FIG. 1 is a flowchart showing a specific scenario embodiment of the above-mentioned anomaly detection model evaluation method. Figure 2 As shown, this scenario embodiment may include:
[0185] Step 201, obtaining an anomaly detection model to be evaluated;
[0186] Step 202, obtaining key performance indicators;
[0187] Step 203, the anomaly detection model outputs a result, including a dynamic threshold upper limit value, a dynamic threshold lower limit value and an anomaly output;
[0188] Step 204, calculating an abnormal ratio metric of the abnormal output;
[0189] Step 205, calculating the periodic matching measure between the key performance indicator and the upper and lower bounds;
[0190] Step 206, calculating a repeatability false alarm metric for the abnormal output;
[0191] Step 207, calculating the smoothness measure of the upper bound value and the lower bound value;
[0192] Step 208, calculating the underlying distribution metric of the anomaly score of the output result;
[0193] Step 209, generating an evaluation report of the anomaly detection model;
[0194] Step 210, calculating the evaluation score of the anomaly detection model.
[0195] Figure 3 A schematic diagram of the structure of an anomaly detection model evaluation device provided in another embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0196] Reference Figure 3 , the anomaly detection model evaluation device 300 may include:
[0197] The acquisition module 301 is used to acquire key performance indicators within a preset time range at preset time intervals;
[0198] A detection module 302 is used to input the key performance indicator into the anomaly detection model to be evaluated, and obtain an output result of the anomaly detection model output, wherein the output result includes at least one of a dynamic threshold upper limit value, a dynamic threshold lower limit value, and an abnormal output;
[0199] A first determination module 303 is used to determine an evaluation metric according to the output result, where the evaluation metric includes at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric;
[0200] The second determination module 304 is used to determine the evaluation result of the anomaly detection model according to the evaluation metric.
[0201] In some embodiments, when the evaluation metric includes an abnormal ratio metric, the output result includes an abnormal output, and the first determination module 303 may also be used to:
[0202] According to the abnormal output, determine the abnormal point ratio and abnormal hour ratio, wherein the abnormal output includes multiple sub-outputs, and each sub-output carries time information, the abnormal point ratio is used to indicate the ratio of the number of the first sub-output to the total number of sub-outputs in the abnormal output, and the abnormal hour ratio is used to indicate the ratio of the number of hours including the first sub-output in each hour of the sub-output to the total hours, and the output result of the first sub-output is abnormal;
[0203] Determine the abnormal ratio metric based on the abnormal point ratio and abnormal hour ratio.
[0204] In some embodiments, when the evaluation metric includes a periodic matching metric and the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value, the first determination module 303 may also be used to:
[0205] Extract m data segments from the key performance indicators, the upper bound value of the dynamic threshold, and the lower bound value of the dynamic threshold, respectively, each data segment includes a preset number of data of multiple consecutive cycles, and m is an integer greater than 1;
[0206] For each data segment, cyclic autocorrelation processing is performed, and the processed data is subjected to Fourier transform processing to obtain the periodicity score corresponding to the data segment;
[0207] Determining a first periodicity score of the key performance indicator according to the m periodicity scores corresponding to the m data segments of the key performance indicator;
[0208] Determine a second periodicity fraction of the upper limit value of the dynamic threshold according to the m periodicity fractions corresponding to the m data segments of the upper limit value of the dynamic threshold;
[0209] Determine a third periodicity fraction of the dynamic threshold lower limit value according to the m periodicity fractions corresponding to the m data segments of the dynamic threshold lower limit value;
[0210] A periodicity matching metric is determined based on the first periodicity score, the second periodicity score, and the third periodicity score.
[0211] In some embodiments, when the evaluation metric includes a repeatability false alarm metric, the output result includes an abnormal output, and the first determination module 303 may also be used to:
[0212] Initialize the abnormal hours list and cumulative hours list within the preset period;
[0213] According to the abnormal output, determine each abnormal hour in the abnormal hours list and each cumulative hour in the cumulative hours list;
[0214] Determine multiple repeated abnormality ratios within a preset period based on each abnormal hour and each cumulative hour;
[0215] The maximum value among multiple repeated abnormal ratios is determined as the repeatability false alarm rate;
[0216] Based on the repeatability false alarm rate, determine the repeatability false alarm measure.
[0217] In some embodiments, when the evaluation metric includes a smoothness metric, the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value, and the first determination module 303 may also be used to:
[0218] According to the upper limit value of the dynamic threshold, determine the sliding average sequence of the upper limit value, and according to the lower limit value of the dynamic threshold, determine the sliding average sequence of the lower limit value;
[0219] Determine the first variance of the difference between the upper limit value sliding average sequence and the upper limit value of the dynamic threshold;
[0220] Determine the upper limit value smoothness value based on the first variance and the upper limit value sliding average sequence;
[0221] Determine the second variance of the difference between the lower limit value sliding average sequence and the lower limit value of the dynamic threshold;
[0222] Determine the smoothness value of the lower limit value according to the second variance and the lower limit value sliding average sequence;
[0223] A smoothness measure is determined based on the upper bound smoothness value and the lower bound smoothness value.
[0224] In some embodiments, when the evaluation metric includes an underlying distribution metric of anomaly scores, the output result includes a dynamic threshold upper limit value and a dynamic threshold lower limit value, and the first determination module 303 may also be used to:
[0225] Determine an upper threshold abnormality score after the dynamic threshold upper threshold value is transformed, and a lower threshold abnormality score after the dynamic threshold lower threshold value is transformed;
[0226] Obtaining a preset number of thresholds within the value range of the upper boundary value anomaly score to obtain an upper boundary value threshold list, and obtaining a preset number of thresholds within the value range of the lower boundary value anomaly score to obtain a lower boundary value threshold list;
[0227] Traverse the upper limit value threshold list and the lower limit value threshold list to determine their corresponding upper limit value abnormal probability list and lower limit value abnormal probability list;
[0228] Determine the underlying distribution score of the abnormal score of the upper boundary value according to the abnormal probability list of the upper boundary value, and determine the underlying distribution score of the abnormal score of the lower boundary value according to the abnormal probability list of the lower boundary value;
[0229] The anomaly score underlying distribution metric is determined according to the anomaly score underlying distribution score of the upper boundary value and the anomaly score underlying distribution score of the lower boundary value.
[0230] In some embodiments, the first determining module 303 may also be used to:
[0231] When the probability in the upper bound abnormal probability list does not satisfy monotonically decreasing, the upper bound abnormal score underlying distribution score is determined to be 0, and,
[0232] When the probability in the lower bound abnormal probability list does not satisfy monotonically decreasing, the lower bound abnormal score underlying distribution score is determined to be 0.
[0233] In some embodiments, the first determining module 303 may also be used to:
[0234] When the probability in the upper limit abnormal probability list satisfies monotonically decreasing, the upper limit abnormal probability list is compared with the ideal probability distribution curve to obtain the underlying distribution score of the upper limit abnormal score, and,
[0235] When the probability in the lower bound abnormal probability list satisfies monotonically decreasing, the lower bound abnormal probability list is compared with the ideal probability distribution curve to obtain the underlying distribution score of the lower bound abnormal score.
[0236] In some embodiments, the second determining module 304 may also be used to:
[0237] Based on the evaluation metrics, an evaluation report of the anomaly detection model is generated, and the evaluation results include the evaluation report.
[0238] In some embodiments, the evaluation metrics include an anomaly ratio metric, a periodicity matching metric, a repetitive false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric.
[0239] The second determining module 304 may also be used for:
[0240] An evaluation score of the anomaly detection model is determined based on the anomaly ratio metric, the periodic matching metric, the repeatability false alarm metric, the smoothness metric, and the anomaly score underlying distribution metric, and their respective corresponding weights. The evaluation result includes the evaluation score.
[0241] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and are devices corresponding to the above-mentioned anomaly detection model evaluation method. All implementation methods in the above-mentioned method embodiment are applicable to the embodiments of the device. Its specific functions and technical effects can be found in the method embodiment part, which will not be repeated here.
[0242] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0243] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application is shown.
[0244] The device may include a processor 401 and a memory 402 storing programs or instructions.
[0245] When the processor 401 executes the program, the steps in any of the above method embodiments are implemented.
[0246] Exemplarily, the program may be divided into one or more modules / units, one or more modules / units are stored in the memory 402, and executed by the processor 401 to complete the present application. One or more modules / units may be a series of program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the program in the device.
[0247] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0248] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.
[0249] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) machine-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.
[0250] The processor 401 implements any one of the methods in the above embodiments by reading and executing the program or instruction stored in the memory 402 .
[0251] In one example, the electronic device may further include a communication interface 403 and a bus 404. The processor 401, the memory 402, and the communication interface 403 are connected via the bus 404 and communicate with each other.
[0252] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0253] Bus 404 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 404 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.
[0254] In addition, in combination with the method in the above embodiment, the embodiment of the present application may provide a machine-readable storage medium for implementation. The machine-readable storage medium stores a program or instruction; when the program or instruction is executed by a processor, any one of the methods in the above embodiment is implemented. The machine-readable storage medium can be read by a machine such as a computer.
[0255] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0256] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0257] An embodiment of the present application provides a computer program product, which is stored in a machine-readable storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0258] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0259] The functional modules shown in the above block diagram can be implemented as hardware, software, firmware or their combination. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.
[0260] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0261] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by a computer program or instruction. These programs or instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0262] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A method for evaluating anomaly detection models, characterized in that: The method comprises: Obtain key performance indicators within a preset time range at preset time intervals; Inputting the key performance indicator into the anomaly detection model to be evaluated to obtain an output result of the anomaly detection model, wherein the output result includes at least one of a dynamic threshold upper limit value, a dynamic threshold lower limit value, and an abnormal output; Determine an evaluation metric based on the output result, the evaluation metric comprising at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric; An evaluation result of the anomaly detection model is determined based on the evaluation metric.
2. The method according to claim 1, characterized in that: In a case where the evaluation metric includes the anomaly ratio metric, the output result includes the anomaly output; Determining the evaluation metric according to the output result includes: According to the abnormal output, determine the abnormal point ratio and the abnormal hour ratio, wherein the abnormal output includes multiple sub-outputs, and each sub-output carries time information, the abnormal point ratio is used to indicate the ratio of the number of the first sub-output to the total number of sub-outputs in the abnormal output, and the abnormal hour ratio is used to indicate the ratio of the number of hours including the first sub-output in each hour of the sub-output to the total hours, and the result of the first sub-output is abnormal; The abnormal ratio metric is determined according to the abnormal point proportion and the abnormal hour proportion.
3. The method according to claim 1, characterized in that In the case where the evaluation metric includes the periodic matching metric, the output result includes the dynamic threshold upper limit value and the dynamic threshold lower limit value; Determining the evaluation metric according to the output result includes: Extract m data segments from the key performance indicator, the upper limit value of the dynamic threshold, and the lower limit value of the dynamic threshold, respectively, each of the data segments includes a preset number of data of multiple consecutive cycles, and m is an integer greater than 1; For each data segment, a cyclic autocorrelation process is performed, and the processed data is subjected to a Fourier transform process to obtain a periodicity score corresponding to the data segment; Determining a first periodicity score of the key performance indicator according to the m periodicity scores corresponding to the m data segments of the key performance indicator; Determine a second periodicity fraction of the upper limit value of the dynamic threshold according to the m periodicity fractions corresponding to the m data segments of the upper limit value of the dynamic threshold; Determine a third periodicity fraction of the dynamic threshold lower limit value according to the m periodicity fractions corresponding to the m data segments of the dynamic threshold lower limit value; A periodicity matching metric is determined based on the first periodicity score, the second periodicity score, and the third periodicity score.
4. The method according to claim 1, characterized in that: In a case where the evaluation metric includes the repeatability false alarm metric, the output result includes the abnormal output; Determining the evaluation metric according to the output result includes: Initialize the abnormal hours list and cumulative hours list within the preset period; Determine, based on the abnormal output, each abnormal hour in the abnormal hours list and each accumulated hour in the accumulated hours list; Determine a plurality of repeated abnormality ratios within the preset period according to each abnormal hour and each accumulated hour; Determine the maximum value among the multiple repeated abnormal ratios as the repeatability false alarm rate; The repeatability false alarm metric is determined according to the repeatability false alarm rate.
5. The method according to claim 1, characterized in that In the case where the evaluation metric includes the smoothness metric, the output result includes the dynamic threshold upper boundary value and the dynamic threshold lower boundary value; Determining the evaluation metric according to the output result includes: According to the upper limit value of the dynamic threshold, determine the sliding average sequence of the upper limit value, and according to the lower limit value of the dynamic threshold, determine the sliding average sequence of the lower limit value; Determine a first variance of a difference between the upper limit value sliding average sequence and the upper limit value of the dynamic threshold; Determine an upper limit value smoothness value according to the first variance and the upper limit value sliding average sequence; Determine a second variance of the difference between the lower limit value sliding average sequence and the dynamic threshold lower limit value; Determine a lower limit value smoothness value according to the second variance and the lower limit value sliding average sequence; The smoothness metric is determined based on the upper bound smoothness value and the lower bound smoothness value.
6. The method according to claim 1, characterized in that In the case where the evaluation metric includes the anomaly score underlying distribution metric, the output result includes the dynamic threshold upper bound value and the dynamic threshold lower bound value; Determining the evaluation metric according to the output result includes: Determine an upper threshold abnormality score after the dynamic threshold upper threshold value is converted, and a lower threshold abnormality score after the dynamic threshold lower threshold value is converted; Obtaining a preset number of thresholds within the value range of the upper threshold anomaly score to obtain an upper threshold list, and obtaining the preset number of thresholds within the value range of the lower threshold anomaly score to obtain a lower threshold list; Traversing the upper threshold value list and the lower threshold value list to determine the upper abnormal probability list and the lower abnormal probability list corresponding to each of them; Determine the underlying distribution score of the abnormal score of the upper boundary value according to the upper boundary value abnormal probability list, and determine the underlying distribution score of the abnormal score of the lower boundary value according to the lower boundary value abnormal probability list; The anomaly score underlying distribution metric is determined according to the anomaly score underlying distribution score of the upper boundary value and the anomaly score underlying distribution score of the lower boundary value.
7. The method according to claim 6, characterized in that The step of determining the underlying distribution score of the abnormal score of the upper boundary value according to the upper boundary value abnormal probability list, and determining the underlying distribution score of the abnormal score of the lower boundary value according to the lower boundary value abnormal probability list, comprises: In the case where the probability in the upper limit abnormal probability list does not satisfy monotonically decreasing, the abnormal score underlying distribution score of the upper limit is determined to be 0, and, When the probabilities in the lower bound abnormal probability list do not satisfy monotonically decreasing trend, the abnormal score underlying distribution score of the lower bound is determined to be 0.
8. The method according to claim 7, characterized in that The step of determining the abnormal score underlying distribution score of the upper boundary value according to the upper boundary value abnormal probability list, and determining the abnormal score underlying distribution score of the lower boundary value according to the lower boundary value abnormal probability list, further includes: When the probability in the upper limit abnormal probability list satisfies monotonically decreasing, the upper limit abnormal probability list is compared with the ideal probability distribution curve to obtain the abnormal score underlying distribution score of the upper limit, and, When the probabilities in the lower limit value abnormal probability list satisfy monotonically decreasing trend, the lower limit value abnormal probability list is compared with the ideal probability distribution curve to obtain the abnormal score underlying distribution score of the lower limit value.
9. An anomaly detection model evaluation device, characterized in that: The device comprises: An acquisition module is used to acquire key performance indicators within a preset time range at preset time intervals; A detection module, used for inputting the key performance indicator into the anomaly detection model to be evaluated, and obtaining an output result output by the anomaly detection model, wherein the output result includes at least one of a dynamic threshold upper limit value, a dynamic threshold lower limit value, and an abnormal output; A first determination module is used to determine an evaluation metric according to the output result, wherein the evaluation metric includes at least two of an anomaly ratio metric, a periodic matching metric, a repeatability false alarm metric, a smoothness metric, and an anomaly score underlying distribution metric; The second determination module is used to determine the evaluation result of the anomaly detection model according to the evaluation metric.
10. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 8.
