Dynamic threshold method and system based on median and absolute median deviation

Abstract

According to the dynamic threshold value method and system based on the median and absolute median deviation, the median is used for replacing the average value, and compared with the average value, the median is not sensitive to an extreme value, so that the influence of noise data on a result can be effectively reduced, and the set threshold value is more stable and reliable; the improved absolute median deviation is used for replacing the standard deviation, so that the insensitivity of the median to an abnormal value is kept, and for a data set in non-normal distribution, more stable dispersion estimation is provided; besides, a dynamically adjusted threshold value coefficient is designed, the threshold value is automatically adjusted according to the characteristics of a data set, it is ensured that the change trend of data can be more sensitively captured, and meanwhile the phenomenon of false alarm or missing alarm caused by improper setting of a fixed threshold value is avoided. After the method is adopted, the short-term fluctuation and the long-term trend of the data can be reflected more accurately, and a more reliable basis is provided for decision makers.

Description

Technical Field

[0001] The invention relates to a dynamic threshold method and system based on median and absolute median deviation, and belongs to the field of database anomaly detection. Background Art

[0002] At present, no matter which industry, because data is very important, the database responsible for data storage has been given an important role accordingly. In order to maintain database stability and high database performance, it is an important and necessary means to monitor key indicators such as database query response time, throughput, cache hit rate, and number of concurrent connections, and to detect anomalies and issue alarms in a timely manner.

[0003] Dynamic threshold is a commonly used method in the field of database anomaly detection. This method dynamically adjusts the indicator threshold according to data change trends or seasonal changes. Compared with the static threshold method that sets the threshold and does not change, the dynamic threshold method helps improve the accuracy of anomaly detection.

[0004] Existing dynamic threshold methods are mainly implemented using methods such as statistics and machine learning. These methods often have some limitations. For example, many methods are based on mean and standard deviation, are very sensitive to noise data, and have difficulty processing data with large changes, and have poor robustness. For another example, some methods use fixed coefficients to determine the width of the threshold. The impact of fixed coefficients is similar to that of static thresholds. They cannot dynamically synchronize with the trend of data changes, which can easily lead to oversensitivity or undersensitivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 This is a diagram of the system structure of the present invention.

[0006] Figure 2 The figure is a calculation flow chart of the present invention. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a dynamic threshold method based on median and absolute median deviation, which comprises the following steps:

[0008] (1) Data collection and preprocessing: Collect key indicator data of the controlled database, organize the collected data, and process missing values ​​and abnormal values;

[0009] (2) Dynamic threshold calculation: Calculate the dynamic upper and lower thresholds of each key indicator of the controlled database;

[0010] (3) Real-time anomaly detection: monitor newly arrived data points in real time and detect whether there are anomalies based on the upper and lower thresholds of the dynamic threshold. If there are anomalies, an alarm is triggered;

[0011] (4) System configuration: allows users to configure parameters in the method;

[0012] Among them, the dynamic threshold is calculated by using the dynamic threshold method based on the median and absolute median deviation. The data of each indicator this week are predicted based on the historical data of each indicator in the past four weeks to calculate the dynamic threshold.

[0013] Furthermore, step (2) comprises the following steps:

[0014] 2.1 Calculate the median and MAD at the hourly level

[0015] Dataset D contains hourly data points for each metric over the past four weeks. is the data subset of the ith indicator in the hth hour in the past wth week, where w=1,2,3,4, 1≤i≤N, N is the total number of indicators that the user is concerned about; 1≤h≤24×7, indicating the current time in a week. Calculate the median of the ith indicator in the hth hour Among them, median() is the median calculation function, For the dataset The median of all data in; Calculate the absolute median deviation of the i-th indicator at the h-th hour Where C is a constant coefficient, For the dataset Each data in The median of the absolute value of the difference, After expansion, it is equivalent to

[0016] 2.2 Calculate the base MAD for each indicator

[0017] For each indicator i, calculate the absolute median deviation from a benchmark using all values ​​of that indicator over the past 4 weeks , where data set D i is a data subset consisting of all data of the ith indicator in the past 4 weeks, is the median of the ith indicator in the past 4 weeks, median(D i ) is the data set D i The median of all data in For the dataset D i Each data in The median of the absolute values ​​of the differences.

[0018] 2.3 Dynamically adjust the threshold coefficient

[0019] Calculate MAD h,i and MADi Ratio Design piecewise functions The coefficient α controls the steepness of the function;

[0020] 2.4 Calculating dynamic thresholds

[0021] Calculate the dynamic threshold of the ith indicator at the hth hour, that is, the upper and lower thresholds, which are U h,i and L h,i , Among them, k is the coefficient.

[0022] Based on the above method, the present invention proposes a dynamic threshold system based on median and absolute median deviation, which includes a data acquisition and preprocessing module, a dynamic threshold calculation module, a real-time anomaly detection module and a system configuration module. Specifically:

[0023] (1) Data collection and preprocessing module: This module collects key indicator data of the controlled database, organizes the collected data, and processes missing values ​​and abnormal values;

[0024] (2) Dynamic threshold calculation module: calculates the dynamic upper and lower thresholds of each key indicator of the controlled database;

[0025] (3) Real-time anomaly detection module: This module monitors newly arrived data points in real time and detects whether there are anomalies based on the upper and lower thresholds of the dynamic threshold. If there are anomalies, an alarm is triggered;

[0026] (4) System configuration module: This module allows users to configure the parameters in the method;

[0027] Among them, the dynamic threshold calculation module adopts a dynamic threshold method based on the median and absolute median deviation, predicts the data of each indicator this week according to the historical data of each indicator in the past four weeks, and calculates the dynamic threshold.

[0028] Furthermore, the dynamic threshold calculation module includes the following steps:

[0029] 2.1 Calculate the median and MAD at the hourly level

[0030] Dataset D contains hourly data points for each metric over the past four weeks. is the data subset of the ith indicator in the hth hour in the past wth week, where w=1,2,3,4, 1≤i≤N, N is the total number of indicators that the user is concerned about; 1≤h≤24×7, indicating the current time in a week. Calculate the median of the ith indicator in the hth hour Among them, median() is the median calculation function, For the dataset The median of all data in; Calculate the absolute median deviation of the i-th indicator at the h-th hour Where C is a constant coefficient, For the dataset Each data in The median of the absolute value of the difference between After expansion, it is equivalent to

[0031] 2.2 Calculate the base MAD for each indicator

[0032] For each indicator i, calculate the absolute median deviation from a benchmark using all values ​​of that indicator over the past 4 weeks Among them, the data set D i is a data subset consisting of all data of the ith indicator in the past 4 weeks, is the median of the ith indicator in the past 4 weeks, median(D i ) is the data set D i The median of all data in For the dataset D i Each data in The median of the absolute values ​​of the differences.

[0033] 2.3 Dynamically adjust the threshold coefficient

[0034] Calculate MAD h,i and MAD i Ratio Design piecewise functions The coefficient α controls the steepness of the function;

[0035] 2.4 Calculating dynamic thresholds

[0036] Calculate the dynamic threshold of the ith indicator at the hth hour, that is, the upper and lower thresholds, which are U h,i and L h,i , Among them, k is the coefficient.

[0037] The method of the present invention uses the median instead of the mean. Compared with the mean, the median is insensitive to extreme values, so it can effectively reduce the impact of noise data on the results, so that the set threshold is more stable and reliable; it also uses an improved absolute median deviation instead of the standard deviation, which not only maintains the insensitivity of the median to outliers, but also provides a more robust discreteness estimate for non-normally distributed data sets; in addition, a dynamically adjusted threshold coefficient is designed to automatically adjust the size of the threshold according to the characteristics of the data set, ensuring that the change trend of the data can be captured more sensitively, while avoiding false positives or omissions caused by improper setting of fixed thresholds. After adopting the present invention, both short-term fluctuations and long-term trends of the data can be more accurately reflected, providing a more reliable basis for decision makers. DETAILED DESCRIPTION

[0038] Example 1

[0039] In order to solve the above problems, the present invention proposes a dynamic threshold method based on median and absolute median deviation. According to the above method, the present invention designs a dynamic threshold system, the software structure of which is as follows: Figure 1 As shown. Figure 1 It can be seen that the system mainly includes data acquisition and preprocessing module, dynamic threshold calculation module, real-time anomaly detection module and system configuration module. The main functions of each module are:

[0040] (1) Data collection and preprocessing module: This module is responsible for collecting key indicator data of the controlled database, organizing the collected data, processing missing values ​​and abnormal values, and preparing for subsequent processing steps.

[0041] (2) Dynamic threshold calculation module: According to the dynamic threshold method based on median and absolute median deviation, the dynamic upper and lower thresholds of each key indicator of the controlled database are calculated.

[0043] (3) Real-time anomaly detection module: monitors newly arrived data points in real time and detects whether there are anomalies based on the upper and lower thresholds of the dynamic threshold. If there are anomalies, an alarm is triggered.

[0044] (4) System configuration module: This module allows users to configure the parameters in the method.

[0045] In this system, the core module is the dynamic threshold calculation module, which mainly adopts the dynamic threshold method based on median and absolute median deviation designed by the present invention. This method predicts the data of each indicator this week based on the historical data of each indicator in the past 4 weeks and calculates its dynamic threshold. The calculation process of this method is as follows Figure 2As shown. According to the above flow chart, the specific steps of the method are:

[0046] (1) Data collection and preprocessing: Collect key indicator data of the controlled database, organize the collected data, and process missing values ​​and abnormal values. After preprocessing, on the one hand, the latest collected data is sent to the real-time anomaly detection step for anomaly detection, and on the other hand, the latest collected data is stored in the system database.

[0047] (2) Calculate the median and MAD at the hourly level: Data set D contains the hourly data points of each indicator in the past four weeks. It is represented as the data subset of the i-th indicator in the h-th hour in the past w-th week, where w=1,2,3,4, 1≤i≤N, N represents the total number of indicators that the user is concerned about; 1≤h≤24*7, represents the current time in a week. The median of the i-th indicator at the h-th hour can be calculated The calculation formula is:

[0048]

[0049] Among them, median() is the median calculation function, Representation dataset The median of all data in .

[0050] The absolute median deviation of the ith indicator at the hth hour is recorded as MAD h,i , and its calculation formula is:

[0051]

[0052] Where C is a constant coefficient, Representation dataset Each data in The median of the absolute value of the difference, After expansion, it is equivalent to

[0053] (3) Calculate the benchmark MAD of each indicator: For each indicator i, use all the values ​​of the indicator in the past 4 weeks to calculate the absolute median deviation of a benchmark, recorded as MAD i , and its calculation formula is:

[0054]

[0055] Among them, the data set D i represents the data subset consisting of all data of the i-th indicator in the past 4 weeks, It represents the median of the ith indicator in the past 4 weeks, and its calculation formula is:

[0056]

[0057] median(D i ) represents the data set D i The median of all data in Denotes the dataset D i Each data in The median of the absolute values ​​of the differences.

[0058] (4) Dynamically adjust the threshold coefficient

[0059] Calculate MAD h,i and MAD i The ratio of is denoted as r, that is:

[0060]

[0061] Design a piecewise function f(r), whose function value will adjust the threshold coefficient in the next step. Its expression is:

[0062]

[0063] The coefficient α is used to control the steepness of the function.

[0064] (5) Dynamic threshold calculation

[0065] Calculate the dynamic threshold of the ith indicator at the hth hour, that is, the upper and lower thresholds, which are represented by U h,i and L h,i , and the calculation formulas are:

[0066]

[0067]

[0068] Among them, k is the coefficient.

[0069] (6) Real-time anomaly detection

[0070] For each newly arrived data point x, check whether it exceeds the upper and lower bounds of the corresponding indicator and the corresponding hourly dynamic threshold.

[0071] (7) Output results

[0072] If a data point x exceeds the threshold, an anomaly detection alert is triggered immediately.

[0073] It should be noted that in the actual operation of the algorithm, according to the actual situation, it can be selected to perform the MAD h,i , MAD i and r are updated to reflect the latest data distribution.

[0074] The present invention can dynamically adjust the threshold value according to the actual fluctuation of the data, thereby improving the accuracy of anomaly detection.

[0075] In order to illustrate the process of the whole method, the present invention provides specific embodiments as follows:

[0076] Assume that the total number of indicators that users are concerned about is N=3, and the current time is 9:00 a.m. on Monday, that is, h=9. The data at 9:00 a.m. every Monday in the past four weeks are shown in the following table. For simplicity, only the average value within the hour is provided.

[0077] Weekly Indicator 1 Indicator 2 Indicator 3 1 10 20 30 2 12 22 32 3 14 24 34 4 16 26 36

[0078] The average daily indicator data for the past four weeks are shown in the following table. For simplicity, only the average for one week is provided.

[0079] index on Monday Tuesday Wednesday Thursday Friday Saturday Sunday Indicator 1 10 11 12 13 14 15 16 Indicator 2 20 21 22 23 24 25 26 Indicator 3 30 31 32 33 34 35 36

[0080] In the actual calculation process, C=1.4826, k=1.5, α=0.5 are taken.

[0081] According to the above settings, the specific steps of this method are:

[0082] (1) Data collection and preprocessing: Collect key indicator data of the controlled database, organize the collected data, and process missing values ​​and abnormal values. After preprocessing, on the one hand, the latest collected data is sent to the real-time anomaly detection step for anomaly detection, and on the other hand, the latest collected data is stored in the system database.

[0083] In the above practical example, this step has no actual operation content.

[0084] (2) Calculate the median and MAD at the hourly level:

[0085] 1) Calculate the median: The median of indicator 1 at 9 am on Monday is:

[0086]

[0087] Similarly, the median value of indicator 2 at 9 am on Monday is:

[0088]

[0089] The median values ​​for indicator 3 at 9:00 am on Monday were:

[0090]

[0091] 2) Calculate MAD: The MAD of indicator 1 at 9 am on Monday is MAD 9,1 :

[0092] MAD 9,1 =1.4826×median(|10-13|,|12-13|,|14-13|,|16-13|)

[0093] =1.4826×2=2.9652

[0094] Similarly, the MAD of indicator 2 at 9 am on Monday is MAD 9,2 :

[0095] MAD 9,2 =2.9652

[0096] Indicator 3's MAD at 9 am on Monday is MAD 9,3 :

[0097] MAD 9,3 =2.9652

[0098] (3) Calculate the benchmark MAD of each indicator:

[0099] The median value of the first indicator in the past 4 weeks is:

[0100]

[0101] Similarly, the median of the second indicator in the past 4 weeks is:

[0102]

[0103] The median value of the third indicator in the past 4 weeks is:

[0104]

[0105] Therefore, the base MAD of indicator 1 is MAD1:

[0106]

[0107] Similarly, the benchmark MAD of indicator 2 is MAD2=2.9652, and the benchmark MAD of indicator 3 is MAD3=2.9652.

[0108] (4) Dynamically adjust the threshold coefficient

[0109] Calculate MAD 9,1 The ratio of MAD1 is denoted as r1, that is:

[0110]

[0111] Similarly, MAD 9,2 The ratio of MAD2 is denoted as r2=1; MAD 9,3 The ratio of MAD3 is denoted as r3=1.

[0112] According to the definition of piecewise function f(r),

[0113] f(r1)=f(1)=1

[0114] By the same token, f(r2)=1, f(r3)=1.

[0115] (5) Dynamic threshold calculation

[0116] Calculate the upper and lower thresholds U of the first indicator at 9 am on Monday 9,1 and L 9,1 , and the calculation formulas are:

[0117] U 9,1 =13+1×1.5×2.9652=17.4478

[0118] L 9,1 =13-1×1.5×2.9652=8.5522

[0119] Similarly, the upper and lower thresholds U of the second indicator at 9 am on Monday 9,2 and L 9,2 The upper and lower thresholds of the third indicator at 9 am on Monday are 27.4478 and 18.5522 respectively; 9,3 and L 9,3 They are 37.4478 and 28.5522 respectively.

[0120] (6) Real-time anomaly detection

[0121] For each newly arrived data point x, check whether it exceeds the upper and lower bounds of the corresponding indicator and the corresponding hourly dynamic threshold.

[0122] Assume that the newly arrived data point is (x1,x2,x3) = (11,23,33), because x1∈[L 9,1 ,U 9,1 ]=(8.5522,17.4478),x2∈[L 9,2 ,U 9,2 ]=(18.5522,27.4478),x3∈[L 9,3 ,U 9,3 ]=(28.5522,37.4478), so all three indicators are normal.

[0123] Assume that the newly arrived data point is (x1,x2,x3) = (16,29,39), because x1∈[L 9,1 ,U 9,1 ]=(8.5522,17.4478), Therefore, indicator 1 is normal, and indicators 2 and 3 are abnormal.

[0124] (7) Output results

[0125] If a data point x exceeds the threshold, an anomaly detection alert is triggered immediately.

[0126] From this practical example, we can see that the algorithm uses the median instead of the mean. If there is noisy data, it can also avoid the impact of noise to a certain extent. At the same time, the threshold can be dynamically adjusted according to historical data, so that the threshold can better reflect the changing trend of the current data and improve the accuracy of anomaly detection. In addition, the algorithm uses multiple historical data windows (4 weeks) to calculate the median and MAD, making the algorithm more robust. In short, the algorithm is practical and flexible in the anomaly detection process, and can maintain good detection results in a constantly changing environment.

Claims

1. A dynamic threshold method based on median and absolute median deviation, the method comprising the following steps: (1) Data collection and preprocessing: Collect key indicator data of the controlled database, organize the collected data, and process missing values ​​and abnormal values; (2) Dynamic threshold calculation: Calculate the dynamic upper and lower thresholds of each key indicator of the controlled database; (3) Real-time anomaly detection: monitor newly arrived data points in real time and detect whether there are anomalies based on the upper and lower thresholds of the dynamic threshold. If there are anomalies, an alarm is triggered; (4) System configuration: allows users to configure parameters in the method; The characteristic is that: the dynamic threshold is calculated by using the dynamic threshold method based on the median and the absolute median deviation, and the data of each indicator this week is predicted according to the historical data of each indicator in the past 4 weeks to calculate the dynamic threshold.

2. A dynamic threshold method based on median and absolute median deviation as claimed in claim 1, characterized in that: Step (2) comprises the following steps: 2.1 Calculate the median and MAD at the hourly level Dataset D contains hourly data points for each metric over the past four weeks. is the data subset of the ith indicator in the hth hour in the past wth week, where w=1,2,3,4, 1≤i≤N, N is the total number of indicators that the user is concerned about; 1≤h≤24×7, indicating the current time in a week. Calculate the median of the ith indicator in the hth hour Among them, median() is the median calculation function, For the dataset The median of all data in; Calculate the absolute median deviation of the i-th indicator at the h-th hour Where C is a constant coefficient, For the dataset Each data in The median of the absolute value of the difference, After expansion, it is equivalent to 2.2 Calculate the base MAD for each indicator For each indicator i, calculate the absolute median deviation from a benchmark using all values ​​of that indicator over the past 4 weeks Among them, the data set D i is a data subset consisting of all data of the ith indicator in the past 4 weeks, is the median of the ith indicator in the past 4 weeks, median(D i ) is the data set D i The median of all data in For the dataset D i Each data in The median of the absolute values ​​of the differences. 2.3 Dynamically adjust the threshold coefficient Calculate MAD h,i and MAD i Ratio Design piecewise functions The coefficient α controls the steepness of the function; 2.4 Calculating dynamic thresholds Calculate the dynamic threshold of the ith indicator at the hth hour, that is, the upper and lower thresholds, which are U h,i and L h,i , Among them, k is the coefficient.

3. A dynamic threshold system based on median and absolute median deviation, the system comprising a data acquisition and preprocessing module, a dynamic threshold calculation module, a real-time anomaly detection module and a system configuration module, specifically: (1) Data collection and preprocessing module: This module collects key indicator data of the controlled database, organizes the collected data, and processes missing values ​​and abnormal values; (2) Dynamic threshold calculation module: calculates the dynamic upper and lower thresholds of each key indicator of the controlled database; (3) Real-time anomaly detection module: This module monitors newly arrived data points in real time and detects whether there are anomalies based on the upper and lower thresholds of the dynamic threshold. If there are anomalies, an alarm is triggered; (4) System configuration module: This module allows users to configure the parameters in the method; Features: The dynamic threshold calculation module adopts a dynamic threshold method based on the median and absolute median deviation. It predicts the data of each indicator this week based on the historical data of each indicator in the past four weeks and calculates the dynamic threshold.

4. A dynamic threshold system based on median and absolute median deviation as claimed in claim 3, characterized in that: The dynamic threshold calculation module includes the following steps: 2.1 Calculate the median and MAD at the hourly level Dataset D contains hourly data points for each metric over the past four weeks. is the data subset of the ith indicator in the hth hour in the past wth week, where w=1,2,3,4, 1≤i≤N, N is the total number of indicators that the user is concerned about; 1≤h≤24×7, indicating the current time in a week. Calculate the median of the ith indicator in the hth hour Among them, median() is the median calculation function, For the dataset The median of all data in; Calculate the absolute median deviation of the i-th indicator at the h-th hour Where C is a constant coefficient, For the dataset Each data in The median of the absolute value of the difference, After expansion, it is equivalent to 2.2 Calculate the base MAD for each indicator For each indicator i, calculate the absolute median deviation from a benchmark using all values ​​of that indicator over the past 4 weeks Among them, the data set D i is a data subset consisting of all data of the ith indicator in the past 4 weeks, is the median of the ith indicator in the past 4 weeks, median(D i ) is the data set D i The median of all data in For the dataset D i Each data in The median of the absolute values ​​of the differences. 2.3 Dynamically adjust the threshold coefficient Calculate MAD h,i and MAD i Ratio Design piecewise functions The coefficient α controls the steepness of the function; 2.4 Calculating dynamic thresholds Calculate the dynamic threshold of the ith indicator at the hth hour, that is, the upper and lower thresholds, which are U h,i and L h,i , Among them, k is the coefficient.

Images