Self-adaptive adjustment method for time sequence anomaly detection threshold

Abstract

The invention discloses a self-adaptive adjustment method for a time sequence anomaly detection threshold, which dynamically estimates an anomaly upper bound according to anomaly score distribution in a time sequence local window, realizes self-adaptive adjustment of the detection threshold, improves the anomaly detection accuracy, reduces the false alarm rate and the omission ratio, and improves the detection accuracy. And good robustness and practicability are shown.

Description

Technical Field

[0001] The present invention belongs to the field of multivariate time series anomaly detection and relates to an adaptive adjustment method for the anomaly detection threshold of time series. Background Art

[0002] A key step in time series anomaly detection is to set a threshold for the anomaly score to determine whether each moment is an anomaly. Traditional methods usually use a fixed global threshold to mark the moments above the threshold as anomalies. However, this "one-size-fits-all" approach ignores the dynamic characteristics and local change patterns of time series data, resulting in poor detection performance. Existing fixed threshold methods face three main challenges, leading to false positives or false negatives: (i) Time series data usually exhibits obvious non-stationarity, such as trends, cycles, fluctuations, etc.; (ii) Anomaly is a relative concept and should be dynamically determined according to the context; (iii) Real-world data often contains various noises and interferences, such as sensor failures, data omissions, etc. Summary of the Invention

[0003] To solve these problems, the present invention provides an adaptive adjustment method for the anomaly detection threshold of time series. According to the anomaly score distribution within the local window of the time series, the upper bound of the anomaly is dynamically estimated, realizing the adaptive adjustment of the detection threshold, improving the accuracy of anomaly detection, reducing the false positive rate and the missed detection rate, and showing good robustness and practicality.

[0004] The technical solution of the present invention is as follows:

[0005] An adaptive adjustment method for the anomaly detection threshold of time series dynamically estimates the upper bound of the anomaly according to the anomaly score distribution within the local window of the time series, realizing the adaptive adjustment of the detection threshold;

[0006] The adaptive threshold calculation method introduces the setting of a sliding window, uses the exponentially weighted moving average method to smooth the anomaly scores, and sets the threshold according to the difference between the smoothed anomaly scores and the moving average.

[0007] It includes the following steps:

[0008] (1). Use the fast Fourier transform (FFT) to obtain the main frequency from the original data and extract the main features of the time series data. This step helps to understand the periodicity and fluctuation characteristics of the data. By ignoring the DC component, the FFT calculation result will extract the frequency domain features of the data. From the FFT results, select the frequency with the largest amplitude as the main frequency f top . This main frequency f top The reciprocal of will be used as the period length w, that is, the size of the sliding window p, so that the adaptive threshold can flexibly cope with the periodic changes of the data:

[0009] f top = Amp max (FFT(x))

[0010] w = 1 / f top

[0011] where x is the input data, and its size is (batch_size, win_size, dim), which is the shape of the input data; Amp max is the amplitude of the complex spectrum obtained after FFT;

[0012] (2) Set the initial threshold, EWMA value, and the initial value of the anomaly score to ensure a reasonable starting point at the beginning for subsequent calculations:

[0013] Ewma 0 = Thr 0 = Sum 0 = S 0

[0014] where Ewma 0 is the exponentially weighted moving average value at time 0, Thr 0 is the threshold at time 0, Sum 0 is the cumulative anomaly score at time 0, and S 0 is the anomaly score at time 0;

[0015] (3) Dynamically set the length p of the sliding window based on the dominant frequency calculated by FFT. The size of the sliding window will be automatically adjusted according to the periodicity of the data, which can ensure that the statistical characteristics within each window are roughly consistent, thereby improving the accuracy of anomaly detection;

[0016] (4) At each time t, use the exponentially weighted moving average (EWMA) algorithm to update the EWMA value at the current time. The exponentially weighted moving average method is used to smooth the input value to better capture the long-term trend in the data; Given the anomaly score S t at the current time step, its corresponding exponentially weighted moving average value EWMA t can be updated through the following recursive relationship:

[0017] Ewma t+1 = α·Ewma t + (1 - α)·S t

[0018] where Ewma t+1is the exponentially weighted moving average for the next moment. α is the EWMA decay factor and satisfies α ∈ (0, 1) to control the degree of context relevance, and 0.9 is taken. Reasonably setting the EWMA decay factor can make the system neither overly sensitive to instantaneous fluctuations nor ignore potential abnormal changes;

[0019] (5), At each moment, update the anomaly score accumulation value:

[0020] Sum t+1 = Sum t + S t+1

[0021] where Sum t+1 is the anomaly score accumulation for the next moment, and Sum t is the anomaly score accumulation for the current moment,

[0022] S t+1 is the anomaly score for the current moment; the accumulation mechanism helps to capture the overall abnormal state and reduce misjudgment caused by sudden fluctuations at a single moment;

[0023] (6), Calculate the average value within the current window, that is, the central position of the anomaly scores within the current window:

[0024] μ t+1 = Sum t+1 / (t + 1)

[0025] where μ t+1 is the average value within the window for the next moment; it reflects the overall data level within the window and helps to distinguish normal fluctuations from abnormal mutations;

[0026] (7), Calculate the variance within the current window, that is, the degree of fluctuation of the data within the window:

[0027]

[0028] where Var t is the variance of the anomaly scores within the current window; the variance can capture the degree of data dispersion, reflect the fluctuation intensity within the window, and provide fault tolerance for detecting mutations;

[0029] (8), Update the adaptive threshold, which combines the long-term trend and short-term fluctuations of the data to ensure that the threshold can adaptively reflect the real-time changes of the data. Through weighted smoothing, the interference of noise is avoided, and at the same time, the variance term increases the fault tolerance for mutations:

[0030] Thr t+1 = α · Thr t + (1 - α) · (Ewma t+1 + 2 * Var t+1 )

[0031] where Thr t+1 is the threshold updated at the next moment;

[0032] (9), Determine the loop condition, i.e., whether t is less than w. If so, continue the iteration. If the loop ends, output the finally calculated threshold. This loop mechanism ensures that the system can continuously adapt to data changes, achieve real-time dynamic update, and thus improve the accuracy of overall anomaly detection.

[0033] Advantages of the present invention:

[0034] Based on the anomaly score distribution within the local window of the time series, the present invention dynamically estimates the upper bound of anomalies, realizes the adaptive adjustment of the detection threshold, improves the accuracy of anomaly detection, reduces the false alarm rate and the missed detection rate, and demonstrates good robustness and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is the execution flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0037] As Figure 1 shown, a method for adaptively adjusting the anomaly detection threshold of a time series includes the following steps:

[0038] (1), Use the Fast Fourier Transform (FFT) to obtain the main frequency from the original data and extract the main features of the time series data. This step helps to understand the periodicity and fluctuation characteristics of the data. By ignoring the DC component, the FFT calculation result will extract the frequency domain features of the data. FFT converts the time domain signal into the frequency domain signal, allowing us to identify the main frequency of the data. Usually, the main frequency corresponds to the periodic change of the data. From the FFT result, select the frequency with the largest amplitude as the main frequency f top . This main frequency f top The reciprocal of will be used as the period length w, that is, the size of the sliding window p, so that the adaptive threshold can flexibly cope with the periodic changes of the data:

[0039] f top = Amp max (FFT(x))

[0040] w = 1 / f top

[0041] where x is the input data, and its size is (batch_size, win_size, dim), that is, the shape of the input data; Ampmax It is to take the amplitude of the complex spectrum obtained after FFT;

[0042] (2), Set the initial threshold, EWMA value and the initial value of the anomaly score to ensure a reasonable starting point at the beginning and prepare for subsequent calculations:

[0043] Ewma 0 =Thr 0 =Sum 0 =S 0

[0044] where Ewma 0 is the exponentially weighted moving average value at time 0, Thr 0 is the threshold at time 0, Sum 0 is the cumulative anomaly score at time 0, and S 0 is the anomaly score at time 0;

[0045] (3), Dynamically set the length p of the sliding window based on the main frequency calculated by FFT. The size of the sliding window will be automatically adjusted according to the periodic change of the data, which can ensure that the statistical characteristics within each window are roughly consistent, thereby improving the accuracy of anomaly detection;

[0046] (4), At each moment t, use the exponentially weighted moving average (EWMA) algorithm to update the EWMA value at the current moment, and use the exponentially weighted moving average method to smooth the input value in order to better capture the long-term trend in the data; Given the anomaly score S t at the current time step, its corresponding exponentially weighted moving average value EWMA t can be updated through the following recursive relationship:

[0047] Ewma t+1 =α·Ewma t +(1 - α)·S t

[0048] where Ewma t+1 is the exponentially weighted moving average value at the next moment, α is the EWMA decay factor and satisfies α∈(0,1) to control the degree of context relevance, taking 0.9, which means that the value at the current moment will depend more heavily on the result of the previous moment and less on the current anomaly score. This weighted smoothing method helps to reduce the random noise in the data and retains the stationary characteristics of the anomaly signal, making the threshold calculation more stable;

[0049] (5), At each moment, update the cumulative value of the anomaly score:

[0050] Sum t+1 =Sumt +S t+1

[0051] Where Sum t+1 is the accumulation of abnormal scores at the next moment, Sum t is the anomaly score accumulation at the current moment. Through this accumulation method, we can effectively capture the abnormal pattern within the window, and the anomaly score will gradually accumulate over time. t+1 is the anomaly score at the current moment; the accumulation mechanism helps capture the overall abnormal state and reduce misjudgments caused by sudden fluctuations at a single moment;

[0052] (6) Calculate the average value in the current window, that is, the center position of the anomaly score in the current window:

[0053] μ t+1 =Sum t+1 / (t+1)

[0054] where μ t+1 It is the average value in the window at the next moment, which can reflect the overall trend of the anomaly score in the window and provide a benchmark value for subsequent volatility calculations;

[0055] (7) Calculate the variance within the current window, that is, the degree of fluctuation of the data within the window:

[0056]

[0057] Var t is the variance of the anomaly score in the current window. The larger the variance, the more drastic the fluctuation of the data in the current window, which plays an important role in capturing the short-term fluctuation characteristics of the data.

[0058] (8) Update the adaptive threshold, which combines the long-term trend and short-term fluctuation of the data to ensure that the threshold can adaptively reflect the real-time changes of the data. Through weighted smoothing, noise interference is avoided, and the variance term increases the fault tolerance to mutations:

[0059] Thr t+1 =α·Thr t +(1-α)·(Ewma t+1 +2*Var t+1 )

[0060] Among them, Thr t+1 The threshold value updated at the next moment. When the data fluctuates greatly, the threshold value will be increased accordingly to improve the sensitivity of anomaly detection; when the data fluctuates less, the threshold value will be decreased accordingly to reduce the false alarm rate;

[0061] (9) Judge the loop condition, that is, whether t is less than w. If so, continue the iteration. If the loop ends, output the finally calculated threshold. This loop mechanism ensures that the system can continuously adapt to data changes, achieve real-time dynamic updates, and thus improve the accuracy of overall anomaly detection.

[0062] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for adaptively adjusting a time series anomaly detection threshold, characterized in that: The steps include: (1) Use fast Fourier transform to obtain the main frequency from the original data and extract the main features of the time series data. By ignoring the DC component, the fast Fourier transform calculation result will extract the frequency domain characteristics of the data. From the fast Fourier transform result, select the frequency with the maximum amplitude as the main frequency f top ; Main frequency f top The reciprocal of will be used as the cycle length w, that is, the size of the sliding window p, so that the adaptive threshold can flexibly respond to the periodic changes of the data: f top =Amp max (FFT(x)) w=1 / f top Where x is the input data, and its size is (batch_size, win_size, dim), which is the shape of the input data; Amp max It is to take the amplitude of the complex spectrum obtained after FFT; (2) Set the initial threshold, EWMA value, and initial value of the anomaly score to ensure a reasonable starting point at the beginning and prepare for subsequent calculations: Ewma0=Thr0=Sum0=S0 Where Ewma0 is the exponentially weighted moving average at time 0, Thr0 is the threshold at time 0, Sum0 is the accumulated anomaly score at time 0, and S0 is the anomaly score at time 0; (3) Dynamically set the length p of the sliding window based on the main frequency calculated by fast Fourier transform; the size of the sliding window will be automatically adjusted according to the periodic changes of the data to ensure that the statistical characteristics within each window remain roughly consistent, thereby improving the accuracy of anomaly detection; (4) At each time t, the EWMA algorithm is used to update the EWMA value of the current time, and the exponentially weighted moving average method is used to smooth the input value in order to better capture the long-term trend in the data; given the anomaly score S of the current time step t , its corresponding exponentially weighted moving average EWMA t It can be updated through the following recursive relationship: Ewma t+1 =a·Ewma t +(1-α)·S t Among them Ewma t+1 is the exponential weighted moving average of the next moment, α is the EWMA attenuation factor and satisfies α∈(0,1) to control the relevance of the context, and is taken as 0.9; (5) At each moment, update the accumulated value of the anomaly score: Sum t+1 =Sum t +S t+1 Where Sum t+1 The accumulation of abnormal scores at the next moment, Sum t is the anomaly score accumulation at the current moment, S t+1 is the anomaly score at the current moment; (6) Calculate the average value in the current window, that is, the center position of the anomaly score in the current window: μ t+1 =I am t+1 / (t+1) where μ t+1 is the average value in the window at the next moment; (7) Calculate the variance within the current window, that is, the degree of fluctuation of the data within the window: Var t is the variance of the anomaly score in the current window; (8) Update the adaptive threshold, which integrates the long-term trend and short-term fluctuation of the data, ensuring that the threshold can adaptively reflect the real-time changes of the data. Through weighted smoothing, noise interference is avoided. At the same time, the variance term increases the fault tolerance to mutations: Thr t+1 =α·Thr t +(1-α)·(Ewma t+1 +2*Var t+1 ) Among them, Thr t+1 The threshold value updated for the next moment; (9) Determine whether the loop condition, i.e., t is less than w. If so, continue iterating. If the loop ends, output the final threshold calculated.

Images