A time series anomaly detection method and device

Abstract

The embodiment of the present application relates to a kind of time series anomaly detection method and device, the method comprises: obtaining the time series data set to be processed, each time series data in the time series data set at least includes time point and corresponding numerical value;The data in the time series data set is preprocessed, and the preprocessing includes data resampling, missing value processing, data labeling, data formatting and anomaly injection;Based on the time series data set after preprocessing, an anomaly detection model is constructed;Anomaly detection is carried out using the anomaly detection model.The technical scheme provided in the embodiment of the present application, by the reasonable data preprocessing link to time series data set, improve the data quality;By providing different sequence anomaly detection model, and more flexibly link it when monitoring in real time on line, while ensuring accuracy also makes its execution efficiency improves.

Description

Technical Field

[0001] This invention relates to the field of data detection technology, and in particular to a method and apparatus for detecting time series anomalies. Background Technology

[0002] A time series is a sequence of numbers that arranges the values ​​of the same statistical indicator in chronological order of their occurrence. Time series anomaly detection aims to discover outliers in a time series, such as sudden increases or decreases, in order to ensure the stability of the time series and detect anomalies in a timely manner.

[0003] Traditional time series anomaly detection typically relies on threshold settings. However, since most time series exhibit strong periodicity, different thresholds are required for different time periods, often resulting in low accuracy and poor model robustness. With the rapid development of machine learning, machine learning-based time series anomaly detection methods have emerged. While these methods outperform traditional methods, training the model requires a large amount of labeled data, and prediction is time-consuming. Therefore, there is an urgent need for a time series anomaly detection method that offers high accuracy, high efficiency, and high robustness. Summary of the Invention

[0004] Based on the above-mentioned situation of the prior art, the purpose of this invention is to provide a time series anomaly detection method and apparatus to solve the problems of low accuracy, low efficiency, poor robustness and low flexibility in the existing time series anomaly detection process.

[0005] To achieve the above objectives, according to one aspect of the present invention, a time series anomaly detection method is provided, the method comprising:

[0006] S102. Obtain a time series data set to be processed, wherein each time series data in the time series data set includes at least a time point and its corresponding value;

[0007] S104. Preprocess the data in the time series dataset, including data resampling, missing value handling, data labeling, data formatting, and anomaly injection.

[0008] S106. Construct an anomaly detection model based on the preprocessed time series data set;

[0009] S108. Anomaly detection is performed using the aforementioned anomaly detection model.

[0010] Furthermore, the data resampling includes:

[0011] Based on a preset detection frequency, the time series data set is resampled, and the resampled value is the sum of its original value within the detection frequency range.

[0012] Furthermore, the missing value processing includes filling in the missing values ​​with values ​​calculated using the following formula:

[0013]

[0014] Where i, i∈[1,N′] is the index of a certain time series data in the time series data set, N′ is the length of the time series data set, and V i Let i ∈ [1, N′] be the numerical value of the time series data corresponding to index i, n, n ∈ [1, N′] be the index of the missing value in the time series data set, and r, r ∈ [0, +∞] be the user-defined neighborhood range. The function... To count the number of non-zero values ​​in list x.

[0015] Furthermore, the data annotation includes manual annotation, and each time series data after annotation includes at least a time point, its corresponding value, and a label.

[0016] Furthermore, the data formatting includes:

[0017] S10441. Generate the current sequence cs corresponding to index n. n :

[0018] cs n =[V n_w V n_w+1 , ..., V n ];

[0019] S10442. Generate the first reference sequence fs corresponding to index n. n :

[0020]

[0021] S10443. Generate the second reference sequence ss corresponding to index n. n :

[0022]

[0023] S10444, Let the current sequence be cs n First reference sequence fs n Second reference sequence ss n The input data is the index n of the point to be detected, and the value to be detected is V. n The label to be detected is B. nThe above formatting steps are performed sequentially on all subscripts n that meet the conditions to generate data, thus generating a second dataset. Each data item in the second dataset includes the current sequence, the first reference sequence, the second reference sequence, the value to be detected, and the label to be detected.

[0024] Where N′ is the length of the time series data set E′, i, i∈[1,N′] is the index of a certain time series data in the time series data set E′, and T i , i∈[1,N′] represents the time point of the time series data corresponding to index i, V i , i∈[1,N′] is the numerical value of the time series data corresponding to index i, B i , i∈[1, N′] is the label of the time series data corresponding to index i, d, d∈[1, +∞] is the number of time points per day in the time series data set E′, w, w∈[1, +∞] is the size of the time window, f1, f1∈[1, 3] is the first reference number of days, f2, f2∈[2, 7] is the second reference number of days, and f2>f1 must be satisfied, n, n∈[f2d+w, N′] is the index of the data to be transformed in the time series data set E′.

[0025] Furthermore, the anomaly injection includes a 3Sigma-based anomaly injection method:

[0026] S10451. Randomly generate neighborhood factor α within the range [1, w];

[0027] S10452. Randomly generate an influence factor β in the range [1, w], satisfying β < α;

[0028] S10453. Randomly generate the direction factor χ in the range [0, 1].

[0029] S10454. Randomly generate the expansion factor ρ within the range of [4, 10].

[0030] S10455. Calculate the current sequence cs corresponding to index n. n The mean μ and variance σ are within the range [w_α, w].

[0031] S10456, For the current sequence cs corresponding to index n n Elements within the scope of influence Reassign the value using the following formula:

[0032]

[0033] S10457, For the value V to be detected corresponding to the subscript n n Reassign the value to the current sequence cs n The last element, corresponding to the label B to be detected at index n.n Reassigning a value results in an exception.

[0034] Furthermore, the anomaly injection includes a multiplier-based anomaly injection method:

[0035] S104511. Randomly generate the influence factor β within the range [1, w].

[0036] S104512. Randomly generate the direction factor χ in the range [0, 1].

[0037] S104513. Randomly generate the expansion factor ρ within the range of [0.3, 3].

[0038] S104514, For the current sequence cs corresponding to index n n Elements within the scope of influence Reassign values ​​using the following formula:

[0039]

[0040] S104515, For the value V to be detected corresponding to the subscript n. n Reassign the value to the current sequence cs n The last element, corresponding to the label B to be detected at index n. n Reassigning a value results in an exception.

[0041] Furthermore, the construction of the anomaly detection model includes:

[0042] S10611. Let g1 be the threshold corresponding to the numerical range r1 = [0, a], g2 be the threshold corresponding to the numerical range r2 = [a, b], and g3 be the threshold corresponding to the numerical range r3 = [b, +∞]. Let a and b satisfy b > a > 0, and let g1, g2, and g3 satisfy g1 > g2 > g3 > 0.3.

[0043] S10612. Let x be a sequence consisting of the sequences [0, 1, ..., w], [0, 1, ..., 2w+1], and [0, 1, ..., 2w+1].

[0044] S10613. Let y be a new sequence formed by connecting the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected.

[0045] S10614, Fitting a polynomial regression function to x and y Polynomial regression function The power of is a preset value;

[0046] S10615. Calculate the difference threshold g according to the following formula:

[0047]

[0048] S10616. Select the corresponding threshold g1, g2, or g3 based on the range r1, r2, or r3 of the value V to be detected. If the difference threshold g ≤ g i If i∈[1,3], it is considered normal; otherwise, it is considered abnormal.

[0049] Where cs is the current sequence of the data to be detected, fs is the first reference sequence of the data to be detected, ss is the second reference sequence of the data to be detected, V is the value to be detected, and w is the size of the time window.

[0050] Furthermore, the construction of the anomaly detection model includes:

[0051] S10621. Define the threshold corresponding to the numerical range r1 = [0, a] as g1, the threshold corresponding to the numerical range r2 = [a, b] as g2, and the threshold corresponding to the numerical range r3 = [b, +∞] as g3, wherein a and b satisfy b > a > 0, and g1, g2, and g3 satisfy g1 > g2 > g3 > 3;

[0052] S10622. Calculate the mean μ and variance σ of the new sequence formed by connecting the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected.

[0053] S10623. Calculate the difference threshold g according to the following formula.

[0054]

[0055] S10624. Select the corresponding threshold g1, g2, or g3 based on the range r1, r2, or r3 of the value V to be detected. If the difference threshold g ≤ g i If i∈[1,3], it is considered normal; otherwise, it is considered abnormal.

[0056] Furthermore, the construction of the anomaly detection model includes:

[0057] S10631. Define a feature extraction table F, wherein the attributes and number of features in the feature extraction table F are preset values;

[0058] S10632. Traverse all data in the preprocessed time series dataset, and extract data from the current sequence cs based on the feature extraction table F. i First reference sequence fs i Second reference sequence ss i Extracting features f i ;

[0059] S10633, by feature f i and the label to be detected Bi Train a binary classification model M based on XGBoost;

[0060] S10634. For the data to be detected, extract features f from the current sequence cs, the first reference sequence fs, and the second reference sequence ss of the data to be detected according to the feature extraction table F, and input them into the binary classification model M for prediction, and output normal or abnormal results.

[0061] Furthermore, anomaly detection using the aforementioned anomaly detection model includes online real-time detection:

[0062] S10811. Extract the time series data to be detected according to a predetermined time or frequency, and generate the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected;

[0063] S10812. Input the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected into the first model to predict its first label, and output a normal or abnormal result.

[0064] S10813. Input the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected into the second model M2 to predict the second label and output the normal or abnormal result.

[0065] S10814. If both the predicted first label and the second label are normal, the final label is output as normal. If either the predicted first label or the second label is abnormal, the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected are input into the third model to predict its third label, and the normal or abnormal result is output as the final label.

[0066] The first model, the second model, and the third model are the models constructed above.

[0067] Furthermore, anomaly detection using the aforementioned anomaly detection model also includes anomaly feedback:

[0068] If the final label output is abnormal, the feedback data will be manually judged; if it is judged to be normal, its true label will be recorded, and the data and the true label will be recorded together in the database for model optimization.

[0069] According to another aspect of the present invention, a time series anomaly detection device is provided, comprising:

[0070] The data acquisition module is used to acquire a set of time series data to be processed, wherein each time series data in the set of time series data includes at least a time point and its corresponding value;

[0071] The data preprocessing module is used to preprocess the data in the time series dataset. The preprocessing includes data resampling, missing value handling, data labeling, data formatting, and anomaly injection.

[0072] The anomaly detection model building module is used to build an anomaly detection model based on a preprocessed time series data set.

[0073] An anomaly detection module is used to perform anomaly detection using the anomaly detection model.

[0074] In summary, this invention provides a method and apparatus for time series anomaly detection. The method includes: acquiring a time series data set to be processed, wherein each time series data in the time series data set includes at least a time point and its corresponding value; preprocessing the data in the time series data set, the preprocessing including data resampling, missing value handling, data annotation, data formatting, and anomaly injection; constructing an anomaly detection model based on the preprocessed time series data set; and performing anomaly detection using the anomaly detection model. The technical solution provided by this invention improves data quality by performing reasonable data preprocessing on the time series data set; proposes anomaly injection methods based on 3Sigma and multiples, making the anomaly injection method more realistic and better balancing the data; proposes time series anomaly detection models based on multinomial regression and dynamic thresholds, time series anomaly detection models based on 3Sigma and dynamic thresholds, and time series anomaly detection models based on XGBoost, and more flexibly linking them during online real-time monitoring, ensuring accuracy while improving execution efficiency. Attached Figure Description

[0075] Figure 1 This is a flowchart of the time series anomaly detection method provided in the embodiments of the present invention;

[0076] Figure 2 This is a visual schematic diagram of each sequence in the data formatting step of an embodiment of the present invention;

[0077] Figure 3 This is a block diagram of the time series anomaly detection device provided in the embodiments of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0079] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0080] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart of a time series anomaly detection method provided in an embodiment of the present invention, as follows: Figure 1 As shown, the method includes the following steps:

[0081] S102. Obtain the time series data set to be processed. Each time series data in the time series data set includes at least a time point and its corresponding value. In this step, the time series data set can be obtained from the target business scenario data according to actual needs. For example, if the time series data set to be processed is obtained, E, each time series data in the time series data set E should include at least a time point and its corresponding value. The time point may include information such as year, month, day, hour, minute, and second, and can be changed according to the actual business scenario.

[0082] S104. Preprocess the data in the time series dataset. The preprocessing includes data resampling, missing value handling, data labeling, data formatting, and anomaly injection, which will be described below.

[0083] S1041. Data Resampling. Data resampling involves resampling the time series data set E according to a preset detection frequency. The resampled value is the sum of the original values ​​within the detection frequency range. The preset detection frequency can be set according to the actual business scenario, such as 1 minute, 5 minutes, 10 minutes, etc. The resampled time series data set can be denoted as E′.

[0084] S1042. Missing Value Handling. In the time series dataset E′, if the value corresponding to a certain time point is 0, then this value is defined as a missing value. In this embodiment of the invention, the neighborhood mean imputation method is used to imput missing values. The missing values ​​can be imputed using the following formula:

[0085]

[0086] Where i, i∈[1,N′] is the index of a certain time series data in the time series data set, N′ is the length of the time series data set, and V i Let i ∈ [1, N′] be the numerical value of the time series data corresponding to index i, n, n ∈ [1, N′] be the index of the missing value in the time series data set, and r, r ∈ [0, +∞] be the user-defined neighborhood range. The function... For the calculation list The number of non-zero values. Additionally, in the calculation of the above formula, if the value V is reduced due to the subscript i going out of bounds... i If it does not exist, record the value V. i It is 0.

[0087] S1043. Data Labeling. Visualize the time series data set E′ and label each time series data point manually, either as normal or abnormal. Each labeled time series data point should include at least the time point, its corresponding value, and the label.

[0088] S1044. Data Formatting. Iterate through the time series dataset E′, converting each time series data point to a specified format; let N′ be the length of the time series dataset E′, i, i∈[1, N′] be the index of a certain time series data point in the time series dataset E′, and T... i , i∈[1,N′] represents the time point of the time series data corresponding to index i, V i , i∈[1,N′] is the numerical value of the time series data corresponding to index i, B i , i∈[1, N′] is the label of the time series data corresponding to index i, d, d∈[1, +∞] is the number of time points per day in the time series data set E′, w, w∈[1, +∞] is the size of the time window, f1, f1∈[1, 3] is the first reference number of days, f2, f2∈[2, 7] is the second reference number of days, and f2>f1 must be satisfied, n, n∈[f2d+w, N′] is the index of the data to be transformed (the point to be detected) in the time series data set E′. Figure 2 The diagram shows a visual representation of each sequence in the data formatting step.

[0089] Data formatting specifically includes the following steps:

[0090] S10441. Generate the current sequence cs corresponding to index n. n :

[0091] cs n =[V n_w V n_w+1 , ..., V n ];

[0092] S10442. Generate the first reference sequence fs corresponding to index n. n :

[0093]

[0094] S10443. Generate the second reference sequence ss corresponding to index n. n :

[0095]

[0096] S10444, Let the current sequence be cs n First reference sequence fs n Second reference sequence ss n The input data is the index n of the point to be detected, and the value to be detected is V. n The label to be detected is B. n The above formatting steps are performed sequentially on all subscripts n that meet the conditions to generate data, thus generating the second dataset, which is the time series data set E″. Each data in the time series data set E″ includes the current sequence, the first reference sequence, the second reference sequence, the value to be detected, and the label to be detected.

[0097] S1045. Anomaly Injection. Because abnormal data is often scarce in actual business scenarios, it can lead to a severe data imbalance between normal and abnormal data. Therefore, it is necessary to generate abnormal data to achieve data balance. A portion of the data in the time series dataset E″ whose labels are normal is randomly selected for anomaly injection. The anomaly injection can be randomly selected from 3Sigma-based anomaly injection methods and multiple-based anomaly injection methods. Let N″ be the length of the time series dataset E″, i, i∈[1, N″] be the index of a certain data point in the time series dataset E″, and cs... i Let i∈[1, N″] be the current sequence of data corresponding to index i, and fs i , i∈[1,N″] is the first reference sequence of the data corresponding to index i, ss i , i∈[1,N″] is the second reference sequence of the data corresponding to index i, V i , i∈[1,N″] is the value to be detected for the data corresponding to index i, B i, i∈[1,N″] is the label to be detected for the data corresponding to index i, w, w∈[1,+∞] is the size of the time window, n, n∈[1,N″] is the index of the point to be injected into the time series data set E″.

[0098] The 3Sigma-based anomaly injection method includes the following steps:

[0099] S10451. Randomly generate neighborhood factor α within the range [1, w];

[0100] S10452. Randomly generate an influence factor β in the range [1, w], satisfying β < α;

[0101] S10453. Randomly generate the direction factor χ in the range [0, 1].

[0102] S10454. Randomly generate the expansion factor ρ within the range of [4, 10].

[0103] S10455. Calculate the current sequence cs corresponding to index n. n The mean μ and variance σ are within the range [w_α, w].

[0104] S10456, For the current sequence cs corresponding to index n n Elements within the scope of influence Reassign the value using the following formula:

[0105]

[0106] S10457, For the value V to be detected corresponding to the subscript n n Reassign the value to the current sequence cs n The last element, corresponding to the label B to be detected at index n. n Reassigning a value results in an exception.

[0107] The multiple-based anomaly injection method includes the following steps:

[0108] S104511. Randomly generate the influence factor β within the range [1, w].

[0109] S104512. Randomly generate the direction factor χ in the range [0, 1].

[0110] S104513. Randomly generate the expansion factor ρ within the range of [0.3, 3].

[0111] S104514, For the current sequence cs corresponding to index n n Elements within the scope of influence Reassign values ​​using the following formula:

[0112]

[0113] S104515, For the value V to be detected corresponding to the subscript n. n Reassign the value to the current sequence cs n The last element, corresponding to the label B to be detected at index n. n Reassigning a value results in an exception.

[0114] Anomaly injection based on 3Sigma or multiples can be performed on a random subset of data with normal labels in the time series dataset E″. The generated abnormal data has the same format as the formatted data, and the amount of generated data is the difference between the number of normal and abnormal labels in the time series dataset E″. After merging with the time series dataset E″, a new time series dataset E″′ is formed.

[0115] S106. Construct an anomaly detection model based on the preprocessed time-series data set. The model construction process involves training the model on the time-series data set E″′. The detection models are categorized into three types: a time-series anomaly detection model based on multinomial regression and dynamic thresholds, a time-series anomaly detection model based on 3Sigma and dynamic thresholds, and a time-series anomaly detection model based on XGBoost. The construction of the anomaly detection model includes the following steps:

[0116] S1061. Construction of a time series anomaly detection model based on multinomial regression and dynamic threshold; Let cs be the current sequence of the data to be detected, fs be the first reference sequence of the data to be detected, ss be the second reference sequence of the data to be detected, V be the value to be detected, and w be the size of the time window.

[0117] S10611. Let g1 be the threshold corresponding to the numerical range r1 = [0, a], g2 be the threshold corresponding to the numerical range r2 = [a, b], and g3 be the threshold corresponding to the numerical range r3 = [b, +∞]. Let a and b satisfy b > a > 0, and let g1, g2, and g3 satisfy g1 > g2 > g3 > 0.3.

[0118] S10612. Let x be a sequence consisting of the sequences [0, 1, ..., w], [0, 1, ..., 2w+1], and [0, 1, ..., 2w+1].

[0119] S10613. Let y be a new sequence formed by connecting the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected.

[0120] S10614, Fitting a polynomial regression function to x and y Polynomial regression function The power of is a preset value;

[0121] S10615. Calculate the difference threshold g according to the following formula:

[0122]

[0123] S10616. Select the corresponding threshold g1, g2, or g3 based on the range r1, r2, or r3 of the value V to be detected. If the difference threshold g ≤ g i If i∈[1,3], it is considered normal; otherwise, it is considered abnormal.

[0124] S1062. Construction of a time series anomaly detection model based on 3Sigma and dynamic threshold; Let cs be the current sequence of the data to be detected, fs be the first reference sequence of the data to be detected, ss be the second reference sequence of the data to be detected, V be the value to be detected, and w be the size of the time window.

[0125] S10621. Define the threshold corresponding to the numerical range r1 = [0, a] as g1, the threshold corresponding to the numerical range r2 = [a, b] as g2, and the threshold corresponding to the numerical range r3 = [b, +∞] as g3, wherein a and b satisfy b > a > 0, and g1, g2, and g3 satisfy g1 > g2 > g3 > 3;

[0126] S10622. Calculate the mean μ and variance σ of the new sequence formed by connecting the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected.

[0127] S10623. Calculate the difference threshold g according to the following formula.

[0128]

[0129] S10624. Select the corresponding threshold g1, g2, or g3 based on the range r1, r2, or r3 of the value V to be detected. If the difference threshold g ≤ g i If i∈[1,3], it is considered normal; otherwise, it is considered abnormal.

[0130] S1063. Time series anomaly detection model based on XGBoost; Let N″′ be the length of the time series data set E″′, i, i∈[1,N″′] be the index of a certain data in the time series data set E″′, and cs i Let i∈[1, N″′] be the current sequence of data corresponding to index i, and fs i , i∈[1,N″′] is the first reference sequence of the data corresponding to index i, ss i , i∈[1,N″′] is the second reference sequence of the data corresponding to index i, B i, i∈[1,N″′] is the label to be detected for the data corresponding to index i, cs is the current sequence of the data to be detected, fs is the first reference sequence of the data to be detected, and ss is the second reference sequence of the data to be detected.

[0131] S10631. Define a feature extraction table F, wherein the attributes and number of features in the feature extraction table F are preset values;

[0132] S10632. Traverse all data in the preprocessed time series dataset, and extract data from the current sequence cs based on the feature extraction table F. i First reference sequence fs i Second reference sequence ss i Extracting features f i ;

[0133] S10633, by feature f i and the label to be detected B i Train a binary classification model M based on XGBoost;

[0134] S10634. For the data to be detected, extract features f from the current sequence cs, the first reference sequence fs, and the second reference sequence ss of the data to be detected according to the feature extraction table F, and input them into the binary classification model M for prediction, and output normal or abnormal results.

[0135] S108. Anomaly detection is performed using the aforementioned anomaly detection model. The anomaly detection process can be deployed online, including real-time online monitoring and anomaly feedback, which will be explained below.

[0136] S1081. Online Real-Time Monitoring. Let the first model M1 be a time series anomaly detection model based on multinomial regression and dynamic threshold, the second model M2 be a time series anomaly detection model based on 3Sigma and dynamic threshold, and the third model M3 be a time series anomaly detection model based on XGBoost. Let cs be the current sequence of the data to be detected, fs be the first reference sequence of the data to be detected, and ss be the second reference sequence of the data to be detected.

[0137] S10811. Extract the time series data to be detected according to a predetermined time or frequency, and generate the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected;

[0138] S10812. Input the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected into the first model to predict its first label, and output a normal or abnormal result.

[0139] S10813. Input the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected into the second model M2 to predict the second label and output the normal or abnormal result.

[0140] S10814. If both the predicted first and second labels are normal, the final label is output as normal. If either the predicted first or second label is abnormal, the current sequence cs of the data to be detected, the first reference sequence fs of the data to be detected, and the second reference sequence ss of the data to be detected are input into the third model to predict its third label, and the normal or abnormal result is output as the final label.

[0141] S1082. Anomaly Feedback. If the final label output is abnormal, the feedback data will be manually judged; if it is judged to be normal, its true label will be recorded, and the data and the true label will be recorded together in the database for model optimization.

[0142] Embodiments of the present invention also provide a time series anomaly detection device. Figure 3 The diagram shows the configuration of the detection device 300, which includes:

[0143] Data acquisition module 301 is used to acquire a time series data set to be processed, wherein each time series data in the time series data set includes at least a time point and its corresponding value;

[0144] Data preprocessing module 302 is used to preprocess the data in the time series data set. The preprocessing includes data resampling, missing value handling, data labeling, data formatting, and anomaly injection.

[0145] Anomaly detection model building module 303 is used to build anomaly detection models based on preprocessed time series data sets;

[0146] Anomaly detection module 304 is used to perform anomaly detection using the anomaly detection model.

[0147] The specific process by which each module in the time series anomaly detection device provided in the above embodiments of the present invention implements its function is the same as the steps of the time series anomaly detection method provided in the above embodiments of the present invention. Therefore, repeated descriptions will be omitted here.

[0148] In summary, this invention relates to a time series anomaly detection method and apparatus. The method includes: acquiring a time series data set to be processed, wherein each time series data in the time series data set includes at least a time point and its corresponding value; preprocessing the data in the time series data set, the preprocessing including data resampling, missing value handling, data annotation, data formatting, and anomaly injection; constructing an anomaly detection model based on the preprocessed time series data set; and performing anomaly detection using the anomaly detection model. The technical solution provided by this invention improves data quality by performing reasonable data preprocessing on the time series data set; proposes anomaly injection methods based on 3Sigma and multiples, making the anomaly injection method more realistic and better balancing the data; proposes time series anomaly detection models based on multinomial regression and dynamic thresholds, time series anomaly detection models based on 3Sigma and dynamic thresholds, and time series anomaly detection models based on XGBoost, and more flexibly linking them during online real-time monitoring, ensuring accuracy while improving execution efficiency.

[0149] It should be understood that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of this invention, technical features of the above embodiments or different embodiments can also be combined, steps can be implemented in any order, and many other variations exist regarding different aspects of one or more embodiments of the invention as described above, which are not provided in the details for the sake of brevity. The specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A time series anomaly detection method, characterized by, The method comprises: S102, acquiring a time series data set to be processed, each time series data in the time series data set comprising at least a time point and a corresponding numerical value; S104, pre-processing data in the time series data set, the pre-processing comprising data resampling, missing value processing, data labeling, data formatting and anomaly injection; the anomaly injection comprising a 3Sigma-based anomaly injection method: S10451, randomly generating a neighborhood factor a in the range of [1, w]; S10452, randomly generating an influence factor β in the range of [1, w], satisfying β < a; S10453, randomly generating a direction factor χ in the range of [0, 1]; S10454, randomly generating an inflation factor ρ in the range of [4, 10]; S10455、compute the current sequence cs corresponding to the subscript n n mean μ and variance σ in the range [w - a, w] S10456, the current sequence cs corresponding to the subscript n n Elements within the sphere of influence Reassigning using the following formula: S10457, the to-be-detected value V corresponding to the subscript n n is re-assigned as the current sequence cs n , the last element of the current sequence cs, the to-be-detected label B corresponding to the subscript n n is re-assigned as an exception; wherein, N" is the length of the second data set, i, i∈[1, N"] is the index of a piece of data in the second data set, cs i , i∈[1, N"] is the current sequence of the data corresponding to index i, fs i , i∈[1, N"] is the first reference sequence of the data corresponding to index i, ss i , i∈[1, N"] is the second reference sequence of the data corresponding to index i, V i , i∈[1, N"] is the to-be-detected numerical value of the data corresponding to index i, B i , i∈[1, N"] is the to-be-detected label of the data corresponding to index i, w, w∈[1, +∞) is the size of a time window, n, n∈[1, N"] is the index of a to-be-anomaly-injected point in the second data set; or, The anomaly injection comprises a multiple-based anomaly injection method: S104511, randomly generating an influence factor β in the range of [1, w]; S104512, randomly generating a direction factor x in the range of [0, 1]; S104513, randomly generating an inflation factor ρ in the range of [0.3, 3]; S104514, the current sequence cs corresponding to the subscript n n Elements within the sphere of influence Reassigning using the following formula: S104515、the to-be-detected value V corresponding to the subscript n n is re-assigned as the current sequence cs n the last element of the current sequence cs, the to-be-detected label B corresponding to the subscript n n is re-assigned as an exception; wherein N" is the length of the second dataset, i, i∈[1, N"] is the index of a piece of data in the second dataset, cs i , i∈[1, N"] is the current sequence of the data corresponding to index i, fs i , i∈[1, N"] is the first reference sequence of the data corresponding to index i, ss i , i∈[1, N"] is the second reference sequence of the data corresponding to index i, V i , i∈[1, N"] is the value to be detected of the data corresponding to index i, B i , i∈[1, N"] is the label to be detected of the data corresponding to index i, w, w∈[1, +∞) is the size of the time window, n, n∈[1, N"] is the index of the abnormal injection point in the second dataset. S106, constructing an anomaly detection model based on the pre-processed time series data set; S108, performing anomaly detection by using the anomaly detection model.

2. The method of claim 1, wherein, The data resampling comprises: Resampling the time series data set according to a preset detection frequency, and the resampled numerical value being the sum of the original numerical values in the detection frequency range; The missing value processing comprises filling the missing values with the numerical value calculated by the following formula: wherein i, i ∈ [1, N'], is the index of a time series data in the time series data set, N' is the length of the time series data set, V i is the value of the time series data corresponding to the index i, n, n ∈ [1, N'], is the index of the missing value in the time series data set, r, r ∈ [0, +∞), is the custom neighborhood range, and the function is the number of non-zero values in the calculation list . The data labeling comprises manual labeling, and each time series data after labeling comprising at least a time point, a corresponding numerical value and a label.

3. The method according to claim 1 or 2, characterized in that, The data formatting comprises: S10441、generate the current sequence cs corresponding to the subscript n n : cs n = [V n-w , V n-w+1 ,..., V n ] ; S10442、generate the first reference sequence fs corresponding to the subscript n n : S10443、generate the second reference sequence ss corresponding to the subscript n n : S10444, the current sequence cs n , the first reference sequence fs n , the second reference sequence ss n , the input data of the to-be-detected point subscript n, the to-be-detected value is V n , the to-be-detected label is B n , the above formatting step is sequentially performed on all subscripts n satisfying the condition to generate data, a second data set is generated, and each piece of data in the second data set includes a current sequence, a first reference sequence, a second reference sequence, a to-be-detected value, and a to-be-detected label; wherein, N' is the length of the time series data set E', i, i∈[1, N'] is the index of a piece of time series data in the time series data set E', T i , i, i∈[1, N'] is the time point of the time series data corresponding to the index i, V i , i, i∈[1, N'] is the value of the time series data corresponding to the index i, B i , i, i∈[1, N'] is the label of the time series data corresponding to the index i, d, d∈[1, +∞) is the number of time points per day in the time series data set E', w, w∈[1, +∞) is the time window size, f1, f1∈[1, 3] is the first reference number of days, f2, f2∈[2, 7] is the second reference number of days, and it needs to satisfy f2>f1, n, n∈[f2d+w, N'] is the index of the data to be converted in the time series data set E'.

4. The method of claim 1, wherein, The construction of the anomaly detection model comprises: S10611, setting a threshold g1 corresponding to a numerical value range r1 = [0, a], a threshold g2 corresponding to a numerical value range r2 = [a, b], and a threshold g3 corresponding to a numerical value range r3 = [b, +∞), wherein a and b satisfy b > a > 0, and g1, g2 and g3 satisfy g1 > g2 > g3 > 0.3; S10612, setting x as a sequence composed of a sequence [0, 1,…, w], a sequence [0, 1,…, 2w+1] and a sequence [0, 1,…, 2w+2] connected in sequence; S10613, setting y as a new sequence composed of a current sequence cs of the data to be detected, a first reference sequence fs of the data to be detected and a second reference sequence ss of the data to be detected connected in sequence; S10614, fitting a polynomial regression function from x and y polynomial regression function the power is a preset value; S10615, calculating a difference threshold g according to the following formula: S10616, select the corresponding threshold g1, g2 or g3 according to the range r1, r2 or r3 where the to-be-detected value V is located, and determine that the difference threshold g is less than or equal to g i If g < g, i∈[1,3] is determined to be normal, otherwise it is determined to be abnormal; Wherein, cs is the current sequence of the data to be detected, fs is the first reference sequence of the data to be detected, ss is the second reference sequence of the data to be detected, V is the numerical value to be detected, and w is the time window size.

5. The method of claim 1, wherein, The construction of the anomaly detection model comprises: S10621, define the threshold value corresponding to the numerical range r1 = [0, a] as g1, the threshold value corresponding to the numerical range r2 = [a, b] as g2, and the threshold value corresponding to the numerical range r3 = [b, +∞) as g3, wherein a and b satisfy b > a > 0, and g1, g2 and g3 satisfy g1 > g2 > g3 > 3; S10622, calculate the mean μ and the variance σ of a new sequence composed of the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data; S10623, calculate the difference threshold g according to the following formula: S10624, select the corresponding threshold value g1, g2 or g3 according to the range r1, r2 or r3 where the to-be-detected value V is located, and determine that the difference threshold g≤g i is normal, otherwise, determine that it is abnormal.

6. The method of claim 1, wherein, The construction of the anomaly detection model comprises: S10631, define a feature extraction table F, wherein the attributes and the number of features in the feature extraction table F are preset values; S10632, iterate through all data in the pre-processed time series data set, extract features f from the current sequence cs according to the feature extraction table F i , the first reference sequence fs i , and the second reference sequence ss i ; extract features f i ; S10633、by feature f i and the label B to be detected i Train a binary classification model M based on XGBoost; S10634, for the to-be-detected data, extract features f from the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data according to the feature extraction table F, and input the features f into the binary classification model M for prediction, and output normal or abnormal results.

7. The method of claim 6, wherein, The anomaly detection using the anomaly detection model comprises online real-time detection: S10811, extract the time series data to be detected according to a predetermined time or frequency, and generate the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data; S10812, input the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data into the first model to predict a first label, and output normal or abnormal results; S10813, input the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data into the second model M2 to predict a second label, and output normal or abnormal results; S10814, if both the predicted first label and the second label are normal, output the final label as normal; if either the predicted first label or the second label is abnormal, input the current sequence cs of the to-be-detected data, the first reference sequence fs of the to-be-detected data and the second reference sequence ss of the to-be-detected data into the third model to predict a third label, output normal or abnormal results, and output the results as the final label; The first model is the model constructed in claim 4, the second model is the model constructed in claim 5, and the third model is the model constructed in claim 6.

8. The method of claim 7, wherein, The anomaly detection using the anomaly detection model further comprises anomaly feedback: If the final label output is abnormal, the feedback data is manually determined; if the manual determination is normal, the real label is recorded, and the data and the real label are recorded in the database together for model optimization.

9. A time series anomaly detection apparatus characterized by comprising: Comprise: A data acquisition module is configured to acquire a set of time series data to be processed, each piece of time series data in the set of time series data comprising at least a time point and a corresponding numerical value; A data preprocessing module is configured to preprocess data in the time series data set, and the preprocessing includes data resampling, missing value processing, data labeling, data formatting, and anomaly injection. The anomaly injection includes a 3Sigma-based anomaly injection method. S10451, randomly generating a neighborhood factor a in the range of [1, w]; S10452, randomly generating an influence factor b in the range of [1, w] and satisfying b < a; S10453, randomly generating a direction factor x in the range of [0, 1]; S10454, randomly generating an inflation factor p in the range of [4, 10]; S10455、compute the current sequence cs corresponding to the subscript n n mean μ and variance σ in the range [w - a, w] S10456, the current sequence cs corresponding to the subscript n n elements within the sphere of influence reassigned using the following equation: S10457, the to-be-detected value V corresponding to the subscript n n is re-assigned as the current sequence cs n , the to-be-detected label B corresponding to the subscript n n is re-assigned as an exception; wherein, N" is the length of the second data set, i, i∈[1, N"] is the index of a piece of data in the second data set, cs i , i∈[1, N"] is the current sequence of the data corresponding to the index i, fs i , i∈[1, N"] is the first reference sequence of the data corresponding to the index i, ss i , i∈[1, N"] is the second reference sequence of the data corresponding to the index i, V i , i∈[1, N"] is the to-be-detected numerical value of the data corresponding to the index i, B i , i∈[1, N"] is the to-be-detected label of the data corresponding to the index i, w, w∈[1, +∞) is the size of a time window, n, n∈[1, N"] is the index of the to-be-anomaly-injected point in the second data set; or, The anomaly injection includes a multiple-based anomaly injection method: S104511, randomly generating an influence factor b in the range of [1, w]; S104512, randomly generating a direction factor x in the range of [0, 1]; S104513, randomly generating an inflation factor p in the range of [0.3, 3]; S104514, the current sequence cs corresponding to the subscript n n elements within the sphere of influence reassigned using the following equation: S104515、the to-be-detected value V corresponding to the subscript n n is re-assigned as the current sequence cs n the last element of the current sequence cs, the to-be-detected label B corresponding to the subscript n n is re-assigned as an exception; wherein N" is the length of the second dataset, i, i∈[1, N"] is the index of a piece of data in the second dataset, cs i , i∈[1, N"] is the current sequence of the data corresponding to index i, fs i , i∈[1, N"] is the first reference sequence of the data corresponding to index i, ss i , i∈[1, N"] is the second reference sequence of the data corresponding to index i, V i , i∈[1, N"] is the value to be detected of the data corresponding to index i, B i , i∈[1, N"] is the label to be detected of the data corresponding to index i, w, w∈[1, +∞) is the size of the time window, n, n∈[1, N"] is the index of the abnormal injection point in the second dataset. An anomaly detection model construction module is configured to construct an anomaly detection model based on the preprocessed time series data set; An anomaly detection module is configured to perform anomaly detection using the anomaly detection model.

Images