Time series data anomaly detection method and device
By decomposing the time series data into periodic, trend and residual components, and using different models for detection, combined with voting mechanism or weighted statistics, the problem of missing outliers in the existing technology is solved, and the accuracy and sensitivity of abnormal detection are improved .
Patent Information
- Application Number
- CN202011216002.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-04
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2040-11-04
AI Technical Summary
The existing timing data abnormality detection technology is prone to miss abnormalities whose outliers are not significantly greater than periodic changes or trend changes.
The time series data to be detected is decomposed into periodic components, trend components and residual components, and abnormality detection is performed using different models, and the detection results are combined through voting mechanisms or weighted statistics to determine whether the data is abnormal.
Effectively avoid the situation where the abnormal characteristics of different components are different and the same detection algorithm cannot be effectively detected, reduce the abnormality detection error, and detect abnormalities whose outliers are not significantly greater than periodic changes or trend changes.
Smart Images
Figure CN114443331B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method and device for detecting anomalies in time series data. Background Art
[0002] In the operation process of cloud virtualization system, the system's ability to detect resource call and usage anomalies is very important. The sensitivity and accuracy of resource anomaly detection directly affect the system operation effect.
[0003] In some detection technologies, a certain anomaly detection algorithm is used to directly determine whether the original time series data to be detected is abnormal.
[0004] The inventors have found that the above detection technology has the problem of missing some anomalies. For example, anomalies whose values are not significantly larger than periodic changes or trend changes are easily missed. Summary of the invention
[0005] The disclosed embodiment decomposes the time series data to be detected into a periodic component, a trend component and a residual component, performs anomaly detection on each component independently, and then combines the detection results of each component to determine whether the time series data to be detected is abnormal. This can effectively avoid the situation where different components have different anomaly characteristics and the same detection algorithm cannot effectively detect data with different characteristics, and reduce anomaly detection errors. For example, it can detect anomalies whose anomalies are not significantly greater than periodic changes or trend changes.
[0006] Some embodiments of the present disclosure provide a method for detecting anomalies in time series data, including:
[0007] Obtain the time series data to be detected;
[0008] Decomposing the time series data to be detected into a period component, a trend component and a residual component;
[0009] Determine whether the period component, trend component and residual component are abnormal;
[0010] Whether the time series data to be detected is abnormal is determined based on whether the period component is abnormal, whether the trend component is abnormal, and whether the residual component is abnormal.
[0011] In some embodiments, decomposing the time series data to be detected into a period component, a trend component and a residual component comprises:
[0012] The seasonal trend decomposition algorithm based on local weighted regression is used to decompose the time series data to be detected into a period component, a trend component and a residual component.
[0013] In some embodiments, determining whether the period component, the trend component, and the residual component are abnormal respectively includes:
[0014] Using the first time series prediction model to determine whether the periodic component is abnormal;
[0015] Using the second time series prediction model to determine whether the trend component is abnormal;
[0016] Use the isolation forest model to determine whether the residual component is abnormal;
[0017] The first time series prediction model and the second time series prediction model are the same or different time series prediction models.
[0018] In some embodiments, determining whether the time series data to be detected is abnormal includes: determining whether the time series data to be detected is abnormal based on whether the periodic component is abnormal, whether the trend component is abnormal, and whether the residual component is abnormal, using a preset voting mechanism.
[0019] In some embodiments, the voting mechanism includes:
[0020] When any one component is abnormal, the time series data to be detected is abnormal;
[0021] Alternatively, when most of all components are abnormal, the time series data to be detected is abnormal;
[0022] Alternatively, when all component anomalies are abnormal, the time series data to be detected is abnormal;
[0023] Alternatively, weighted statistics are performed on the abnormal results of all components, and if the weighted statistical results meet the abnormality judgment condition, it is determined that the time series data to be detected is abnormal.
[0024] In some embodiments, the first time series prediction model is obtained by training using the periodic component of historical time series data; the second time series prediction model is obtained by training using the trend component of historical time series data; and the isolation forest model is obtained by training using the residual component of historical time series data.
[0025] In some embodiments, the time series forecasting model includes an autoregressive AR model, a moving average MA model, an autoregressive moving average ARMA model, or a difference autoregressive moving average ARIMA model.
[0026] In some embodiments, the timing data to be detected includes CPU resource timing data, memory resource timing data, storage resource timing data, and network resource timing data of the cloud virtualization system.
[0027] Some embodiments of the present disclosure provide a time series data anomaly detection device, including:
[0028] Memory; and
[0029] A processor coupled to the memory is configured to execute a time series data anomaly detection method based on instructions stored in the memory.
[0030] Some embodiments of the present disclosure provide a time series data anomaly detection device, including:
[0031] A module for acquiring data to be detected, configured to acquire time series data to be detected;
[0032] A time series data decomposition module is configured to decompose the time series data to be detected into a period component, a trend component and a residual component;
[0033] An abnormality detection module is configured to determine whether the period component, the trend component and the residual component are abnormal respectively; and determine whether the time series data to be detected is abnormal according to whether the period component is abnormal, whether the trend component is abnormal and whether the residual component is abnormal;
[0034] The output module is configured to output a detection result of whether the time series data to be detected is abnormal.
[0035] In some embodiments, the time series data anomaly detection device further includes: a training data storage module and a training module; wherein,
[0036] The training data storage module is configured to store historical time series data;
[0037] The time series data decomposition module is further configured to decompose the historical time series data into a period component, a trend component and a residual component;
[0038] The training module is configured to respectively train a first time series prediction model, a second time series prediction model and an isolation forest model using the periodic component, trend component and residual component of historical time series data, wherein the first time series prediction model and the second time series prediction model are the same or different time series prediction models.
[0039] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of a method for detecting anomalies in time series data when the program is executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.
[0041] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.
[0042] Figure 1 A flowchart of a method for detecting anomalies in time series data according to some embodiments of the present disclosure is shown.
[0043] Figure 2 A flowchart of a model training method according to some embodiments of the present disclosure is shown.
[0044] Figure 3 A schematic diagram showing a time series data anomaly detection device according to some embodiments of the present disclosure.
[0045] Figure 4 Schematic diagrams showing devices for detecting anomalies in time series data according to other embodiments of the present disclosure. DETAILED DESCRIPTION
[0046] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.
[0047] Unless otherwise specified, descriptions such as “first” and “second” in the present disclosure are used to distinguish different objects and are not used to indicate meanings such as size or time sequence.
[0048] Figure 1 A flowchart of a method for detecting anomalies in time series data according to some embodiments of the present disclosure is shown.
[0049] like Figure 1 As shown, the time series data anomaly detection method of this embodiment includes: steps 110-140.
[0050] In step 110, the time series data to be detected is obtained.
[0051] For example, the time series data to be detected is obtained from the cloud virtualization system.
[0052] The timing data to be detected includes, for example, CPU resource timing data, memory resource timing data, storage resource timing data, network resource timing data, etc. of the cloud virtualization system, but is not limited to the examples given.
[0053] In step 120, the time series data to be detected is decomposed into a period component, a trend component and a residual component.
[0054] The time series data to be detected is decomposed into a period component, a trend component and a residual component by using a seasonal trend decomposition algorithm based on local weighted regression. The seasonal trend decomposition algorithm is, for example, a seasonal trend decomposition procedure based on Loess (STL), where loess is a locally weighted regression algorithm.
[0055] In step 130, anomaly detection is performed on each component independently to determine whether the periodic component, trend component and residual component are abnormal.
[0056] Use the first time series prediction model to determine whether the periodic component is abnormal, for example, extract the periodic component S, and use the first time series prediction model to predict the periodic component when The difference with S is greater than the threshold τ S When , it is determined to be abnormal; the second time series prediction model is used to determine whether the trend component is abnormal, for example, the trend component T is extracted, and the trend component is predicted by the second time series prediction model when The difference with T is greater than the threshold τ T When the residual component is abnormal, it is judged as abnormal; the isolation forest model is used to judge whether the residual component is abnormal, for example, the residual component R is extracted, and the tree length of the residual component R is determined by the isolation forest model. By comparing the tree length of the residual component R with the expected tree length, it is judged whether the residual component R is abnormal. For example, if the tree length of the residual component R is less than the expected tree length, the residual component R is judged to be abnormal. Among them, the first time series prediction model and the second time series prediction model are the same or different time series prediction models. It will be combined later Figure 2 The training methods of each model are described in detail. The first time series prediction model is obtained by training using the period component of historical time series data; the second time series prediction model is obtained by training using the trend component of historical time series data; and the isolation forest model is obtained by training using the residual component of historical time series data.
[0057] Among them, the time series prediction model includes an autoregressive (AR) model, a moving average (MA) model, an autoregressive moving average (ARMA) model, or an autoregressive integrated moving average (ARIMA) model.
[0058] If the time series is a stationary non-white noise series, their mean and variance are constants. For this type of series, AR, MA, ARMA and other models can be used. If the autocorrelation graph of the time series is tailing and the partial correlation graph is truncated, use the AR model; if the autocorrelation graph of the time series is truncated and the partial correlation graph is tailing, use the MA model; if the autocorrelation graph and partial correlation graph of the time series are both tailing, use the ARMA model. If the time series is a non-stationary series, they can be converted into stationary series and then fitted according to the algorithm of the stationary series. For example, if the time series is stationary after difference, the ARIMA model can be used for fitting.
[0059] In step 140, based on whether the period component is abnormal, whether the trend component is abnormal, and whether the residual component is abnormal, the detection results of each component are combined to determine whether the time series data to be detected is abnormal.
[0060] According to whether the periodic component is abnormal, whether the trend component is abnormal, and whether the residual component is abnormal, a preset voting mechanism is used to determine whether the time series data to be detected is abnormal.
[0061] The voting mechanism includes: when any component is abnormal, the time series data to be detected is abnormal; or, when the majority of all components are abnormal, the time series data to be detected is abnormal; or, when all components are abnormal, the time series data to be detected is abnormal; or, weighted statistics are performed on the abnormal results of all components, and if the weighted statistical results meet the abnormal judgment conditions, it is determined that the time series data to be detected is abnormal.
[0062] The above embodiment, by decomposing the time series data to be detected into a periodic component, a trend component and a residual component, independently performs anomaly detection on each component, and then combines the detection results of each component to determine whether the time series data to be detected is abnormal. This can effectively avoid the situation where the abnormal characteristics of different components are different and the same detection algorithm cannot effectively detect data with different characteristics, and reduce the error of anomaly detection. For example, it can detect anomalies whose abnormal values are not significantly larger than periodic changes or trend changes.
[0063] Figure 2 A flowchart of a model training method according to some embodiments of the present disclosure is shown.
[0064] like Figure 2 As shown, the model training method of this embodiment includes: steps 210-240.
[0065] In step 210, a plurality of historical time series data are obtained.
[0066] For example, obtaining historical time series data from a cloud virtualization system.
[0067] In step 220, each piece of historical time series data is decomposed into a period component S, a trend component T and a residual component R.
[0068] In step 230, the models corresponding to the components are trained independently.
[0069] In step 230a, a first time series prediction model corresponding to the periodic component is trained: the periodic component of each historical time series data is predicted using the first time series prediction model. According to the forecast period component of the historical time series data The difference information between the predicted period component S and the actual period component S is obtained, for example, by updating the parameters of the first time series prediction model according to the gradient descent method until the preset training termination condition is met, for example, the predicted period component The gap between the actual cycle component S is smaller than the preset gap or reaches the preset number of training times.
[0070] In step 230b, train a second time series prediction model corresponding to the trend component: use the second time series prediction model to predict the trend component of each historical time series data According to the forecast trend component of the historical time series data The difference information between the predicted trend component T and the actual trend component T is obtained, for example, by updating the parameters of the second time series prediction model according to the gradient descent method until the preset training termination condition is met, for example, the predicted trend component The gap between the true trend component T and the actual trend component T is smaller than the preset gap or the preset number of training times is reached.
[0071] In step 230c, the isolation forest model corresponding to the residual component is trained: mainly including constructing a binary search tree (Binary Search Tree) by adopting multiple iterations, and then forming these binary search trees into a forest (i.e., the isolation forest model). Specifically, (1) constructing a binary search tree iTree: first randomly select X samples from the training data (i.e., the residual component R of the historical time series data). If iTree has reached the specified height or only one sample is left, the algorithm converges. Otherwise, recursively construct a binary search tree, put the samples smaller than the current root node into the left child node, and put the samples larger than the current root node into the right child node; (2) constructing a binary tree forest iForest: iteratively repeat the process of step (1) according to the sample data capacity to create a binary search tree iTree, and form the generated iTree into a binary tree forest. When the binary tree forest iForest is constructed, the sample can be predicted. The prediction process is to recursively traverse the binary search tree in order and record the path length from the root node to the leaf node (i.e., the tree length).
[0072] In step 240, the detection results of each component are combined to determine whether the historical time series data is abnormal.
[0073] For historical time series data, whether the period component, trend component and residual component are abnormal is determined by using a preset voting mechanism. The specific voting mechanism is described above.
[0074] Thus, the corresponding models of each component are obtained through independent training.
[0075] Figure 3 A schematic diagram showing a time series data anomaly detection device according to some embodiments of the present disclosure.
[0076] like Figure 3 As shown, the time series data anomaly detection device 300 of this embodiment includes: modules 310-340, and also includes modules 350-360.
[0077] The to-be-detected data acquisition module 310 is configured to acquire the time series data to be detected.
[0078] The time series data decomposition module 320 is configured to decompose time series data (such as time series data to be detected or historical time series data) into a period component, a trend component and a residual component.
[0079] The anomaly detection module 330 is configured to determine whether the periodic component, trend component and residual component are abnormal respectively; and determine whether the time series data (such as the time series data to be detected or the historical time series data) is abnormal based on whether the periodic component is abnormal, whether the trend component is abnormal and whether the residual component is abnormal.
[0080] The output module 340 is configured to output a detection result of whether the time series data to be detected is abnormal.
[0081] The training data storage module 350 is configured to store historical time series data.
[0082] The training module 360 is configured to respectively train a first time series prediction model, a second time series prediction model and an isolation forest model using the periodic component, trend component and residual component of historical time series data, wherein the first time series prediction model and the second time series prediction model are the same or different time series prediction models.
[0083] The model trained by the training module 360 can be output to the anomaly detection module 330, so that the anomaly detection module 330 uses each model to determine whether the corresponding components are abnormal, including: using the first time series prediction model to determine whether the periodic component is abnormal; using the second time series prediction model to determine whether the trend component is abnormal; using the isolation forest model to determine whether the residual component is abnormal.
[0084] During prediction, the to-be-detected data acquisition module 310 , the time series data decomposition module 320 , the anomaly detection module 330 , and the output module 340 are executed in sequence.
[0085] During training, the training data storage module 350, the time series data decomposition module 320, and the training module 360 are executed in sequence.
[0086] Figure 4 Schematic diagrams showing devices for detecting anomalies in time series data according to other embodiments of the present disclosure.
[0087] like Figure 4 As shown, the time series data anomaly detection device 400 of this embodiment includes:
[0088] The memory 410 and the processor 420 coupled to the memory 410 , wherein the processor 420 is configured to execute the time series data anomaly detection method in any one of the embodiments based on the instructions stored in the memory 410 .
[0089] For example, obtain the time series data to be detected; decompose the time series data to be detected into a periodic component, a trend component and a residual component; respectively determine whether the periodic component, the trend component and the residual component are abnormal; determine whether the time series data to be detected is abnormal based on whether the periodic component is abnormal, whether the trend component is abnormal and whether the residual component is abnormal.
[0090] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.
[0091] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the time series data anomaly detection method in any one of the embodiments.
[0092] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more non-transient computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.
[0093] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0096] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A time series data anomaly detection method, characterized in that: include: Obtain the time series data to be detected; Decomposing the time series data to be detected into a period component, a trend component and a residual component; Determining whether the period component, the trend component and the residual component are abnormal respectively, including: using the first time series prediction model to determine whether the period component is abnormal; using the second time series prediction model to determine whether the trend component is abnormal; using the isolation forest model to determine whether the residual component is abnormal; wherein the first time series prediction model and the second time series prediction model are the same or different time series prediction models; According to whether the periodic component is abnormal, whether the trend component is abnormal, and whether the residual component is abnormal, a preset voting mechanism is used to determine whether the time series data to be detected is abnormal, including: when any component is abnormal, the time series data to be detected is abnormal; or, when most of all components are abnormal, the time series data to be detected is abnormal; or, when all components are abnormal, the time series data to be detected is abnormal; or, weighted statistics are performed on the abnormal results of all components, and if the weighted statistical results meet the abnormal judgment conditions, the time series data to be detected is determined to be abnormal.
2. The method according to claim 1, characterized in that: Decomposing the time series data to be detected into a period component, a trend component and a residual component includes: The seasonal trend decomposition algorithm based on local weighted regression is used to decompose the time series data to be detected into a period component, a trend component and a residual component.
3. The method according to claim 1, characterized in that The first time series prediction model is obtained by training using the periodic component of historical time series data; The second time series prediction model is obtained by training using the trend component of historical time series data; The isolation forest model is trained using the residual components of historical time series data.
4. The method according to claim 1, characterized in that The time series forecasting models include an autoregressive AR model, a moving average MA model, an autoregressive moving average ARMA model, or a difference autoregressive moving average ARIMA model.
5. The method according to any one of claims 1 to 4, characterized in that: The time series data to be detected includes the CPU resource time series data, memory resource time series data, storage resource time series data, and network resource time series data of the cloud virtualization system.
6. A time series data anomaly detection device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the time series data anomaly detection method according to any one of claims 1 to 5 based on instructions stored in the memory.
7. A time series data anomaly detection device, comprising: A module for acquiring data to be detected, configured to acquire time series data to be detected; A time series data decomposition module is configured to decompose the time series data to be detected into a period component, a trend component and a residual component; The anomaly detection module is configured to judge whether the periodic component, the trend component and the residual component are abnormal respectively, including: using the first time series prediction model to judge whether the periodic component is abnormal; using the second time series prediction model to judge whether the trend component is abnormal; using the isolation forest model to judge whether the residual component is abnormal; wherein the first time series prediction model and the second time series prediction model are the same or different time series prediction models; according to whether the periodic component is abnormal, whether the trend component is abnormal and whether the residual component is abnormal, using a preset voting mechanism, determining whether the time series data to be detected is abnormal, including: when any component is abnormal, the time series data to be detected is abnormal; or, when most of all components are abnormal, the time series data to be detected is abnormal; or, when all components are abnormal, the time series data to be detected is abnormal; or, performing weighted statistics on the abnormal results of all components, if the weighted statistical results meet the abnormal judgment conditions, determining that the time series data to be detected is abnormal; The output module is configured to output a detection result of whether the time series data to be detected is abnormal.
8. The time series data anomaly detection device according to claim 7, further comprising: Training data storage module and training module; wherein, The training data storage module is configured to store historical time series data; The time series data decomposition module is further configured to decompose the historical time series data into a period component, a trend component and a residual component; The training module is configured to respectively train a first time series prediction model, a second time series prediction model and an isolation forest model using the periodic component, trend component and residual component of historical time series data, wherein the first time series prediction model and the second time series prediction model are the same or different time series prediction models.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the time series data anomaly detection method according to any one of claims 1 to 5.