Abnormity detection system based on time sequence algorithm

Abstract

The invention discloses an anomaly detection system based on a time sequence algorithm. The anomaly detection system is used for solving the problems that a single anomaly detection method is low in accuracy, high in complexity and poor in applicability. The system comprises a data acquisition module which is responsible for acquiring key performance index data of the IT system; the data processing module is used for cleaning and converting the collected data; the data storage module is used for storing the processed data to a time sequence database; the feature extraction module is used for extracting data mode features by using a MultiRocket algorithm; the classification module is used for classifying the feature data by adopting a random forest classifier; the anomaly detection module dynamically selects an anomaly detection algorithm to detect data anomaly according to the classification result; and the alarm module transmits the abnormal information to the operation and maintenance management platform. According to the method, the data is classified, and the proper anomaly detection algorithm is dynamically selected based on the classification result, so that the limitation of a single anomaly detection method is overcome, and the applicability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of security monitoring, and in particular to an anomaly detection system based on a time series algorithm. Background Art

[0002] With the rapid development of the Internet, IT systems are becoming increasingly large and complex. To ensure the safe and stable operation of IT systems, key performance indicators (KPIs) are widely used to monitor the health of IT systems, such as the CPU usage and memory usage of servers or the service availability and page views of application systems. In order to monitor the operating status of IT systems in real time and efficiently, an automated anomaly detection system must be used.

[0003] KPI data is time series data collected at a fixed frequency. In large and complex IT systems, there are many types of KPI data, and their fluctuation trends can usually be abstracted into three categories: stable, cyclical, and irregular fluctuations.

[0004] At present, anomaly detection algorithms for time series include statistical anomaly detection algorithms, machine learning anomaly detection algorithms, and deep learning anomaly detection algorithms. Statistical anomaly detection algorithms have high computational efficiency and good detection effects on linear and normally distributed data, but poor detection effects on complex data such as nonlinear and non-normal distributions, and often require manual adjustment of parameters, lacking adaptability. Machine learning-based and deep learning-based anomaly detection algorithms can effectively process complex nonlinear and non-normal distribution data, but due to the high complexity of the model, they usually require a large amount of data and computing resources, take a long time to train, and have high computing costs.

[0005] In summary, a single anomaly detection algorithm lacks sufficient flexibility when dealing with complex and diverse KPI data, resulting in low detection accuracy or high consumption of computing resources, and it is difficult to achieve a good balance between detection accuracy and computational complexity. Therefore, there is an urgent need for an anomaly detection system based on a time series algorithm that can automatically classify KPI data and select a suitable anomaly detection algorithm based on the classification results. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an anomaly detection system based on a time series algorithm, which classifies the collected KPI data through a time series algorithm, dynamically selects a suitable anomaly detection algorithm for anomaly detection according to the data category, and transmits the anomaly information to the operation and maintenance management platform.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] An anomaly detection system based on a time series algorithm, characterized in that it includes a data acquisition module, a data processing module, a data storage module, a feature extraction module, a classification module, an anomaly detection module and an alarm module;

[0009] The data acquisition module obtains IT system KPI data in real time through Agent, SNMP protocol, and HTTP protocol; the data processing module cleans and converts the collected data; the data storage module stores the processed data persistently in TimescaleDB; the feature extraction module extracts features from the data through the MultiRocket algorithm; the classification module selects the random forest classifier to train the classification model and classify the data; the anomaly detection module uses the corresponding anomaly detection algorithm to perform anomaly detection based on the data category; the alarm module transmits the abnormal information to the operation and maintenance management platform through alarm media such as email, SMS, and enterprise WeChat.

[0010] Furthermore, the data collection module collects KPI data of the IT system at a specified frequency through Agent, SNMP protocol, HTTP protocol, etc.

[0011] Furthermore, the data processing module performs data cleaning and conversion on the collected data, including the following steps:

[0012] The n nearest neighbor mean method is used to fill in missing data values, where the missing values ​​include data that are not obtained at the specified collection frequency or data that are collected at the specified collection frequency but the observation value is empty;

[0013] Wavelet filtering technology is used to process data noise. The data is decomposed into multiple wavelet coefficients of different scales and frequencies through wavelet transform. A specific filter is applied to smooth the wavelet coefficients. The processed wavelet coefficients are recombined to generate data after noise removal.

[0014] The maximum-minimum normalization method is used to normalize the data and linearly transform the data to the interval [0,1] to facilitate subsequent feature extraction and model training.

[0015] Furthermore, the data storage module performs time series data storage based on TimescaleDB, and stores the processed data efficiently and persistently.

[0016] Furthermore, the feature extraction module uses the MultiRocket algorithm to extract data time series features, including the following steps:

[0017] Perform first-order difference processing on the preprocessed data to obtain the original time series and the time series after first-order difference, and generate two representations for subsequent feature extraction;

[0018] Construct a large number of randomly generated convolution kernels to perform convolution operations on the two representations, and perform convolution operations on the original time series and the time series after the first-order difference respectively. By adjusting different bias values ​​in the convolution process, the convolution results are adjusted to capture the local patterns and structural features of the data;

[0019] The convolution results are pooled to extract four key statistical features, including positive value ratio, positive value mean, positive value index mean, and longest positive value interval range.

[0020] Furthermore, the classification module uses a random forest classifier to classify the extracted data features. The classifier is obtained in advance through training and can classify the data into the following three categories: stable type, periodic type and irregular fluctuation type.

[0021] Furthermore, the anomaly detection module uses a corresponding anomaly detection algorithm to perform anomaly detection based on the data category, uses a 3-Sigma anomaly detection algorithm for stationary data, and uses an anomaly detection algorithm based on an LSTM prediction model combined with 3-Sigma for periodic data and irregular fluctuation data.

[0022] Furthermore, the 3-Sigma anomaly detection algorithm determines points that deviate from the data mean by more than 3 times the standard deviation as abnormal data.

[0023] Furthermore, the anomaly detection algorithm based on the LSTM prediction model combined with 3-Sigma includes the following steps:

[0024] Design and build an LSTM model, determine the number of layers and hidden units, and train the LSTM model using historical data to capture complex dynamic changes in time series;

[0025] Use the trained LSTM model to predict the data and calculate the residual between each predicted value and the observed value;

[0026] The 3-Sigma anomaly detection algorithm is used to determine points where the residual deviation exceeds 3 times the standard deviation as anomalies.

[0027] Furthermore, the alarm module integrates Alertmanager to transmit abnormal information to the operation and maintenance management platform through alarm media such as email, SMS, and enterprise WeChat.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention proposes an anomaly detection system based on a time series algorithm, which can dynamically and adaptively select a suitable anomaly detection algorithm according to the characteristics of different types of KPI data. The system can accurately identify various time series data patterns such as stable, periodic, and irregular fluctuations, thereby ensuring the adaptive selection of the optimal anomaly detection algorithm for each data type. Compared with detection that relies on fixed thresholds or deep learning models with high computational overhead, the accuracy and adaptability of detection are significantly improved, and the consumption of computing resources is reduced.

[0030] The MultiRocket classification algorithm adopted in the present invention is efficient and concise, and can maintain extremely low computational complexity and fast response time when processing large-scale KPI data. Compared with deep neural networks, MultiRocket not only greatly reduces the system's computing resource requirements, but also can quickly and accurately complete KPI data classification in real-time monitoring scenarios.

[0031] The present invention adopts an anomaly detection algorithm based on LSTM prediction model combined with 3-Sigma, uses LSTM to predict the trend of time series data, and combines 3-Sigma rules to detect anomalies between the residuals of predicted values ​​and observed values. The algorithm improves the accuracy of anomaly detection of periodic data and irregular fluctuation data. The modular design of the system also allows more anomaly detection algorithms to be flexibly expanded according to business needs, further improving the versatility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of an anomaly detection system provided in an embodiment of the present invention.

[0033] Figure 2 A flowchart of the feature extraction and classification steps of the MultiRocket algorithm provided in an embodiment of the present invention.

[0034] Figure 3 A flowchart of the abnormality detection steps provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0036] like Figure 1 As shown, the present invention proposes an anomaly detection system based on a time series algorithm, comprising a data acquisition module (11), a data processing module (12), a data storage module (13), a feature extraction module (14), a classification module (15), an anomaly detection module (16) and an alarm module (17).

[0037] The data collection module (11) collects KPI data of the IT system at a specified frequency through Agent, SNMP protocol, HTTP protocol, etc. The key performance indicators of various IT systems are specifically shown in Table 1.

[0038] Table 1: IT system key performance indicator collection table

[0039]

[0040] The data processing module (12) uses the n nearest neighbor mean method and wavelet filtering technology to clean and convert the collected data.

[0041] The data storage module (13) stores the processed data persistently in the time series table of TimescaleDB according to the predefined time series data model, and starts the automatic compression strategy for historical data to save storage space. The time series data model consists of timestamp, data value and metadata. The timestamp is used to record the data collection time, the data value is used to store the monitoring indicator observation value, and the metadata contains the attribute information of the data (host ID, monitoring item, etc.). The automatic compression strategy will compress historical data over 30 days. The compressed data is in read-only state and supports query but cannot be modified.

[0042] The feature extraction module (14) uses the MultiRocket algorithm to extract four types of key statistical features of the data, including positive value ratio, positive value mean, positive value index mean, and longest positive value interval range; the classification module (15) uses a random forest classifier to classify the data according to the extracted features, and classifies the data into three types: stable type, periodic type, and irregular fluctuation type.

[0043] Specifically, Figure 2 As shown, feature extraction and classification of KPI data based on the MultiRocket algorithm includes the following steps:

[0044] Processed KPI time series data Perform a first-order difference transformation to obtain a first-order difference time series, and use the two representations of the original time series and the time series after the first-order difference for subsequent feature extraction;

[0045]

[0046] Construct 84 random convolution kernels, each with a length of 9 and initial weights of -1 and 2. The convolution kernel is a combination of initial weights, such as [-1,-1,-1,2,2,2,2,2,2];

[0047] A different dilation rate d is applied to each convolution kernel. The dilation rate d determines the step size of the convolution kernel when it is applied to the time series, capturing patterns of different scales in the time series, which are distributed in the exponential range. , the max calculation formula is as follows:

[0048]

[0049] max is the maximum expansion degree, where is the length of the input time series, is the length of the convolution kernel;

[0050] Use the convolution kernel to perform convolution operation on the two representations, select different bias values ​​from the convolution result, and adjust the convolution result;

[0051] Convolution results Pooling operation is performed to generate four types of statistical features, including positive value proportion PPV, positive value mean MPV, positive value index mean MIPV, and longest positive value interval range LSPV. The calculation formula is as follows:

[0052]

[0053] in represents the indicator function, when The value is 1 when , otherwise it is 0.

[0054]

[0055] Where m represents the number of positive values ​​in the set, represents the i-th positive value.

[0056] Where m represents the number of positive values ​​in the set, Represents the index corresponding to the jth positive value in the set. If the number of positive values ​​is 0, the average of the positive index is set to -1.

[0057]

[0058] Where i and j represent the start and end indexes of the continuous positive value interval.

[0059] The pooled statistical features are combined into a feature matrix, and the feature matrix is ​​classified using a random forest classifier to classify the time series into stationary, cyclical, and irregularly fluctuating types.

[0060] The anomaly detection module (16) dynamically selects a corresponding anomaly detection algorithm based on the data category to perform anomaly detection. For stable data, the 3-Sigma anomaly detection algorithm is used. For periodic data and irregular fluctuation data, the anomaly detection algorithm based on the LSTM prediction model combined with 3-Sigma is used.

[0061] Specifically, Figure 3 As shown in the figure, the 3-Sigma anomaly detection algorithm and the anomaly detection algorithm based on the LSTM prediction model combined with the 3-Sigma anomaly detection algorithm perform anomaly detection on the KPI data. The anomaly detection process includes the following steps:

[0062] The 3-Sigma anomaly detection algorithm uses a sliding window to split the time series data. For each point to be detected , get the k-nearest neighbor window , calculate the mean value for this window and standard deviation , use 3-Sigma anomaly detection algorithm to treat the detection points To test and judge Is it distributed in If it exceeds this range, it is considered abnormal. and standard deviation The calculation formula is as follows:

[0063]

[0064]

[0065] Based on the LSTM prediction model and 3-Sigma anomaly detection algorithm, the time series data is segmented by sliding windows. , get the k-nearest neighbor window , input the sliding window data into the trained LSTM model to generate the predicted value , calculate the residuals between the predicted and observed values , use the 3-Sigma anomaly detection algorithm to determine the residual Is it abnormal, residual The calculation formula is as follows:

[0066]

[0067] The alarm module (17) is integrated with Alertmanager, which is used to format the abnormal information into alarm information and transmit the alarm information to the operation and maintenance management platform according to the preset routing rules. The abnormal information is pushed to the Alertmanager in JSON format, including the alarm status, alarm level, alarm name, alarm group, abnormal device, abnormal information, abnormal time and other fields in the form of key-value. The Alertmanager parses and formats the received JSON data, converts it into structured alarm information, and sends the alarm information to the designated receiver in the designated medium according to the grouping strategy and alarm strategy defined in the routing rules.

[0068] The above examples are only specific embodiments of the present invention, but the present invention is not limited thereto. Under the condition of respecting the experimental results and not departing from the spirit and essence of the present invention, any technician familiar with the art can make various corresponding changes and deformations under the enlightenment of the embodiments of the present invention, but these corresponding changes and deformations should be covered within the protection scope of the claims of the present invention.

Claims

1. An anomaly detection system based on time series algorithm, characterized in that: The system comprises a data collection module (11), a data processing module (12), a data storage module (13), a feature extraction module (14), a classification module (15), an anomaly detection module (16) and an alarm module (17); the data collection module (11) acquires IT system KPI data in real time through Agent, SNMP protocol, HTTP protocol and the like; the data processing module (12) cleans and converts the collected data; the data storage module (13) stores the processed data persistently in TimescaleDB; the feature extraction module (14) extracts features from the data through the MultiRocket algorithm; the classification module (15) selects a random forest classifier to train a classification model and classifies the data based on the extracted features; the anomaly detection module (16) performs anomaly detection using a corresponding anomaly detection algorithm according to the data category; the alarm module (17) integrates Alertmanager and transmits anomaly information to an operation and maintenance management platform through a multi-channel alarm mechanism.

2. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The data collection module (11) collects key performance indicator (KPI) data of the IT system at a specified frequency through Agent, SNMP protocol, HTTP protocol, etc.

3. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The data processing module (12) performs data cleaning and conversion on the collected data, including the following steps: The n nearest neighbor mean method is used to fill in missing data values, where the missing values ​​include data that are not collected at the specified frequency or data that are collected at the specified frequency but have empty values; Wavelet filtering technology is used to process data noise. The data is decomposed into multiple wavelet coefficients of different scales and frequencies through wavelet transform. A specific filter is applied to smooth the wavelet coefficients. The processed wavelet coefficients are recombined to generate data after noise removal. The maximum-minimum normalization method is used to normalize the data and linearly transform the data to the interval [0,1] to facilitate subsequent feature extraction and model training.

4. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The data storage module (13) stores the data persistently in TimescaleDB.

5. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The feature extraction module (14) uses the MultiRocket algorithm to extract data time series features, including the following steps: Perform first-order difference processing on the data to obtain the original time series and the time series after first-order difference, and generate two representations for subsequent feature extraction; Construct a large number of randomly generated convolution kernels to perform convolution operations on the two representations, and perform convolution operations on the original time series and the time series after the first-order difference respectively. By adjusting different bias values ​​in the convolution process, the convolution results are adjusted to capture the local patterns and structural features of the data; The convolution results are pooled to extract four key statistical features, including positive value ratio, positive value mean, positive value index mean, and longest positive value interval range.

6. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The classification module (15) uses a random forest classifier to classify the extracted feature data. The classifier is obtained through pre-training and can classify the data into the following three categories: stable type, periodic type and irregular fluctuation type.

7. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The anomaly detection module (16) uses a corresponding anomaly detection algorithm to perform anomaly detection based on the data category. For stable data, a 3-Sigma anomaly detection algorithm is used. For periodic data and irregular fluctuation data, an anomaly detection algorithm based on an LSTM prediction model combined with 3-Sigma is used.

8. The anomaly detection system based on time series algorithm according to claim 7, characterized in that: The 3-Sigma anomaly detection algorithm determines data that deviates from the data mean by more than 3 times the standard deviation as abnormal data.

9. The anomaly detection system based on time series algorithm according to claim 7, characterized in that: The anomaly detection algorithm based on the LSTM prediction model combined with 3-Sigma includes the following steps: Design and build an LSTM model, determine the number of layers and hidden units, and train the LSTM model using historical data to capture complex dynamic changes in time series; Use the trained LSTM model to predict the data and calculate the residual between each predicted value and the observed value; The 3-Sigma anomaly detection algorithm is used to determine the value of the residual deviation exceeding 3 times the standard deviation as abnormal.

10. The anomaly detection system based on time series algorithm according to claim 1, characterized in that: The alarm module (17) is integrated with Alertmanager to transmit abnormal information to the operation and maintenance management platform via alarm media such as email, SMS, and enterprise WeChat.

Images