Lightweight time series data anomaly detection method based on machine learning

Abstract

The invention relates to a lightweight time series data anomaly detection method based on machine learning. The lightweight time series data anomaly detection method based on machine learning is provided. Carrying out integrity check and periodical check on the historical data, and carrying out periodical prediction; performing integration processing on a periodic prediction result and meta-information of the to-be-trained node to obtain a prediction result containing the meta-information; and acquiring real-time data from an online real-time streaming database, and performing anomaly detection on the real-time data according to the prediction result containing the meta-information. The method has the beneficial effects that whether the data is suitable for periodic prediction or not is determined through periodic verification of the time series data, a periodic prediction value or a pre-estimation alarm threshold value which is manually set is obtained and is used as a reference value for anomaly detection, and thus massive real-time streams are monitored in equipment with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of time series data detection, and more specifically, to a lightweight time series data anomaly detection method based on machine learning. Background Art

[0002] Existing industrial time series data anomaly detection algorithms include: statistical-based methods, such as Z-score data threshold marking and seasonal decomposition; model-based methods, such as autoregressive integrated moving average model and long short-term memory network; machine learning-based methods, such as isolation forest and support vector machine; deep learning-based methods, such as AutoEncoder.

[0003] However, the current mainstream time series data anomaly detection methods have the following defects: 1 In statistical methods, due to the differences in seasonality and abnormal intervals of data with different business attributes, in actual situations, general anomaly detection involving multiple data usually cannot unify the prediction algorithm and can only detect situations with large deviation values.

[0004] 2 In the model-based method, the selection of model parameters is relatively sensitive, and the data needs to be tested for stationarity and differential processing. At the same time, the model fitting is relatively complex and the computational burden is heavy.

[0005] 3 In machine learning-based methods, feature selection and extraction are very challenging and usually cannot handle the sequential relationship of time series.

[0006] 4 Among the deep learning-based methods, the training of models such as autoencoders is relatively complex. In actual production environments, due to limited model deployment resources, they are prone to crashes. At the same time, training tens of thousands of data takes a long time, and the results incorporate too many features and have poor interpretability. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies in the prior art and propose a lightweight time series data anomaly detection method based on machine learning.

[0008] In a first aspect, a lightweight time series data anomaly detection method based on machine learning is provided, comprising: Step 1: According to the meta information of the node to be trained, historical data of the node to be trained is obtained from a specified table and a specified time range of the online real-time stream database as a training set; Step 2: Perform integrity check and periodicity check on the historical data, and perform periodicity prediction; Step 3: Integrate the periodic prediction results and the meta information of the node to be trained to obtain the prediction results containing the meta information; Step 4: Acquire real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result including the meta-information.

[0009] Preferably, in step 1, the meta information of the node to be trained includes the target database of the node, the unique identification column name, and the length of the historical data cycle.

[0010] Preferably, step 2 comprises: Step 2.1, perform integrity check on the historical data, remove excessively small and excessively large values ​​in the historical data, and perform linear interpolation processing on the historical data to obtain a filtered and cleaned training set; Step 2.2: Periodically check the filtered and cleaned training set and make periodic predictions, including: Step 2.2.1, divide the filtered and cleaned training set into a pre-training set and a test set; Step 2.2.2, decomposing the pre-training set into seasonal and trend components to obtain trend components and seasonal components. If the proportion of seasonal components is less than a preset threshold, then exit the prediction; Step 2.2.3: Fit and predict the pre-training set whose seasonal component ratio is greater than or equal to the preset threshold, and calculate the mean absolute error between the prediction result and the test set; if the error is within the acceptable range, perform integrated learning on the complete historical data, assign different weights and obtain the final weighted result as the prediction result.

[0011] Preferably, in step 2.2.3, if the error exceeds the acceptable range, a kernel density estimation algorithm is used to estimate the distribution of the historical data, and an estimated alarm threshold is manually set, and the estimated alarm threshold is used as the prediction result.

[0012] Preferably, step 4 comprises: Step 4.1, obtain real-time data from the online real-time stream database, aggregate it according to the preset value, input the aggregated data stream into the message queue, and execute the online real-time stream detection task; Step 4.2: Determine the abnormality of the data through an anomaly detector. If the abnormality exceeds a set value, an abnormal event is determined.

[0013] Preferably, step 4.2 comprises: Step 4.2.1, key-control and divert the real-time data so that each node has a corresponding anomaly detector; Step 4.2.2, initializing the anomaly detector; Step 4.2.3: Determine whether the data length in the anomaly detector is greater than the preset length. If so, comprehensively refer to the predicted value corresponding to the current moment, the estimated alarm threshold, the tolerance for short-term data fluctuations, and the historical moving average to determine the degree of deviation of the data, and use the degree of deviation as the abnormality of the current data; if not, use the estimated alarm threshold as the abnormality of the current data; Step 4.2.4: If the abnormality exceeds the set value, it is determined that an abnormal event has occurred.

[0014] As a preference, it also includes: Step 5: When an abnormal event occurs, determine the level of the abnormal event and automatically process abnormal events of different levels.

[0015] In a second aspect, a lightweight time series data anomaly detection system based on machine learning is provided, which is used to execute any method described in the first aspect, including: An acquisition module is used to acquire historical data of the node to be trained as a training set in a specified library table and a specified time range of an online real-time stream database according to the meta information of the node to be trained; A prediction module, used to perform integrity check and periodicity check on the historical data, and perform periodic prediction; An integration module is used to integrate the results of periodic prediction and the meta information of the node to be trained to obtain the prediction results containing the meta information; The detection module is used to obtain real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result containing the meta information.

[0016] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0017] In a fourth aspect, an electronic device is provided, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement any method as described in the first aspect.

[0018] The beneficial effects of the present invention are: 1. The present invention proposes a lightweight time series data anomaly detection scheme based on machine learning. By periodically checking the time series data, it is determined whether the data is suitable for periodic prediction, and a periodic prediction value is obtained or an estimated alarm threshold is manually set, which is used as a reference value for anomaly detection, thereby realizing the monitoring of massive real-time streams in devices with limited resources.

[0019] 2. The present invention proposes a time series data training management system, which manages the time series data measurement points that need to be monitored and maintains the basic information of the measurement points, such as the source of training data, the length of historical cycles, etc. At the same time, the scheduled tasks are scheduled to train the measurement points, and the prediction results are exported and sent to the designated server.

[0020] 3. The present invention proposes a real-time stream abnormal event reporting and alarm scheme, which uses a machine learning detection scheme to monitor the real-time stream of time series data and report abnormal situations. The reported abnormal events are re-determined and alarmed, and different alarms are automatically processed through alarm configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of lightweight anomaly detection based on machine learning provided by an embodiment of the present invention; Figure 2 A periodic verification flow chart provided for an embodiment of the present invention; Figure 3 A flow chart of prediction result management provided by an embodiment of the present invention; Figure 4 A flow chart of the working principle of the detector provided by an embodiment of the present invention; Figure 5 This is a flowchart of the alarm center provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0023] Embodiment 1: In order to solve the problems of the prior art, Example 1 of the present application provides a lightweight time series data anomaly detection method based on machine learning, which solves the common feature extraction and time series processing problems of machine learning through the Prophet framework. The Prophet framework can automatically identify the seasonal and trend components in the data. It uses a piecewise linear or smoothing model to capture the changes in trends and allows users to customize the seasonal components (that is, cyclical components) of weeks and days. In this application, according to the business characteristics of the data, only the possibility of single-day periodicity and single-week periodicity is set and medium- and short-term predictions are performed. In the time series, the missing values ​​of the time series are linearly interpolated according to the business attributes, and the values ​​that are too high or too low are truncated.

[0024] Specifically, Figure 1As shown in the figure, the method consists of two parts: offline training prediction and online real-time stream operation. The lightweight time series data anomaly detection method based on machine learning includes: Step 1: According to the meta information of the node to be trained, the historical data of the node to be trained is obtained from the specified library table and specified time range of the online real-time stream database as a training set.

[0025] In step 1, the meta information of the node to be trained includes information such as the target database of the node, the unique identification column name, the length of the historical data cycle, etc.

[0026] It should be noted that this application performs separate training for each single-dimensional data. Each single-dimensional data stores the metadata of the training node in the structured database, including the training configuration information. The training management platform will start the corresponding training task for each training node based on the metadata, and the training results obtained are also aggregated, organized, and saved as a single data file in units of single tasks.

[0027] For example, after the time series data training management platform initiates a scheduled training task, the training task retrieves metadata from a structured database such as MySQL, such as the library table where the node is located and the historical period. When the historical period is 7 days, a connection to the online real-time stream database is established and the specified library table is entered to pull the data from the previous 7 days as a training set.

[0028] Step 2: Perform integrity check and periodicity check on the historical data, and perform periodic prediction.

[0029] like Figure 2 As shown, step 2 includes: Step 2.1: perform integrity check on the historical data, remove excessively small and excessively large values ​​in the historical data, and perform linear interpolation processing on the historical data to obtain a filtered and cleaned training set.

[0030] For example, the integrity check needs to remove the values ​​that are too small below the 2% quantile and the values ​​that are too large above 98%.

[0031] Step 2.2: Periodically check the filtered and cleaned training set and make periodic predictions, including: Step 2.2.1: Divide the filtered and cleaned training set into a pre-training set and a test set.

[0032] For example, the training set of n days is divided into two parts, the training set of the last two days is used as the test set, and the data set of the first n-2 days is used as the pre-training set.

[0033] Step 2.2.2: Perform seasonal and trend decomposition (using LOESS, STL) on the pre-training set to obtain trend component and seasonal component. If the proportion of seasonal component is less than a preset threshold (such as 10%), the data is considered unsuitable for periodic forecasting.

[0034] Step 2.2.3, fit and predict the pre-training set whose seasonal component ratio is greater than or equal to the preset threshold, obtain the prediction results for the next two days, and calculate the mean absolute error (MAE) between the prediction results and the test set. The formula is as follows:

[0035] in, is the number of data sets, For the Time series data values, For the predicted value.

[0036] If the error is within an acceptable range, the complete historical data is integrated and learned, different weights are assigned, and the final weighted result is obtained as the prediction result.

[0037] For example, if the MAE is greater than 30% of the average value of the test set, the data is considered not suitable for periodic prediction, and the kernel density estimation algorithm (KDE) is used to estimate the distribution of historical data, and the estimated alarm threshold is manually set, and the estimated alarm threshold is used as the prediction result. The formula of the kernel density estimation algorithm is as follows.

[0038]

[0039] in, It's on point The estimated density at is the sample size, is the bandwidth parameter that affects the degree of smoothing, is the first data points, is the kernel function, and in this paper, the Gaussian kernel is used, that is:

[0040] For example, if the MAE is greater than 30% of the average value of the test set, the data is considered unsuitable for periodic forecasting, and the distribution of historical data is estimated using the kernel density estimation algorithm (KDE), and the estimated alarm threshold is manually set, and the estimated alarm threshold is used as the forecast result. If the error is within an acceptable range, the complete historical data is integrated and learned using models such as Multilayer Perceptron (MLP), DeepAR, and LSTM, and different weights are assigned to obtain the final weighted result as the forecast result.

[0041] Step 3: Integrate the periodic prediction results and the meta-information of the nodes to be trained to obtain the prediction results containing the meta-information.

[0042] like Figure 3 As shown, the obtained prediction data or pre-manual estimated thresholds are integrated, and the meta-information of the pre-trained nodes, such as the target database, target server, and time dimension, are integrated and processed. A transmission request is initiated by a scheduled task and sent to a specified folder of a specified server as one of the judgment bases for real-time flow detection.

[0043] Step 4: Acquire real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result including the meta-information.

[0044] Embodiment 2: Based on Example 1, Example 2 of the present application provides a more specific lightweight time series data anomaly detection method based on machine learning, including: Step 1: According to the meta information of the node to be trained, the historical data of the node to be trained is obtained from the specified library table and specified time range of the online real-time stream database as a training set.

[0045] Step 2: Perform integrity check and periodicity check on the historical data, and perform periodic prediction.

[0046] Step 3: Integrate the periodic prediction results and the meta-information of the nodes to be trained to obtain the prediction results containing the meta-information.

[0047] Step 4: Acquire real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result including the meta-information.

[0048] like Figure 4 As shown, step 4 includes: Step 4.1: Get real-time data from the online real-time stream database, aggregate it according to the preset value, input the aggregated data stream into the message queue (such as Kafka), and execute the online real-time stream detection task.

[0049] For example, aggregation according to preset values ​​includes: obtaining the "aggregation granularity" = 1 minute in the meta information of the node a to be trained. Using the Watermark and Window methods of the real-time stream processing framework Flink, sum the data values ​​arriving within 1 minute, and set the timestamp of the result to the timestamp of the first data arriving in the minute. The aggregated data is still in the form of a stream, and the time granularity is consistent with the "aggregation granularity" in the meta information.

[0050] Step 4.2: Determine the abnormality of the data through an anomaly detector. If the abnormality exceeds a set value, an abnormal event is determined.

[0051] Step 4.2 includes: Step 4.2.1: Key-control and divert the real-time data so that each node has a corresponding anomaly detector.

[0052] Keyed diversion means: in stream processing applications, data is grouped using its uniquely identified field as a key, and each parallel instance processes data with a specific key. Each key will correspond to its own partition and maintain its own state in the partition. In this application, the state refers to arrays and variables such as "historical moving average" and "short-term data fluctuations" in the "detector". Specifically, the keyBy operation is used in the stream processing framework Flink to perform keyed diversion. This operation partitions the original stream data to ensure that after the above aggregation operation, each aggregated data is the sum of the data reported by the same device within one minute, which corresponds to the prediction result described above.

[0053] Step 4.2.2: Initialize the anomaly detector.

[0054] For example, if the detector is not initialized or the interval from the last detection is 30 minutes or more, the detector is initialized.

[0055] Step 4.2.3, determine whether the data length in the anomaly detector is greater than the preset length. If so, comprehensively refer to the predicted value corresponding to the current moment, the estimated alarm threshold, the short-term data fluctuation tolerance, and the historical moving average to determine the degree of deviation of the data, and use the degree of deviation as the abnormality of the current data; if not, use the estimated alarm threshold as the abnormality of the current data.

[0056] For example, it is determined whether the length of the short-term data in the detector is greater than 5 minutes. If it is greater than 5 minutes, the prediction value corresponding to the current minute, the estimated alarm threshold, the tolerance of short-term data fluctuations, and the historical moving average are comprehensively referenced to determine the degree of deviation of the data as the abnormality of the current data. Otherwise, only the manually set threshold is used for comparison.

[0057] Among them, the short-term data volatility tolerance in this article adopts volatility normalization, and its formula is as follows:

[0058]

[0059] in, is the normalized volatility, , are the lowest and highest volatility in the sample, respectively. is the sample size, is the first data points, is the sample mean.

[0060] The historical moving average in this article uses the exponential moving average (EMA), and its formula is as follows:

[0061] in, It's time The exponential moving average of It's time The exponential moving average of It's time The time series data value of is the smoothing factor.

[0062] Step 4.2.4: If the abnormality exceeds the set value, it is determined that an abnormal event has occurred.

[0063] Step 5: When an abnormal event occurs, determine the level of the abnormal event and automatically process abnormal events of different levels.

[0064] Specifically, Figure 5 As shown in the figure, the real-time stream detection task sends abnormal events to the message queue, the alarm center subscribes to the abnormal event queue, maintains the abnormal event status library, and continuously updates the level of each event, continuous occurrence time, whether it has been taken over, and emergency contact information. According to the abnormal alarm configuration, it determines whether the event has risen, and automatically processes events of different levels.

[0065] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0066] Embodiment 3: Based on Example 1, Example 3 of the present application provides a lightweight time series data anomaly detection system based on machine learning, including: An acquisition module is used to acquire historical data of the node to be trained as a training set in a specified library table and a specified time range of an online real-time stream database according to the meta information of the node to be trained; A prediction module, used to perform integrity check and periodicity check on the historical data, and perform periodic prediction; An integration module is used to integrate the results of periodic prediction and the meta information of the node to be trained to obtain the prediction results containing the meta information; The detection module is used to obtain real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result containing the meta information.

[0067] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiment 1. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

Claims

1. A lightweight time series data anomaly detection method based on machine learning, characterized in that: include: Step 1: According to the meta information of the node to be trained, historical data of the node to be trained is obtained from a specified table and a specified time range of the online real-time stream database as a training set; Step 2: Perform integrity check and periodicity check on the historical data, and perform periodicity prediction; Step 3: Integrate the periodic prediction results and the meta information of the node to be trained to obtain the prediction results containing the meta information; Step 4: Acquire real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result including the meta-information.

2. The lightweight time series data anomaly detection method based on machine learning according to claim 1 is characterized in that: In step 1, the meta information of the node to be trained includes the target database of the node, the unique identification column name, and the length of the historical data cycle.

3. The lightweight time series data anomaly detection method based on machine learning according to claim 1 is characterized in that: Step 2 includes: Step 2.1, perform integrity check on the historical data, remove excessively small and excessively large values ​​in the historical data, and perform linear interpolation processing on the historical data to obtain a filtered and cleaned training set; Step 2.2: Periodically check the filtered and cleaned training set and make periodic predictions, including: Step 2.2.1, divide the filtered and cleaned training set into a pre-training set and a test set; Step 2.2.2, decomposing the pre-training set into seasonal and trend components to obtain trend components and seasonal components. If the proportion of seasonal components is less than a preset threshold, then exit the prediction; Step 2.2.3: Fit and predict the pre-training set whose seasonal component ratio is greater than or equal to the preset threshold, and calculate the mean absolute error between the prediction result and the test set; if the error is within the acceptable range, perform integrated learning on the complete historical data, assign different weights and obtain the final weighted result as the prediction result.

4. The lightweight time series data anomaly detection method based on machine learning according to claim 3 is characterized in that: In step 2.2.3, if the error exceeds the acceptable range, the kernel density estimation algorithm is used to estimate the distribution of the historical data, and the estimated alarm threshold is manually set, and the estimated alarm threshold is used as the prediction result.

5. The lightweight time series data anomaly detection method based on machine learning according to claim 4 is characterized in that: Step 4 includes: Step 4.1, obtain real-time data from the online real-time stream database, aggregate it according to the preset value, input the aggregated data stream into the message queue, and execute the online real-time stream detection task; Step 4.2: Determine the abnormality of the data through an anomaly detector. If the abnormality exceeds a set value, an abnormal event is determined.

6. The lightweight time series data anomaly detection method based on machine learning according to claim 5 is characterized in that: Step 4.2 includes: Step 4.2.1, key-control and divert the real-time data so that each node has a corresponding anomaly detector; Step 4.2.2, initializing the anomaly detector; Step 4.2.3: Determine whether the data length in the anomaly detector is greater than the preset length. If so, comprehensively refer to the predicted value corresponding to the current moment, the estimated alarm threshold, the tolerance for short-term data fluctuations, and the historical moving average to determine the degree of deviation of the data, and use the degree of deviation as the abnormality of the current data; if not, use the estimated alarm threshold as the abnormality of the current data; Step 4.2.4: If the abnormality exceeds the set value, it is determined that an abnormal event has occurred.

7. The lightweight time series data anomaly detection method based on machine learning according to claim 6 is characterized in that: Also includes: Step 5: When an abnormal event occurs, determine the level of the abnormal event and automatically process abnormal events of different levels.

8. A lightweight time series data anomaly detection system based on machine learning, characterized in that: Used to perform the method according to any one of claims 1 to 7, comprising: An acquisition module is used to acquire historical data of the node to be trained as a training set in a specified library table and a specified time range of an online real-time stream database according to the meta information of the node to be trained; A prediction module, used to perform integrity check and periodicity check on the historical data, and perform periodic prediction; An integration module is used to integrate the results of periodic prediction and the meta information of the node to be trained to obtain the prediction results containing the meta information; The detection module is used to obtain real-time data from an online real-time stream database, and perform anomaly detection on the real-time data according to the prediction result containing the meta information.

9. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 7.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7.

Images