Streaming data anomaly detection method

Abstract

The invention discloses an anomaly detection method for streaming data, and belongs to the field of data analysis. According to the method, each dimension of data of the multi-dimensional streaming data is subjected to independent anomaly detection. A streaming data processing mode is combined with a machine learning algorithm, a mode of combining a local density outlier detection algorithm LOF and an autoregression AR algorithm is adopted, on one hand, high consumption of server CPU and memory resources in conventional streaming data anomaly detection is made up, and on the other hand, the problem that an LOF anomaly threshold is difficult to determine is solved; the anomaly detection accuracy is improved by more than one time, and the misjudgment rate is greatly reduced. Based on the strong correlation of the time series data, the adaptive AR model is introduced, and the model fitting degree is continuously updated by using credible data, so that the data analysis capability is further improved, and the accuracy of abnormal data detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a method for detecting anomalies in streaming data. Background Art

[0002] Running equipment will periodically report a variety of status data, such as signal strength, operating current, operating temperature and other data. When performing anomaly detection on this type of data, ordinary threshold judgment methods cannot detect anomalies for trend and periodic data; the abnormal data analysis method based on statistical models has high requirements on the distribution of data. In streaming data, the data distribution is unknown in most cases; if a single LOF algorithm is used to detect anomalies on the data, the LOF anomaly threshold will be difficult to determine, which seriously affects the accuracy of anomaly detection.

[0003] Traditional threshold detection and traditional statistical methods are no longer suitable for the detection of multi-dimensional streaming data, and their misjudgment rate of data context anomalies is high; the use of historical data to obtain comprehensive coefficients and then judge the current data has a lag, and new characteristics of the data cannot be considered in a timely manner; when a single LOF algorithm is used to detect anomalies in data, the LOF anomaly threshold is difficult to select. Summary of the invention

[0004] The present invention provides a method for detecting anomalies in streaming data, which effectively overcomes the problem that the LOF anomaly threshold is difficult to determine in anomaly detection and improves the accuracy of anomaly data detection.

[0005] The present invention provides the following technical solutions:

[0006] A method for detecting anomalies in streaming data, the method comprising:

[0007] S10: pre-processing the original stream data containing the device status data, selecting a rolling counting window with a set step length to segment the pre-processed original stream data, and obtaining data streams DS0 of each window;

[0008] S20: Use the KeyBy operator of the Flink computing framework to partition the device status data contained in the data stream DS0 of the current window to obtain the isolated data stream DS1 of the current window;

[0009] S30: for the isolated data stream DS1 of the current window, obtaining the observed value of each data point of the N-th dimension data stream in the current window, and calculating the LOF value of each data point;

[0010] S40: Arrange the LOF values ​​of each data point in descending order, and take the first several data points of the time series data obtained after the descending order arrangement as suspicious points to form a suspicious set RS1;

[0011] S50: Initializing an autoregressive model, and using the initialized autoregressive model to predict each data point of the time series data in turn to obtain a predicted value of each data point;

[0012] S60: Determine whether the observed value of each data point is within the set confidence interval, if not, and if the data point is in the suspicious set RS1, determine the data point as an abnormal point, and add the data point to the abnormal set RS2;

[0013] S70: Output the exception set RS2 to the message queue.

[0014] Furthermore, the preprocessing in S10 includes: deleting data lacking a device identifier from the original stream data, and deleting redundant data at the same time;

[0015] The setting step size of the rolling counting window is determined according to the uploading period of the original stream data.

[0016] Further, the S20 includes:

[0017] Using the KeyBy operator of the Flink computing framework, the device status data contained in the data stream DS0 of the current window is partitioned according to the different geographical location information of the device, so as to obtain the isolated data stream DS1 of the current window in different geographical locations; wherein the different geographical locations include different longitudes and latitudes and different altitudes.

[0018] Furthermore, the LOF value of each data point is calculated by the following formula:

[0019]

[0020] Among them, LOF k (A) is the LOF value of data point A, k is the neighbor parameter set, LRD k (A) is the local reachability density at the kth distance of data point A, N k (A) is the set of all data points in the kth distance neighborhood of data point A, X j For the set N k The jth data point in (A) is LRD k (X j ) is the data point X j The local reachability density at the kth distance, ||N k (A)|| is the set N k The number of data points in (A);

[0021]

[0022] RD(A,X j) is the data point X j The k-th reachable distance to data point A.

[0023] Furthermore, k=20, and the values ​​of the plurality of data points are determined according to the step size of the rolling counting window.

[0024] Furthermore, the autoregressive model is initialized by a pre-prepared training data set, and the training data set includes a plurality of real data points of the N-th dimensional data stream.

[0025] Furthermore, the S60 further includes:

[0026] If the observed value of a data point is not within the set confidence interval and the data point is not in the suspicious set RS1, the credibility of the data point in the autoregressive model prediction is restored.

[0027] Furthermore, the S60 further includes:

[0028] If the observed value of the data point is within the set confidence interval, the data point is updated and added to the training data set;

[0029] The S70 further includes:

[0030] After the anomaly detection of the current window is completed, the autoregressive model is reinitialized using the updated training data set.

[0031] Furthermore, the confidence interval is determined based on the residual of the predicted value and the observed value of the data point.

[0032] The present invention has the following beneficial effects:

[0033] In the present invention, for multi-dimensional streaming data, since each dimensional data is independent of each other, a separate anomaly detection is performed on each dimensional data. Since anomaly detection is continuously performed on small batches of data in the rolling counting window during the processing of streaming data, and an anomaly detection method combining the LOF algorithm with the autoregressive model is adopted, on the one hand, it makes up for the high consumption of server CPU and memory resources in conventional streaming data anomaly detection, and on the other hand, it solves the problem that the LOF anomaly threshold is difficult to determine, thereby more than doubling the anomaly detection accuracy and greatly reducing the misjudgment rate.

[0034] The present invention combines streaming data processing with machine learning algorithms, and adopts a combination of the local density outlier detection algorithm LOF and the autoregressive AR algorithm, effectively overcoming the problem of LOF being too high and difficult to determine whether it is an outlier in anomaly detection. Based on the strong correlation of time series data, an adaptive AR model is introduced, and the model fit is continuously updated with reliable data, which further improves the data analysis capability and the accuracy of abnormal data detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method for detecting abnormalities in streaming data of the present invention;

[0036] Figure 2 is the observed value of each data point of a set of signal strength data;

[0037] Figure 3 for Figure 2 The LOF value of each data point;

[0038] Figure 4 for Figure 3 Distribution diagram of LOF values ​​of various data points shown;

[0039] Figure 5 for Figure 2 The residuals between the observed and predicted values ​​for each data point. DETAILED DESCRIPTION

[0040] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0041] The embodiment of the present invention provides a method for detecting abnormalities in streaming data. Figure 1 As shown, the method includes:

[0042] S10: Preprocessing the original stream data containing the device status data, selecting a rolling counting window WN with a set step size to segment the preprocessed original stream data, and obtaining data streams DS0 of each window.

[0043] The setting step of the rolling counting window is determined according to the upload cycle of the original stream data. When preprocessing, the original stream data is deleted from the data lacking the device identifier meid, and the redundant data at the same time is deleted.

[0044] S20: Use the KeyBy operator of the Flink computing framework to partition the device status data contained in the data stream DS0 of the current window to obtain the isolated data stream DS1 of the current window.

[0045] Exemplarily, the device status data contained in the data stream DS0 of the current window is partitioned according to different geographical location information such as different longitudes and latitudes, and different altitudes of the device, to obtain isolated data streams DS1 of the current window with different longitudes and latitudes and altitudes. Data with different longitudes and latitudes and altitudes are isolated from each other.

[0046] S30: For the isolated data stream DS1 of the current window, obtain the observed value of each data point of the N-th dimension data stream DS1_DIM_N of DS1 in the current window, and calculate the LOF value (Local Outlier Factor) of each data point.

[0047] In the present invention, for multi-dimensional streaming data, since each dimension of data is independent of each other, a separate anomaly detection is performed on each dimension of data. Specifically, the LOF value calculation process of each data point is as follows:

[0048] 1. Calculate the local reachability density of the starting data point at the kth distance in the current window, and calculate the local reachability density of all data points in the kth distance neighborhood of the starting data point;

[0049] 2. Calculate the LOF value of the starting data point in the current window.

[0050] 3. Perform the above 1 to 2 for all data points in the data stream DS1_DIM_N, and calculate the LOF value of each data point in the current window.

[0051] Among them, the local reachable density LRD at the kth distance of data point A k (A) can be calculated by the following formula:

[0052]

[0053] The LOF value of data point A can be calculated using the following formula:

[0054]

[0055] LOF k (A) is the LOF value of data point A, k is the neighbor parameter set, LRD k (A) is the local reachability density at the kth distance of data point A, N k (A) is the set of all data points in the kth distance neighborhood of data point A, X j For the set N k The jth data point in (A) is LRD k (X j ) is the data point X j The local reachability density at the kth distance, ||N k (A)|| is the set N k The number of data points in (A), RD(A,X j ) is the data point X j The kth reachable distance to data point A. To reduce the amount of computation, according to the data characteristics, the nearest neighbor parameter k = 20 can be selected.

[0056] S40: Arrange the LOF values ​​of the data points in descending order, and take the first several (for example, C) data points of the time series data obtained after the descending order as suspicious points to form a suspicious set RS1.

[0057] The value of C is related to the step size of the rolling counting window and can be adjusted according to actual conditions. Generally, 3, 5, 10, etc. can be selected.

[0058] S50: Initialize an autoregressive (AR) model, and use the initialized autoregressive model to predict each data point of the time series data in turn to obtain a predicted value of each data point.

[0059] An autoregressive model is a statistical method for processing time series. It uses the previous data of the same variable (such as X) (such as X1 to X2) to calculate the t-1 ) to predict this period (X t ) performance. In the present invention, the autoregressive model can be initialized by a pre-prepared training data set, and the training data set includes a plurality of real data points of the N-th dimension data stream DS1_DIM_N after passing through S10 to S20 of the present invention.

[0060] S60: Determine whether the observed value of each data point is within the set confidence interval. If the observed value of the data point is not within the confidence interval and if the data point is in the suspicious set RS1, the data point is determined as an abnormal point and added to the abnormal set RS2.

[0061] That is, the non-confidence interval data set generated by the AR model is intersected with the suspicious set RS1 in the LOF detection to form the final abnormal set RS2. The specific setting of the confidence interval can be adjusted according to the actual situation, for example, it can be determined according to the residual of the predicted value and the observed value of the data point.

[0062] If the observed value of a data point is not within the set confidence interval and the data point is not in the suspicious set RS1, the credibility of the data point in the autoregressive model prediction is restored.

[0063] If the observed value of a data point is within the set confidence interval, the data point is updated and added to the training data set. After all the predictions for the current window data are completed, the updated training data set can be used to re-initialize the autoregressive model and adjust its model parameters to ensure the real-time and accuracy of the AR model.

[0064] S70: Output the exception set RS2 to the message queue. Then, the process may return to S20 to perform anomaly detection on the Nth dimension data stream in the next window until all windows in the batch have completed anomaly detection.

[0065] In the present invention, for multi-dimensional streaming data, since each dimensional data is independent of each other, a separate anomaly detection is performed on each dimensional data. Since anomaly detection is continuously performed on small batches of data in the rolling counting window during the processing of streaming data, and an anomaly detection method combining the LOF algorithm with the autoregressive model is adopted, on the one hand, it makes up for the high consumption of server CPU and memory resources in conventional streaming data anomaly detection, and on the other hand, it solves the problem that the LOF anomaly threshold is difficult to determine, thereby more than doubling the anomaly detection accuracy and greatly reducing the misjudgment rate.

[0066] The present invention combines streaming data processing with machine learning algorithms, and adopts a combination of the local density outlier detection algorithm LOF and the autoregressive AR algorithm, effectively overcoming the problem of LOF being too high and difficult to determine whether it is an outlier in anomaly detection. Based on the strong correlation of time series data, an adaptive AR model is introduced, and the model fit is continuously updated with reliable data, which further improves the data analysis capability and the accuracy of abnormal data detection.

[0067] The present invention is described more vividly below with specific experimental examples:

[0068] 1. In the multi-dimensional data stream uploaded by the device, a certain dimensional data stream DS0 is selected for anomaly detection. After preprocessing operations such as window segmentation within the Flink computing framework, DS0 generates a data stream DS1. In this experimental example, the signal strength (denoted as DS1_DIM_SignalIntensity) dimensional data is used for anomaly detection. The signal strength data upload cycle is 1 minute. Considering the maximum allowable delay of anomaly detection and the limited server resources, the step size of the rolling counting window WN is set to 100, and a set of observation values ​​of each data point of the signal strength data is obtained, as shown in the following figure. Figure 2 shown.

[0069] 2. Calculate the LOF of each data point of the batch of signal strength data. The LOF of each data point is as follows: Figure 3 Its distribution diagram is shown as Figure 4 As shown. The number of suspicious set RS1 can be adjusted by yourself, and generally 3, 5, or 10 can be selected. The determination of this value is mainly determined by the step size of the rolling counting window and the data quality. According to the number of historical abnormal data, the data points corresponding to the top 5 are selected to form the suspicious set RS1.

[0070] The data points corresponding to the top 5 LOF rankings form the suspicious set RS1 as shown in the following table:

[0071] Ranking LOF Suspicious point data Data Index 1 -4.968757069029737 -109.48680997383245 80 2 -3.708150569778918 -104.77126209882567 79 3 -3.234588031222777 -102.91366290080633 46 4 -2.977136420107820 -64.42628053149312 14 5 -2.918137139974395 -101.65099929359216 40

[0072] 3. Initialize the autoregressive (AR) model. The data comes from a set of real observation values ​​prepared in advance. The set of real observation values ​​is the signal strength data processed by steps S10 to S20 of the present invention to ensure the accuracy of the obtained AR model. In this experimental example, a 10th-order AR model is used for fitting.

[0073] Then set the confidence interval, the residual between the predicted value and the observed value is as follows Figure 5 As shown, the residual conforms to the normal distribution, and the confidence interval can be selected to be within μ±3σ. In this experimental example, μ=-0.3168, σ=7.3946, and the calculated confidence interval is: [-22.5007, 21.8670].

[0074] The non-confidence interval residual data are shown in the following table:

[0075] Unconfident interval residual data Data Index -24.077130646223424 80

[0076] The non-confidence interval data set generated by the AR model is intersected with the suspicious set RS1 in the LOF detection to form the final anomaly set RS2, and the result is: (data Index) 80.

[0077] After excluding the data with Index 80 from the batch of window data, the remaining data is added to the training data set of the AR model, and the parameters of the AR model are updated.

[0078] The exception set RS2 is output to the message queue, and the anomaly detection of this batch of window data is completed.

[0079] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in streaming data, characterized in that: The method comprises: S10: pre-processing the original stream data containing the device status data, selecting a rolling counting window with a set step length to segment the pre-processed original stream data, and obtaining data streams DS0 of each window; S20: Use the KeyBy operator of the Flink computing framework to partition the device status data contained in the data stream DS0 of the current window to obtain the isolated data stream DS1 of the current window; S30: for the isolated data stream DS1 of the current window, obtaining the observed value of each data point of the N-th dimension data stream in the current window, and calculating the LOF value of each data point; S40: Arrange the LOF values ​​of the data points in descending order, and take the first several data points of the time series data obtained after the descending order arrangement as suspicious points to form a suspicious set RS1; S50: Initializing an autoregressive model, and using the initialized autoregressive model to predict each data point of the time series data in turn to obtain a predicted value of each data point; S60: Determine whether the observed value of each data point is within the set confidence interval, if not, and if the data point is in the suspicious set RS1, determine the data point as an abnormal point, and add the data point to the abnormal set RS2; S70: Output the exception set RS2 to the message queue.

2. The method for detecting anomalies in streaming data according to claim 1, characterized in that: The preprocessing in S10 includes: deleting data lacking device identification and deleting redundant data at the same time from the original stream data; The setting step size of the rolling counting window is determined according to the uploading period of the original stream data.

3. The method for detecting anomalies in streaming data according to claim 1, characterized in that: The S20 includes: Using the KeyBy operator of the Flink computing framework, the device status data contained in the data stream DS0 of the current window is partitioned according to the different geographical location information of the device, so as to obtain the isolated data stream DS1 of the current window in different geographical locations; wherein the different geographical locations include different longitudes and latitudes and different altitudes.

4. The method for detecting anomalies in streaming data according to claim 1, characterized in that: The LOF value of each data point is calculated by the following formula: Among them, LOF k (A) is the LOF value of data point A, k is the neighbor parameter set, LRD k (A) is the local reachability density at the kth distance of data point A, N k (A) is the set of all data points in the kth distance neighborhood of data point A, X j For the set N k The jth data point in (A) is LRD k (X j ) is the data point X j The local reachability density at the kth distance, ||N k (A)|| is the set N k The number of data points in (A); RD(A,X j ) is the data point X j The k-th reachable distance to data point A.

5. The method for detecting anomalies in streaming data according to claim 4, characterized in that: k=20, the values ​​of the plurality of data points are determined according to the step length of the rolling counting window.

6. The method for detecting anomalies in streaming data according to any one of claims 1 to 5, characterized in that: The autoregressive model is initialized by a pre-prepared training data set, where the training data set includes a plurality of real data points of the N-th dimensional data stream.

7. The method for detecting anomalies in streaming data according to claim 6, characterized in that: The S60 further includes: If the observed value of a data point is not within the set confidence interval and the data point is not in the suspicious set RS1, the credibility of the data point in the autoregressive model prediction is restored.

8. The method for detecting anomalies in streaming data according to claim 7, characterized in that: The S60 further includes: If the observed value of the data point is within the set confidence interval, the data point is updated and added to the training data set; The S70 further includes: After the anomaly detection of the current window is completed, the autoregressive model is reinitialized using the updated training data set.

9. The method for detecting anomalies in streaming data according to claim 8, characterized in that: The confidence interval is determined based on the residual error between the predicted and observed values ​​of the data point.

Images