Time sequence anomaly detection and correction method based on ensemble learning
Through the integrated learning method, a variety of evaluation indicators and algorithm selectors are combined to obtain exception points and correction values in time series data, solving the problem of low accuracy of abnormal detection in the existing technology, and realizing the use of massive data Effectively identifying abnormal points.
Patent Information
- Application Number
- CN202411502310.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-14
AI Technical Summary
In time series data, it is difficult for the prior art to accurately detect abnormal points, especially when the data type is agnostic, irregular abnormal distribution and uneven abnormal points, and different algorithms have different detection results on the same data.
Using an integrated learning method, by obtaining single-dimensional time series data, integrating multiple evaluation indicators, using an algorithm selector to filter appropriate algorithms from the exception algorithm library, and obtaining votes and weights based on integrated learning, comprehensively obtaining exception points and Correction value.
Effectively identify abnormal points in massive time series data, reduce screening of maintenance personnel, provide robust data guarantees, and improve system security.
Smart Images
Figure CN119442095A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of anomaly detection, and in particular to a time series anomaly detection and correction method based on ensemble learning. Background Art
[0002] Time series anomaly detection is a method to identify global anomalies and contextual anomalies in the time dimension. It not only needs to detect outliers but also detect anomalies that do not conform to the context rules. In the production process, sensors generate massive amounts of data over time. Anomaly detection can locate anomalies in a specific time period, which can reduce the screening work of system maintenance personnel.
[0003] A major difficulty of supervised algorithms is that the workload of anomaly labeling is large and subjective. This patent uses an unsupervised anomaly detection algorithm, which does not require manual labeling. Due to the characteristics of time series data such as unknown anomaly types, irregular anomaly distribution, and uneven anomaly points, it is difficult to accurately find anomalies. Different anomaly detection algorithms have different detection results for the same data; the effects of the same algorithm on different data sets also vary greatly. Due to the diversity of anomaly types, it is impossible to find all outliers through a single algorithm. Real-time systems have high requirements for the precision of anomaly detection methods. Using a suitable detection algorithm can avoid frequent alarm information filling up the database. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a time series anomaly detection and correction method based on ensemble learning to solve the problem that it is difficult and inaccurate to find anomalies in large-scale data.
[0005] Technical solution: The time series anomaly detection and correction method based on ensemble learning described in the present invention comprises the following steps:
[0006] (1) Obtain a single-dimensional time series Y = {Y_1, Y_2, …, Y_n}, containing discrete or continuous outlier data.
[0007] (2) Based on the time series data Y, a variety of evaluation indicators are integrated, and the algorithm selector is used to screen out suitable algorithms from the constructed anomaly algorithm library, and the algorithms are sorted according to their fitness.
[0008] (3) Obtain the ticket from the selected multiple algorithms through ensemble learning, and weight the ticket from high to low according to the degree of adaptability The combined votes yielded outliers.
[0009] (4) The correction value of the outlier point is obtained based on the votes and weights.
[0010] Furthermore, step (2) is specifically as follows: the time series data Y is tested according to three algorithms: prediction, regression, and seasonal decomposition to obtain evaluation indicators.
[0011] Furthermore, the prediction or regression algorithm evaluates the degree of abnormality by comparing the difference between the actual and predicted or regressed values, i.e., the size of the reconstruction error:
[0012]
[0013] in, is the predicted value or regression value; y groundtruth is the true value.
[0014] Furthermore, the seasonal decomposition algorithm uses the remainder to represent the score of the outlier and uses the median instead of the trend component T:
[0015] score=YS-median(Y)
[0016] Where Y is the time series data; S is the seasonal component of the data; and median(Y) is the median.
[0017] Furthermore, the evaluation indicators include:
[0018] (S1) The error with the true value, where the prediction error of the prediction algorithm, the fitting error of the regression algorithm, and the correction error of the seasonal algorithm are expressed by mean square error;
[0019] (S2) Distance metric: Calculate the distance between the modified data obtained by each algorithm. The algorithm with a larger local density has a higher score. The high-dimensional data distance metric methods used include improved cosine similarity and Pearson correlation coefficient. The local density of the algorithm, i.e., the algorithm selector score, is:
[0020]
[0021] Among them, d ij is the distance from algorithm i to j, d c is the cutoff distance.
[0022] The ranking R of M algorithms on N indicators MN , multiple ranking lists are aggregated into a comprehensive ranking list using the branch and bound method to obtain the final algorithm ranking R M ; The rank correlation coefficient is:
[0023]
[0024] Among them, a ij ={1,-1,0} represents win, loss and draw respectively. All indicators have the same weight, and the optimization objective function is
[0025] Further, step (3) is specifically as follows: including: single point anomaly check and interval anomaly check; wherein,
[0026] The specific single-point anomaly is as follows: collect the votes of all algorithms, and the inspection results obtained according to the anomaly score are 0 and 1; the formula is as follows: Let the votes of point j be T j =∑ i f(ticket ij -δ i ), where f is a binary function, expressed as δ is the threshold;
[0027] Furthermore, the specific details of interval anomaly check are as follows: Interval anomaly check refers to checking all anomalies in a period of time at a time. For the same input data, each method has the same number of votes, and votes are allocated according to the size of the anomaly score. The formula is as follows:
[0028]
[0029] Among them, Ticket is the total number of votes; Score ij is the anomaly score of detection method i at point j; ticket ij is the vote of detection method i in dimension j.
[0030] Then the weighted votes of detection method i corresponding to outlier point j are Compare the voting results and the threshold δ to obtain the outliers:
[0031]
[0032] in, is the weight of detection method i; ticket ij is the vote of detection method i at point j.
[0033] Furthermore, step (4) is as follows: the prediction algorithm and regression algorithm are replaced by the prediction value and regression value; the Savitzky-Golay smoothing filter is used for the fluctuation component in the seasonal decomposition algorithm, and the result after adding it to the seasonal component and the trend component is used as the correction value:
[0034] The final correction value obtained according to the algorithm weighted value is:
[0035]
[0036] Where, M is the number of detection algorithms; is the weight of algorithm i; is the correction value of algorithm i.
[0037] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the time series anomaly detection and correction methods based on ensemble learning is implemented.
[0038] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, any one of the time series anomaly detection and correction methods based on ensemble learning is implemented.
[0039] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it can find abnormal points in massive time series data and provide an expandable abnormality detection algorithm library. In the actual production process and system operation process, it can reduce the screening work of maintenance personnel, provide more robust data for subsequent data analysis, and better ensure the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a principle block diagram of the present invention.
[0041] Figure 2 It is a flow chart of the present invention.
[0042] Figure 3 It is a flow chart of the algorithm selector of the present invention.
[0043] Figure 4 It is a flow chart of the integrated learning of the present invention. DETAILED DESCRIPTION
[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0045] like Figure 1 As shown, the embodiment of the present invention provides a time series anomaly detection and correction method based on ensemble learning, comprising the following steps:
[0046] (1) Obtain a single-dimensional time series Y = {Y_1, Y_2, …, Y_n}, containing discrete or continuous outlier data.
[0047] (2) Based on the time series data Y, a variety of evaluation indicators are combined, and the algorithm selector is used to select suitable algorithms from the constructed abnormal algorithm library, and the algorithms are sorted according to their fitness. Specifically, the time series data Y is tested according to the three algorithms of prediction, regression, and seasonal decomposition to obtain evaluation indicators. The prediction or regression algorithm evaluates the degree of abnormality by comparing the difference between the actual and predicted or regression values, that is, the size of the reconstruction error:
[0048]
[0049] in, is the predicted value or regression value; y groundtruth is the true value.
[0050] The seasonal decomposition algorithm uses the remainder to represent the score of the outlier and uses the median instead of the trend component T:
[0051] score=YS-median(Y)
[0052] Where Y is the time series data; S is the seasonal component of the data; and median(Y) is the median.
[0053] Evaluation indicators include:
[0054] (S1) The error with the true value, where the prediction error of the prediction algorithm, the fitting error of the regression algorithm, and the correction error of the seasonal algorithm are expressed by mean square error;
[0055] (S2) Distance metric: Calculate the distance between the corrected data obtained by each algorithm. The algorithm with greater local density has a higher score. The high-dimensional data distance metric methods used include improved cosine similarity and Pearson correlation coefficient.
[0056] Among them, the improved cosine similarity formula is as follows:
[0057]
[0058] Where d(i,j) is the sequence Y i and Y j The distance y u,i ,y u,j are the values of the two sequences in dimension u; is the mean of the two series.
[0059] The Pearson correlation coefficient formula is as follows:
[0060]
[0061] Where d(i,j) is the sequence Y i and Y j distance; n is the sequence length; y i ,y j is the sequence Y i and Y j The point value on .
[0062] The local density of the algorithm, i.e., the algorithm selector score, is:
[0063]
[0064] Among them, d ijis the distance from algorithm i to j, d c is the cutoff distance.
[0065] The ranking R of M algorithms on N indicators MN , multiple ranking lists are aggregated into a comprehensive ranking list using the branch and bound method to obtain the final algorithm ranking R M ; The rank correlation coefficient is:
[0066]
[0067] Among them, a ij ={1,-1,0} represents win, loss and draw respectively. All indicators have the same weight, and the optimization objective function is
[0068] (3) Obtain the ticket from the selected multiple algorithms through ensemble learning, and weight the ticket from high to low according to the degree of adaptability The outliers are obtained by summarizing the votes. The weighted value is positively correlated with the degree of correlation. Since it is difficult to unify the evaluation criteria of different detection methods, the processing methods for single-point anomaly detection and interval anomaly detection are different.
[0069] The specific single-point anomaly is as follows: collect the votes of all algorithms, and the inspection results obtained according to the anomaly score are 0 and 1; the formula is as follows: Let the votes of point j be T j =∑ i f(ticket ij -δ i ), where f is a binary function, expressed as δ is the threshold;
[0070] Single-point anomaly detection is applicable to streaming data services.
[0071] The specific details of interval anomaly check are as follows: Interval anomaly check refers to checking all anomalies in a period of time at a time. For the same input data, each method has the same number of votes, and votes are allocated according to the size of the anomaly score. The formula is as follows:
[0072]
[0073] Among them, Ticket is the total number of votes; Score ij is the anomaly score of detection method i at point j; ticket ij is the vote of detection method i in dimension j.
[0074] Then the weighted votes of detection method i corresponding to outlier point j are Compare the voting results and the threshold δ to obtain the outliers:
[0075]
[0076] in, is the weight of detection method i; ticket ij is the vote of detection method i at point j.
[0077] Interval anomaly checking is suitable for methods with long calculation times and business needs with low real-time requirements.
[0078] (4) The correction value of the outlier is obtained based on the votes and weights. Specifically, the prediction algorithm and regression algorithm are replaced by the prediction value and regression value; the Savitzky-Golay smoothing filter is used for the volatility component in the seasonal decomposition algorithm, and the result after adding it to the seasonal component and trend component is used as the correction value:
[0079] The final correction value obtained according to the algorithm weighted value is:
[0080]
[0081] Where, M is the number of detection algorithms; is the weight of algorithm i; is the corrected value of algorithm i.
[0082] The embodiment of the present invention can be applied to subway energy consumption data based on the EMS energy management system, including the following steps:
[0083] Step 1: Subway energy consumption data is seasonal in terms of day, week, month, quarter, and year, and can be divided into five stages: morning peak, evening peak, flat peak, transition period, and nighttime station closure. Ventilation and air conditioning, traction, and lighting energy consumption account for a high proportion of subway energy consumption, so the ventilation and air conditioning energy consumption sequence of the air conditioning season is selected. The database stores meter data with a time granularity of 3 minutes. In order to reduce the impact of data fluctuations on the results, 15-minute energy consumption increment data is statistically analyzed based on this.
[0084] Step 2: Use the prediction algorithm in the algorithm library to divide the data set into a training set and a test set in a ratio of 7:3. Train the prediction model based on the newly generated incremental data at a fixed frequency before and after the system is put into operation to ensure the performance of the model. The new model will be added to the model library. If the accuracy of the newly obtained model is better than the current optimal model, the system will select and publish the model within the threshold range.
[0085] Step 3: Before the system is put into operation or during the nighttime shutdown period, the statistical energy consumption data is input into the anomaly detection algorithm library, several sets of hyperparameters are selected for the detector based on experience, and a suitable detection algorithm is selected based on the evaluation criteria of the algorithm selector.
[0086] Step 4: Continuous operation phase. Input the energy consumption data into the anomaly detection algorithm library. Single-point anomaly detection requires continuous input of the latest collected data. For the prediction algorithm, the prediction value of the current point will be given; for other algorithms, a sliding window with a data size of more than three weeks is required, and the energy consumption data at the current moment must be included. Interval anomaly detection will perform batch calculations based on newly generated data at a fixed time period or at a fixed frequency after the operation ends.
[0087] Step 5: The outlier votes of different algorithms are weighted with the relevance to obtain the voting results of ensemble learning, and the threshold δ is used to determine whether the current point is an anomaly.
[0088] Step 6: The anomaly score and weighted value are combined to give the correction value of the anomaly point.
[0089] Step 7: Find the corresponding electric meter based on the abnormal point obtained, confirm whether the electric meter is faulty, determine whether it needs to be repaired, and correct the database data.
[0090] The present invention is based on the subway energy consumption system, utilizes the rich time series data collected by the system, and uses a time series anomaly detection method based on ensemble learning to locate abnormal points from massive data, thereby ensuring the normal operation of the system and reducing manual intervention.
[0091] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, any one of the time series anomaly detection and correction methods based on ensemble learning is implemented.
[0092] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the time series anomaly detection and correction methods based on ensemble learning.
Claims
1. A time series anomaly detection and correction method based on ensemble learning, characterized in that: The following steps are involved: (1) Obtain a single-dimensional time series Y = {Y_1, Y_2, …, Y_n}, containing discrete or continuous outlier data. (2) Based on the time series data Y, a variety of evaluation indicators are integrated, and the algorithm selector is used to screen out suitable algorithms from the constructed abnormal algorithm library, and the algorithms are sorted according to their fitness; (3) Obtain the ticket from the selected multiple algorithms through ensemble learning, and weight the ticket from high to low according to the degree of adaptability The combined votes yielded anomalies; (4) The correction value of the outlier point is obtained based on the votes and weights.
2. According to the time series anomaly detection and correction method based on ensemble learning according to claim 1, it is characterized in that: Step (2) is as follows: The time series data Y is tested according to the three algorithms of prediction, regression and seasonal decomposition to obtain the evaluation index.
3. The time series anomaly detection and correction method based on ensemble learning according to claim 2 is characterized in that: The prediction or regression algorithm evaluates the degree of anomaly by comparing the difference between the actual and predicted or regressed values, that is, the size of the reconstruction error: in, is the predicted value or regression value; y groundtruth is the true value.
4. The time series anomaly detection and correction method based on ensemble learning according to claim 2 is characterized in that: The seasonal decomposition algorithm uses the remainder to represent the score of the outlier and uses the median instead of the trend component T: score=YS-median(Y) Where Y is the time series data; S is the seasonal component of the data; and median(Y) is the median.
5. The method for detecting and correcting anomalies in time series based on ensemble learning according to claim 2, characterized in that: Evaluation indicators include: (S1) The error with the true value, where the prediction error of the prediction algorithm, the fitting error of the regression algorithm, and the correction error of the seasonal algorithm are expressed by mean square error; (S2) Distance metric: Calculate the distance between the modified data obtained by each algorithm. The algorithm with a larger local density has a higher score. The high-dimensional data distance metric methods used include improved cosine similarity and Pearson correlation coefficient. The local density of the algorithm, i.e., the algorithm selector score, is: Among them, d ij is the distance from algorithm i to j, d c is the cutoff distance; The ranking R of M algorithms on N indicators MN , multiple ranking lists are aggregated into a comprehensive ranking list using the branch and bound method to obtain the final algorithm ranking R M ; The rank correlation coefficient is: Among them, a ij ={1, -1, 0} represent win, loss and draw respectively. All indicators have the same weight, and the optimization objective function is 6. The method for detecting and correcting anomalies in time series based on ensemble learning according to claim 1, characterized in that: Step (3) is as follows: including: single point anomaly check and interval anomaly check; wherein, The specific single-point anomaly is as follows: collect the votes of all algorithms, and the inspection results obtained according to the anomaly score are 0 and 1; the formula is as follows: Let the votes of point j be T j =∑ i f(ticket ij -δ i ), where f is a binary function, expressed as δ is the threshold value.
7. The method for detecting and correcting anomalies in time series based on ensemble learning according to claim 6, characterized in that: The specific details of interval anomaly check are as follows: Interval anomaly check refers to checking all anomalies in a period of time at a time. For the same input data, each method has the same number of votes, and votes are allocated according to the size of the anomaly score. The formula is as follows: Among them, Ticket is the total number of votes; Score ij is the anomaly score of detection method i at point j; ticket ij is the vote of detection method i in dimension j; Then the weighted votes of detection method i corresponding to outlier point j are Compare the voting results and the threshold δ to obtain the outliers: in, is the weight of detection method i; ticket ij is the vote of detection method i at point j.
8. The method for detecting and correcting anomalies in time series based on ensemble learning according to claim 1, characterized in that: Step (4) is as follows: the prediction algorithm and regression algorithm are replaced by the prediction value and regression value; the Savitzky-Golay smoothing filter is used for the fluctuation component in the seasonal decomposition algorithm, and the result after adding it to the seasonal component and trend component is used as the correction value: The final correction value obtained according to the algorithm weighted value is: Where, M is the number of detection algorithms; is the weight of algorithm i; is the correction value of algorithm i.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for detecting and correcting anomalies in time series based on ensemble learning according to any one of claims 1 to 8 is implemented.
10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for detecting and correcting anomalies in time series based on ensemble learning is implemented according to any one of claims 1 to 8.