A data bias and anomaly detection method based on time series data

Abstract

The application discloses a data deviation and anomaly detection method based on time series data and belongs to the technical field of data mining and machine learning. The method comprises the following steps: performing missing value processing, anomaly value screening and marking, noise reduction and smoothing and normalization on historical normal time series data and to-be-detected time series data; identifying overall deviation based on time series independent analysis and time series correlation analysis; combining an abnormal time window and a business key time node to define a local range, and positioning local deviation through adaptive window adjustment and local special analysis. The application can detect outliers, overall deviation and local deviation, and improve the accuracy and business direction of time series data anomaly identification.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of data mining and machine learning, and specifically relates to a method for detecting biases and anomalies in time-series data. This method integrates multiple techniques such as data preprocessing, statistical analysis, and deep learning to accurately identify outliers, overall biases, and local biases in time-series data, and can be applied to scenarios that rely on time-series data for decision-making. Background Technology

[0002] Time-series data is a collection of data points collected sequentially over time, and it occupies a core position in fields such as industrial manufacturing, financial services, and the Internet of Things. For example, temperature and pressure data of equipment in industrial production lines, and stock prices and trading volumes in financial markets are all presented in time-series form. The stability and accuracy of this type of data directly determine the effectiveness of business decisions. Anomalies in production line data may indicate equipment failures, and deviations in financial transaction data may imply risky behavior. Therefore, the detection of deviations and anomalies in time-series data has significant theoretical and practical value.

[0003] While existing time-series data detection methods have made some progress, they still have many limitations. Traditional statistical methods, such as the 3σ criterion and box plots, can only identify obvious outliers and cannot capture the trend of data deviation over time; they also have poor robustness to noisy data. Machine learning-based methods, such as Isolation Forest and Support Vector Machine, have improved detection accuracy, but they mostly focus on a single type of anomaly and cannot simultaneously address the comprehensive detection needs of outliers, overall bias, and local bias.

[0004] The core pain points of current detection technologies are mainly reflected in three aspects: First, the data preprocessing stage lacks a systematic approach, with simplistic methods for handling missing values ​​and noise, failing to differentiate between outliers and normal noise in conjunction with business scenarios, leading to deviations in subsequent detection results; Second, the "temporal characteristics" of time-series data are not fully utilized, with most methods treating time-series data as ordinary structured data, ignoring the trend and periodic characteristics brought by the time dimension, and failing to effectively identify overall deviations that evolve over time; Third, the detection of local deviations lacks clear range definition methods, making it difficult to accurately locate abnormal data segments within specific time periods, thus reducing the practicality of the detection results.

[0005] To address the aforementioned issues, this invention proposes a hierarchical and progressive time-series data detection method. This method improves data quality through a standardized data preprocessing workflow, identifies overall deviations by combining a dual analysis framework of "time-independent and time-correlated" data, and then locates local deviations through a dynamic range definition strategy. This enables comprehensive and accurate detection of various anomalies and deviations, providing a more reliable technical solution for time-series data applications in various fields. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data deviation and anomaly detection method based on time series data. This method achieves full-dimensional detection of outliers, overall deviation, and local deviation through a three-level architecture of "data preprocessing - overall deviation detection - local deviation detection".

[0007] The present invention adopts the following technical solution:

[0008] Step 1: Systematic Data Preprocessing. A layered, progressive processing strategy optimizes data quality, providing a reliable data foundation for subsequent deviation and anomaly detection. The core process includes the following sub-steps.

[0009] 1.1 Missing value handling: A classification and processing strategy based on the missing rate is adopted. The appropriate solution is selected according to the different proportions of missing data, such as directly removing records with missing values, filling in the missing data with the statistical mean of adjacent data, and judging the effectiveness of data collection in combination with business scenarios.

[0010] 1.2 Preliminary Screening and Labeling of Outliers: Potential outliers are initially identified using classical statistical methods. Then, business rules and prior knowledge in the relevant domain are combined to distinguish whether the outliers are genuine anomalies. For those that are difficult to distinguish, a secondary evaluation is conducted based on the overall detection results.

[0011] 1.3 Data Denoising and Smoothing: To reduce the interference of noise on the detection results, a multi-filter comparison and selection mechanism is adopted. Several mainstream smoothing filters are selected to process the data. By comparing the noise suppression effect and feature preservation degree of each filter, the filter with the best overall performance is finally selected to complete the data smoothing.

[0012] 1.4 Data Normalization: Selective processing based on the differences in the units of measurement of data features. If the differences in the units of different features in the dataset are large, which may affect the subsequent model analysis results, standardization or normalization methods are used to map the data to a uniform interval.

[0013] Step 2: Overall Deviation Detection. This step constructs a dual analysis framework of "time-independent - time-dependent", using historical normal time-series data as a benchmark to comprehensively detect the overall deviation of the target data, specifically including the following sub-steps.

[0014] 2.1 Time-Independent Analysis: Ignoring the time dimension, this approach focuses on the static distribution characteristics of the data and verifies the consistency between the target data and the benchmark data using multi-dimensional indicators. Specifically, this includes: calculating core statistical indicators and comparing the degree of deviation; determining whether the two types of data conform to the same distribution through distribution tests; analyzing the differences in the clustering structures of the two types of data using clustering algorithms; comparing the overlap of data distributions and reconstruction errors in low-dimensional space using dimensionality reduction techniques; and training a reconstruction model based on the benchmark data and determining whether there is an overall bias based on the reconstruction error of the target data.

[0015] 2.2 Time-Series Correlation Analysis: Retaining the time dimension and fully utilizing the temporal evolution characteristics of time-series data improves the accuracy of overall deviation detection. First, a sliding window method is used to divide the time-series data into continuous time windows. The core method of time-series indiscriminate analysis is repeated for each window to achieve dynamic tracking of the overall deviation. Second, based on the trend characteristics of the time-series data, an appropriate time-series model is selected: a traditional time-series model is used for linear trend data, while a deep learning time-series model is used for non-linear trend data. The overall abnormal deviation evolving over time is identified through the model's prediction error or fitting error.

[0016] Step 3: Local Deviation Detection. This step, based on the overall deviation detection results, focuses on abnormal features within a local time range. It achieves precise localization through a "range definition - local analysis" process, and specifically includes the following sub-steps.

[0017] 3.1 Local Time Range Definition: A dual mechanism of "anomaly triggering + business association" is adopted to determine the local analysis range. On the one hand, the abnormal time window identified in the overall deviation detection and its surrounding windows are used as the initial local range; on the other hand, the time periods corresponding to key time nodes in the business scenario are included in the local analysis range. For scenarios where the range definition is unclear, an adaptive window adjustment strategy is adopted to dynamically optimize the size of the local range based on the data fluctuation characteristics.

[0018] 3.2 Local Data Specific Analysis: For the defined local time range data, an adapted preprocessing workflow is first executed, followed by a core analysis method consistent with the overall deviation detection, focusing on comparing the differences between the local data and the corresponding time period data in the benchmark data. If the difference exceeds the preset local deviation threshold, a local deviation is determined to exist within that time period.

[0019] The beneficial effects of this invention are as follows: It proposes a systematic data preprocessing process, which improves data processing quality by classifying and processing missing values ​​and distinguishing between anomalies and noise; it introduces a dual analysis framework of "time-independent and time-correlated", which takes into account both the static distribution characteristics of the data and makes full use of the dynamic evolution characteristics of time-series data; and it adopts a local range definition strategy of "dual triggering + adaptive adjustment" combined with targeted local analysis methods to make the detection results more business-oriented. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the data deviation and anomaly detection method based on time-series data of the present invention, which clearly shows the three-level architecture of "data preprocessing - overall deviation detection - local deviation detection" and the connection relationship between each step.

[0021] Figure 2This diagram illustrates the steps of the data preprocessing stage of the present invention, including the specific processes of missing value handling, outlier screening, data noise reduction, and normalization.

[0022] Figure 3 This is a schematic diagram of the dual analysis framework for overall deviation detection in this invention, showing the process of time-independent analysis and time-correlated analysis. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Without departing from the technical concept of the present invention, equivalent substitutions made by those skilled in the art for specific algorithm models, window lengths, thresholds, or application scenarios may fall within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the detection targets of this invention cover three core issues in time series data: (1) outliers, which refer to a single or a few abnormal data points that deviate significantly from the overall data distribution, usually caused by factors such as accidental interference or momentary failure of the acquisition equipment; (2) overall deviation, which refers to the systematic shift in the overall distribution characteristics or trend of the data compared to the benchmark dataset, reflecting the long-term abnormal evolution of the data; (3) local deviation, which refers to the abnormal data characteristics that are concentrated in a specific time period. These characteristics are not significant in the full time series data, but are prominent in the local time window, and are mostly related to temporary abnormal events in specific business scenarios. The overall method flow includes three progressive steps: systematic data preprocessing, overall deviation detection, and local deviation detection. The processing results of the previous step serve as the input of the next step.

[0025] Step 1: Systematic data preprocessing.

[0026] This step aims to improve data quality and lay the foundation for subsequent detection processes. It ensures the validity of the input data through stratified processing of missing values, outliers, noise, and data standardization. Figure 2 As shown, the input time series data first undergoes missing value processing and outlier filtering and marking, then enters data denoising and smoothing and data normalization processing, and finally forms the preprocessed time series data.

[0027] 1.1 Missing Value Handling: A "classification-based" strategy is adopted. For datasets with a missing value rate below 5%, data points containing missing values ​​are directly deleted. For datasets with a missing value rate between 5% and 30%, the average of the adjacent data points before and after the missing value is used for imputation. For datasets with a missing value rate above 30%, the data acquisition equipment is assessed for malfunction based on the business scenario. If the equipment is functioning normally, linear interpolation is used for imputation; if the equipment is malfunctioning, the data for that time period is marked and re-acquired. This stratified missing value handling avoids excessive interpolation when missing values ​​are few and also avoids misinterpreting equipment malfunctions as normal missing values ​​when missing values ​​are severe.

[0028] 1.2 Preliminary Screening and Labeling of Outliers: Potential outliers are initially identified using box plots, followed by the construction of a rule base based on domain knowledge. For industrial data, outliers exceeding the rated parameter range of the equipment are classified as genuine anomalies. If it is difficult to distinguish between normal noise and genuine anomalies, they are recorded separately in a temporary file and temporarily removed from the detection dataset for later back-verification using the overall detection results. The output of this step includes preliminarily cleaned data, records of potential anomalies, and records awaiting secondary verification, enabling subsequent overall detection results to back-verify the anomaly judgment made in the preprocessing stage.

[0029] 1.3 Data Denoising and Smoothing: A strategy of "visual comparison + index evaluation" was adopted to select filters. Moving average filter, Gaussian filter and Kalman filter were applied to smooth the data respectively. By comparing the evaluation index and drawing the trend graph of the smoothed data with the original data, the filter with the best noise suppression effect was selected. For high-frequency data with high fluctuation frequency, wavelet transform was used for multi-scale noise reduction to retain the key features of the data.

[0030] The moving average filter can be expressed as:

[0031]

[0032] The weights of the Gaussian filter can be expressed as:

[0033]

[0034] in, Indicates the first One original time series data point, Indicates time The corresponding smoothed output value, Indicates the moving average window length. In a Gaussian filter, the first... The weight of each sampling location, Indicates the center of the Gaussian kernel. denoted as Gaussian kernel standard deviation. Formula (1) is used to perform mean smoothing on the original data within a sliding window, and formula (2) is used to assign different smoothing weights based on the distance between the sampling position and the kernel center.

[0035] 1.4 Data Normalization: If the dimensions of different features in the dataset differ significantly, min-max normalization is used to map the data to the [0,1] interval; if the data conforms to a normal distribution, Z-score normalization is used to eliminate the influence of dimensions on subsequent models. The normalized data serves as the unified input for overall bias detection.

[0036] Filter performance evaluation metrics may include those shown in Table 1.

[0037] Table 1 Filter Performance Evaluation Indicators

[0038]

[0039] in, This indicates the number of data points participating in the evaluation. Indicates time The original data, This indicates the corresponding smoothed data. This indicates the trend change of the original data within the evaluation interval. This indicates the amount of trend change in smoothed data within the same interval. A smaller MSE indicates a smaller smoothing bias, a larger SNR indicates a better noise reduction effect, and a TPR closer to 1 indicates a stronger trend preservation ability.

[0040] Step 2: Overall deviation detection.

[0041] This step constructs a dual analysis framework of "time-independent and time-correlated" based on the benchmark dataset. Using historical normal time-series data as a reference, the target dataset... (The time series data to be tested) undergoes an overall deviation analysis, taking into account both the data's distribution and temporal characteristics. For example... Figure 3 As shown, the overall deviation detection includes two parallel or progressive analysis branches: time-independent analysis and time-correlated analysis. Its output includes the overall deviation judgment result, abnormal time window, reconstruction error, prediction error, and related index values.

[0042] 2.1 Time-independent analysis: This analytical dimension focuses on the static distribution characteristics of the data, and is verified through multiple indicators. and Consistency. First, calculate statistical indicators such as mean, variance, and skewness. Indicators and If the deviation exceeds a preset threshold, such as 20%, it is marked as a potential deviation; secondly, the KS test is used to determine... and Do they conform to the same distribution? A value less than 0.05 indicates a distribution bias; furthermore, the K-means clustering algorithm is used to... and Clustering is performed, comparing the number of clusters and the distance between cluster centers of the two classes of data. If the Euclidean distance between the cluster centers exceeds a threshold, a bias is identified. Additionally, PCA is used to reduce the dimensionality of the data. Point clouds in principal component space and If the overlap is less than 70%, or the reconstruction error exceeds a preset threshold, an overall deviation is determined to exist; finally, using If the autoencoder model is trained, The mean reconstruction error after inputting the model exceeds If the reconstruction error is three times the standard deviation, then the existence of an overall bias is confirmed.

[0043] The autoencoder reconstruction error can be expressed as:

[0044]

[0045] The overlap of distributions in the principal component space of PCA can be expressed as:

[0046]

[0047] in, Represents the input sample vector. This represents the reconstructed sample vector. Representing feature dimension, and They represent the first Dimensional input values ​​and reconstructed values, Represents the benchmark dataset The set of points in the principal component space Represents the target dataset The set of points in the principal component space This represents the overlap between the two types of point clouds. Formulas (3) and (4) can be used to transform the differences in model reconstruction and low-dimensional distribution into quantitative indicators that can be thresholded.

[0048] 2.2 Time-Series Correlation Analysis: This analysis dimension utilizes the temporal characteristics of the data to improve detection accuracy. First, a sliding window method is used to... and The data is divided into several consecutive time windows, with the window size determined by the data sampling frequency, for example, one window every 10 minutes. The method in step 2.1 is applied to each window for analysis to achieve dynamic tracking of the overall deviation. Next, a time-series-specific model is constructed. For linear trend data, the ARIMA model is trained for prediction. The theoretical value is used; if the average absolute error between the actual value and the predicted value exceeds a threshold, the deviation is marked. For nonlinear data, an LSTM model is trained, and the model is compared to the theoretical value. and The prediction error is used to identify overall anomalies that evolve over time.

[0049] The mean absolute error can be expressed as:

[0050]

[0051] in, Indicates time The actual observed value, This represents the predicted value output by the time series model. This indicates the number of data points within the window. This metric measures the degree to which the target data deviates from the theoretically predicted trajectory as it evolves over time.

[0052] Step 3: Local deviation detection.

[0053] This step, based on the overall deviation detection results, focuses on data anomalies within a specific time period and achieves precise local deviation location through "range definition-local analysis".

[0054] 3.1 Local Time Range Definition: A "dual trigger" mechanism is used to determine the local time range to be analyzed. First, when a window detected as an anomaly in the sliding window analysis of step 2, that window and one window before and after it form the initial local range. Second, key time nodes are set based on the business scenario, and specific time periods before and after these nodes are included in the local analysis range. For scenarios where range definition is difficult, an adaptive window adjustment strategy is adopted, dynamically shrinking or expanding the window range according to the data fluctuation frequency to ensure that the local range can contain complete anomaly features without introducing too much normal data. When the variance within the window is greater than the global variance, i.e. When the variance within the window is less than the global variance, the window is expanded to ensure that all abnormal features are included. The adjustment range is limited to 0.5-2.0 times the initial window to avoid over-adjustment.

[0055] The adaptive window length can be expressed as:

[0056]

[0057] in, This indicates the adjusted local window length. Indicates the initial window length. This represents the variance of the data fluctuation within the current window. This represents the variance of global time series data. Formula (6) can be used to compress the window to focus on anomalies when local fluctuations are severe, and to appropriately expand the window to cover complete anomaly segments when local fluctuations are weak.

[0058] 3.2 Local Data Specific Analysis: For the data within the defined local time range, the preprocessing procedure in step 1 is repeated, and the missing value handling, filter selection, and normalization parameters are optimized according to the characteristics of the local data. Subsequently, the statistical analysis, clustering, PCA, and autoencoder methods in step 2 are applied to focus on comparing the local data with the historical normal data of the corresponding time period, such as the same time period on the same workday. Data. If the deviation index exceeds the specific threshold of a local scenario, a local deviation is determined, and the corresponding time range, deviation type, index value, and triggering rule are output.

[0059] In one optional system implementation, the method of the present invention can be executed by a data acquisition module, a preprocessing module, an overall deviation detection module, a local deviation detection module, and a result output module. The data acquisition module is used to acquire a benchmark dataset. and target dataset The preprocessing module performs missing value handling, outlier screening and marking, noise reduction, smoothing, and normalization. The overall deviation detection module performs time-independent analysis and time-correlation analysis and outputs the overall deviation results. The local deviation detection module defines the local scope based on the abnormal time window and key business time nodes and performs local specific analysis. The result output module outputs outliers, overall deviation, local deviation, and the corresponding time range, index value, and processing suggestions. Data is passed between these modules in the order of preprocessing results, overall detection results, local scope results, and final detection results, ensuring that each step in the claims has clearly defined inputs, processing, and outputs.

[0060] In another implementation, the method of the present invention can be executed by an electronic device including a processor and a memory, the memory storing a computer program, which the processor executes to perform the steps described above. The computer program can also be stored in a computer-readable storage medium for deployment in time-series data monitoring scenarios across different industries.

Claims

1. A method for detecting data bias and anomaly based on time series data, characterized in that, Includes the following steps: S1, obtaining historical normal time series data as a reference data set , obtaining to-be-detected time series data as a target data set , performing systematic data preprocessing on the reference data set and the target data set , respectively, the systematic data preprocessing including missing value processing, preliminary screening and marking of outliers, data denoising and smoothing, and data normalization, to obtain preprocessed reference data and preprocessed target data; S2. Using the preprocessed benchmark data as a reference, perform overall deviation detection on the preprocessed target data. The overall deviation detection includes time-independent analysis and time-correlation analysis. The time-independent analysis is used to ignore the time dimension and compare the static distribution differences between the benchmark data and the target data. The time-correlation analysis is used to retain the time dimension and compare the prediction error or fitting error of the target data during the evolution of time, and output the overall deviation judgment result and the abnormal time window. S3. Perform local deviation detection based on the overall deviation judgment result and the abnormal time window. Determine the initial local time range by combining abnormal triggering and business association, and adaptively adjust the local time range according to the local data fluctuation characteristics. Perform local special analysis on the target data within the adjusted local time range and compare the difference with the historical normal data of the corresponding time period. When the difference exceeds the preset local deviation threshold, output the local deviation judgment result, deviation time range and deviation index value. 2.The method of claim 1, wherein, The missing value handling in step S1 adopts a classification processing strategy based on the missing rate: when the missing rate is less than 5%, the data points containing missing values ​​are deleted; when the missing rate is 5%-30%, the average value of the adjacent data before and after the missing data point is used to fill it; when the missing rate is greater than 30%, it is determined whether the data acquisition device is abnormal according to the business scenario. If the device is normal, linear interpolation is used to fill it; if the device is abnormal, the data in the corresponding time period is marked and re-acquired. 3.The method of claim 1, wherein, The initial screening and labeling of outliers in step S1 includes: using box plots to initially identify potential outliers; building a rule base based on domain knowledge to determine whether potential outliers are real outliers; and recording data points that cannot be distinguished as normal noise or real outliers as data to be verified in the second step, and performing reverse verification based on the overall deviation detection results in step S2. 4.The method of claim 1, wherein, The data denoising and smoothing in step S1 includes: smoothing the time-series data using a moving average filter, a Gaussian filter, and a Kalman filter, respectively, and selecting the filter with the best overall performance based on mean square error, signal-to-noise ratio, and trend preservation; wherein, the moving average filter satisfies: ； The weights of the Gaussian filter satisfy: ； wherein represents the original timing data point, represents the smoothed output value at time t, represents the moving average window length, represents the weight of the sample position in the Gaussian filter, represents the center of the Gaussian kernel, represents the standard deviation of the Gaussian kernel.

5. The data deviation and anomaly detection method based on time-series data according to claim 4, characterized in that, the mean square error , signal-to-noise ratio and trend preservation respectively ； ； ； In the formula, represents the number of data points participating in evaluation, represents the time of the original data, represents the corresponding smoothed data, represents the trend change amount of the original data in the evaluation interval, represents the trend change amount of the smoothed data in the same interval. 6.The method of claim 1, wherein, The time-independent analysis in step S2 includes: calculating the mean, variance, and skewness of the baseline data and the target data and comparing the degree of deviation; using the KS test to determine whether the baseline data and the target data conform to the same distribution; using the K-means clustering algorithm to compare the number of clusters and the distance between cluster centers of the two types of data; using PCA to compare the point cloud overlap between the target data and the baseline data in the principal component space; and training an autoencoder model based on the baseline data and judging whether there is an overall deviation based on the reconstruction error after inputting the target data into the autoencoder.

7. The data deviation and anomaly detection method based on time-series data according to claim 6, characterized in that, autoencoder reconstruction error and principal component space distribution overlap respectively satisfy: ； ； In the formula, represents an input sample vector, represents a reconstructed sample vector, represents a feature dimension, represents a point set of the reference data in the principal component space, represents a point set of the target data in the principal component space.

8. The data deviation and anomaly detection method based on time-series data according to claim 1, characterized in that, The time series correlation analysis in step S2 includes: dividing the reference data and the target data into a plurality of continuous time windows by using a sliding window method, and performing the time series independent analysis on each time window; predicting by using an ARIMA model for linear trend data, and predicting by using an LSTM model for nonlinear trend data, and calculating the average absolute error determining whether the target data has overall abnormalities evolving over time, wherein: ； In the formula, Indicates time The actual observed value, This represents the predicted value output by the time series model.

9. The data deviation and anomaly detection method based on time-series data according to claim 1, characterized in that, The adaptive adjustment of the local time range in step S3 includes: when the variance within the window... Greater than global variance At that time, narrow the local time range to focus on outlier areas; when the variance of fluctuation within the window Less than global variance At the same time, the local time range is expanded to cover complete anomaly features; the adjusted local window length satisfy: ； In the formula, This indicates the initial window length and limits the local window length to 0.5-2.0 times the initial window length.

10. A data deviation and anomaly detection system based on time-series data, characterized in that, include: The data acquisition module is used to acquire historical normal time-series data as a benchmark dataset. And obtain the time series data to be detected as the target dataset. ; A preprocessing module is used for processing the benchmark dataset. and target dataset Perform missing value handling, initial screening and labeling of outliers, data noise reduction and smoothing, and data normalization; The overall deviation detection module is used to perform time-independent analysis and time-correlated analysis and output the overall deviation judgment result and the abnormal time window; the local deviation detection module is used to define and adaptively adjust the local time range according to the overall deviation judgment result and the abnormal time window, and perform local specific analysis on the data within the local time range; the result output module is used to output outliers, overall deviation, local deviation, and the corresponding deviation time range and deviation index value.

Images