An Adaptive Denoising Method for Time Series Data Based on Residues
By employing an adaptive denoising method based on residuals, and utilizing differential, exponential smoothing, and Z-score methods to calculate the residual threshold, the problem of online denoising of time series data is solved, achieving efficient and stable denoising results, and is applicable to time series data of different dimensions.
Patent Information
- Application Number
- CN202311359667.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-18
AI Technical Summary
Existing technologies struggle to achieve online denoising of time series data, and the inability to unify hyperparameter settings for time series data with different dimensions leads to low denoising efficiency and poor stability, affecting subsequent analysis and applications.
An adaptive denoising method based on residuals is adopted, which calculates the residual threshold through differential, exponential smoothing and Z-score method to achieve adaptive denoising. Only the confidence interval and Z-score step size need to be set, which is suitable for offline and online denoising.
It achieves adaptive denoising of time series data, simplifies parameter settings, improves denoising efficiency and stability, is applicable to time series data of different dimensions, and supports online analysis and application.
Smart Images

Figure CN118013189B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and specifically to an adaptive denoising method for time-series data based on residuals. Background Technology
[0002] Denoising is a data processing technique that uses various methods to filter noise from data, allowing the data to reveal its patterns more clearly and thus maximize its value. Simultaneously, denoising is a part of data preprocessing and a fundamental task in various data application projects such as data visualization, data analysis, and algorithm modeling. The quality of denoising is crucial to the results of data applications, especially for time series data. Statistics show that data preprocessing accounts for 60% of the total workload in the entire time series data analysis process. Because raw time series data often contains outliers and high-frequency noise, failure to effectively remove noise and outliers will significantly impact subsequent data-driven operational steps such as prediction, control, and modeling.
[0003] Compared to discrete denoising methods, time-series data exhibits strong continuity in both time and numerical values, making denoising from a continuous perspective a meaningful research direction. Currently, with the increasing demand for online data analysis and algorithm modeling projects, online data denoising faces challenges. Traditional time-series denoising methods include median smoothing and local outlier detection, which require setting parameters based on the magnitude and amount of noise. These hyperparameters need to be obtained through offline observation, estimation, and statistical pre-computation, resulting in a large data processing workload and a cumbersome and complex denoising process, making online time-series denoising difficult.
[0004] Furthermore, since different data often have different dimensions, it is impossible to achieve uniform hyperparameter settings and adaptive parameter adjustments, and it is impossible to use a small number of hyperparameters to uniformly denoise all features; this is not conducive to online data cleaning and affects the progress of subsequent work. On the other hand, processing data with a single dimension in a single run results in an extremely large amount of data processing, and instability is prone to occur in multiple processing runs, leading to poor denoising results, especially in online denoising environments, where data denoising is more difficult, and denoising efficiency and stability are poor. Summary of the Invention
[0005] The present invention aims to provide a residual-based adaptive denoising method for time series data, which can achieve adaptive denoising of time series data and is applicable to different time series data. It does not require pre-calculation, can be applied to both offline and online denoising, and can provide basic support for online data analysis and application.
[0006] The basic solution provided by this invention is: an adaptive denoising method for time-series data based on residuals, comprising the following steps:
[0007] S1, for the data to be denoised, the data volatility of the data to be denoised is characterized by the magnitude of the difference between the previous and next values, and the overall noise level of the data to be denoised is evaluated by the probability distribution of the volatility.
[0008] S2, calculate the residual of the data to be denoised; use exponential smoothing in the calculation of the residual;
[0009] S3, define the residual threshold and calculate the denoising score;
[0010] S4, taking the residual sequence as the object, uses the Z-score value in the Z-score method to update the residual threshold until the denoising score in S3 is greater than or equal to the noise value calculated in S1.
[0011] The working principle and advantages of this invention are as follows:
[0012] This scheme achieves adaptive denoising of time series data through the probability distribution of residuals and iterative denoising. Notably, the entire process requires only two pre-setting parameters: a confidence interval to characterize small data fluctuations and a step size used to update the Z-score in the Z-score method. This sets the denoising intensity and initiates adaptive denoising. The parameter setting is simple, involves no hyperparameters, and only relates to the denoising intensity (a larger confidence interval indicates less noise and a weaker denoising intensity; a larger step size results in faster denoising but also larger errors). It is independent of the size of the time series data and eliminates the need for hyperparameter pre-calculation as in existing techniques, resulting in a simpler and less computationally intensive calculation process. Furthermore, the parameter settings remain consistent for time series data with different dimensions, enabling denoising for various time series data and demonstrating excellent versatility.
[0013] Furthermore, the iterative denoising method adopted in this solution (corresponding to S3) automatically iterates denoising based on the magnitude of the noise, resulting in a smaller overall data analysis volume and higher denoising efficiency. It is suitable not only for offline denoising but also for online denoising, providing basic support for online data analysis and application, and meeting the increasing demand for online services. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0015] Figure 2 This is a schematic diagram of the original time-series dataset of kiln current data, representing an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0016] Figure 3This is a difference series diagram illustrating an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0017] Figure 4 This is a probability distribution diagram of the difference sequence of an embodiment of the adaptive denoising method for time series data based on residuals according to the present invention;
[0018] Figure 5 This is a diagram illustrating the exponentially smoothed sequence of data from an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0019] Figure 6 This is a residual sequence illustration of an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention;
[0020] Figure 7 This is a probability distribution diagram of the residual sequence in an embodiment of the adaptive denoising method for time series data based on residuals according to the present invention.
[0021] Figure 8 This is a denoising score curve of an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0022] Figure 9 This is a diagram illustrating the denoising result of an embodiment of the adaptive denoising method for time-series data based on residuals according to the present invention.
[0023] Figure 10 This is a diagram showing the denoising results of the conventional Z-score denoising method. Detailed Implementation
[0024] The following detailed explanation illustrates the specific implementation methods:
[0025] The basic implementation examples are as follows: Figure 1 As shown: An adaptive denoising method for time series data based on residuals includes the following steps:
[0026] S1, for the data to be denoised, characterizes the data volatility of the data to be denoised based on the magnitude of the difference between the previous and next values, and evaluates the overall noise level of the data to be denoised based on the probability distribution of the volatility.
[0027] Specifically, let time series data D(t), t=1,2,3,...N, represent a set of N consecutive time points of the dataset to be denoised (i.e., the original dataset); and based on the dataset to be denoised D(t), obtain the difference sequence D. diff (t).
[0028] Furthermore, D diff (t)=D(t+1)-D(t)……………………Equation (1)
[0029] Where t = 1, 2, 3, ..., N-1.
[0030] Time series data exhibits strong continuity between preceding and following values, but significant noise can affect and disrupt this continuity. Here, by calculating the difference between preceding and following values, significant noise can be effectively captured.
[0031] The overall noise level is
[0032] Among them, the critical value corresponding to the 95% confidence interval is used to characterize the larger fluctuation value L. diff The critical value corresponding to the confidence interval X is used to characterize the smaller fluctuation value S. diff 10% <X<50%,0<p<1。
[0033] S2, calculate the residual of the data to be denoised; use exponential smoothing method in the calculation of the residual.
[0034] In the exponential smoothing method, the smoothing coefficient α = 1 - p; p is the overall noise level.
[0035] Furthermore, in this embodiment, the exponential smoothing formula used is:
[0036]
[0037] Among them, the smoothing coefficient can be adaptively adjusted according to the overall noise level, and can capture noise data through residuals as much as possible.
[0038] S3 defines the residual threshold and calculates the denoising score.
[0039] The noise reduction score is F score =Var S′ / Var D′ ...Equation (4)
[0040] Among them, Var S′ Let Var be the variance of the sequence S′(t). D′ Let be the variance of the sequence D′(t);
[0041] D′(t)=D(t)|-L E <E(t)<L E ,t=1,2,3,...N……………………Equation (5)
[0042] D′(t) refers to satisfying condition -L E <E(t)<L E The corresponding dataset sequence to be denoised.
[0043] S′(t)=S(t)|-L E <E(t)<L E,t=1,2,3,...N…………………………Equation (6)
[0044] S′(t) refers to the condition -L E <E(t)<L E The corresponding smooth sequence. L E This is the residual threshold.
[0045] S4, using the residual sequence as the object, updates the residual threshold using the Z-score value in the Z-score method until the denoising score in S3 is greater than or equal to the noise value calculated in S1, i.e., until F is satisfied. score ≥p.
[0046] When updating the residual threshold, the update is based on the following formula:
[0047] Where i represents the iteration number, i = 1, 2, 3, ..., i; the residual sequence is E(t). Let L be the mean of E(t), and σ be the standard deviation of E(t); E (i) represents the residual threshold in the i-th iteration, Z score (i) represents the Z score in the i-th iteration.
[0048] Furthermore, in this embodiment, the update step size of the Z score is d. z And 0 <d z <1. In practical applications, by setting different update step sizes, this method can achieve different levels of denoising effect. The larger the update step size, the faster the denoising speed, but the error is relatively larger.
[0049] Z score The interval range of (i) is (Z score_min Z score_max 。
[0050] Among them, the right-hand critical values corresponding to the 30% and 95% confidence intervals of the E(t) distribution are respectively used as the lower limit L of the residual threshold. E_min and upper limit L E_max .
[0051] Substitute them separately The upper threshold Z of the corresponding Z-score is calculated. score_max The lower limit of the Z-score threshold Z score_min .
[0052] Furthermore, in this embodiment, to accommodate low-noise data, the Z... score_max >3.0, resulting in better noise removal.
[0053] Specifically,
[0054]
[0055] Let Z score (1) = Z score_max If the interval range is not met, the iteration ends.
[0056] The Z score The update formula for (i) is: Z score (i+1)=Z score (i)-d z ...Formula (10)
[0057] Where, d z (0 <d z <1) is Z score (i) update step size.
[0058] When updating the residual threshold, the termination condition of the update iteration is that {F} is satisfied. score ≥p, Z score (i)≤Z score_min Any condition in}; and use D′(t) from the last iteration as the denoising result.
[0059] To facilitate understanding, a practical example is introduced here to illustrate this solution.
[0060] Taking the kiln current data from a cement plant's calcination system as an example, this method is applied to adaptively denoise this type of data, including the following steps:
[0061] Step 1: Initialize parameters, set the confidence interval X to 30%, and set d. z =0.1.
[0062] Step 2: Collect kiln current data for 3 days at a sampling interval of 10 minutes, for a total of 432 samples, as the data to be denoised.
[0063] Specifically, the kiln current series is [714.99963,723.92615,729.6101,723.4875,725.7572,727.77905719.1577,……,728.65643,728.6564,727.77905,731.4412,733.3867,742.77106], as shown in the figure. Figure 2 As shown.
[0064] Then, the difference sequence is obtained through difference calculation, as follows: Figure 3 As shown, it is based on the probability distribution of the difference sequence as follows: Figure 4 As shown, the critical values L corresponding to the 30% and 95% confidence intervals are obtained based on the probability distribution. diff and Sdiff The noise value p of the overall data is shown in Table 1.
[0065] Table 1 Critical values and noise levels
[0066]
[0067] Step 3: Substitute the noise value p into the exponential smoothing formula to obtain the smoothed series of kiln current.
[0068] The smoothed series of kiln current is [714.99963,716.3386,718.32935,719.1031,720.1012,721.25287,720.9386,……,720.688,721.88324,722.89923,723.63116,724.8027,726.0903,728.5924]. The smoothed series data is shown below. Figure 5 As shown, the residual sequence is obtained according to formula (4), and its trend is as follows. Figure 6 As shown.
[0069] Step 4: Calculate the mean and standard deviation of the residual series, and then based on... Figure 7 The probability distribution shown yields the right critical value L for the 30% and 95% confidence intervals. E_min L E_max Substituting these values into equations (8) and (9) respectively, we can calculate the upper limit Z of the corresponding Z-score range. score_max and lower limit Z score_min The specific results are shown in Table 2.
[0070] Table 2. Statistical data and Z-scores of the residual series.
[0071] mean Standard deviation <![CDATA[L E min ]]> <![CDATA[L E max ]]> <![CDATA[Z score max ]]> <![CDATA[Z score min ]]> -0.60 13.30 5.30 33.30 3.0 0.44
[0072] Step 5, Initialization of iteration: Let the iteration number i = 1, the sequence D′(t) = D(t) and the sequence S′(t) = S(t).
[0073] Step 6, if Z score (i)>Z score_min Then Z score (i) Substitute into equation (7) to calculate the residual threshold L E (i); Update the sequence D′(t) and the sequence S′(t) according to equations (5) and (6); otherwise, proceed to step 8.
[0074] Step 7, calculate the denoising score F according to equation (4). score If F score If ≥p, proceed to step 8; otherwise, update Z using equation (10). score(i) Repeat steps 6-7.
[0075] Step 8: Output the latest sequence D′(t) as the denoising result.
[0076] In this example, the iteration was completed 19 times, ending with a denoising score of 0.851 > 0.85.
[0077] The iterative curve of the denoising score is as follows Figure 8 As shown, the denoised data is [714.99963,723.92615,729.6101,723.4875,725.7572,727.77905,719.1577,...,693.904,749.37054,728.65643,728.65643,727.77905,731.4412,733.3867,742.77106]. The comparison results before and after denoising are as follows. Figure 9 As shown.
[0078] To better demonstrate the denoising effect of this solution, the conventional Z-score denoising method is also used to denoise the same kiln current data. The comparison results before and after denoising using the conventional method are shown below. Figure 10 As shown in the figure, this method only locates noise based on data distribution, which fails to remove noise located in the middle of the data, resulting in a low noise removal rate. It also mistakenly removes normal data located at the boundaries, leading to a high false noise rate and poor denoising accuracy. In contrast, this solution avoids these problems, effectively removing noise from various locations with a low false noise rate.
[0079] This embodiment provides a residual-based adaptive denoising method for time series data, which can achieve adaptive denoising of time series data and is applicable to different time series data. It does not require pre-calculation and can be applied to both offline and online denoising, providing basic support for online data analysis and application.
[0080] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.
Claims
1. A residual-based adaptive denoising method for time series data, characterized in that, Includes the following steps: S1, for the data to be denoised, the data volatility of the data to be denoised is characterized by the magnitude of the difference between the previous and next values, and the overall noise level of the data to be denoised is evaluated by the probability distribution of the volatility. with time series data , representing a set of continuous N time to be denoised data; and based on the to-be-denoised data set get difference sequence ; and, ; ; The overall noise size is wherein a threshold value corresponding to a confidence interval of 95% represents a large fluctuation value A threshold value corresponding to a confidence interval X represents a small fluctuation value , , ; S2, calculate the residual of the data to be denoised; use exponential smoothing to calculate the residual and obtain the smoothed sequence S(t); S3, define the residual threshold and calculate the denoising score; The denoising score is ; wherein is the variance of the sequence is the variance of the sequence is the variance of the sequence is the variance of the sequence ; ; The residual threshold; S4, using the residual sequence as the object, updates the residual threshold using the Z-score value in the Z-score method until the denoising score in S3 is met. The noise value calculated in S1.
2. The adaptive denoising method for time-series data based on residuals according to claim 1, characterized in that, In S2, the smoothing coefficient in the exponential smoothing method p represents the overall noise level.
3. The adaptive denoising method for time-series data based on residuals according to claim 1, characterized in that, In S4, the residual threshold is updated based on the following formula: ; Where i represents the iteration number. The residual sequence is , for The mean, for Standard deviation; This represents the residual threshold in the i-th iteration. Indicates the first The Z-score value in the next iteration.
4. The adaptive denoising method for time-series data based on residuals according to claim 3, characterized in that, The range of the interval is ; Among them, the right-hand critical values corresponding to the 30% and 95% confidence intervals of the E(t) distribution are used as the lower limits of the residual threshold, respectively. and upper limit Substitute into The upper threshold of the corresponding Z-score is calculated. and the lower limit of the Z-score threshold .
5. The adaptive denoising method for time-series data based on residuals according to claim 4, characterized in that, The .
6. The residual-based adaptive denoising method for time-series data according to claim 4, characterized in that, When updating the residual threshold, the termination condition of the update iteration is that it satisfies... Any condition in; and in the last iteration As a result of noise reduction.
7. The adaptive denoising method for time-series data based on residuals according to claim 4, characterized in that, In the Z-score method, the update step size of the Z-score value is: and .
Citation Information
Patent Citations
Single-index anomaly detection method based on operation and maintenance monitoring
CN111858231A
Noise Reduction Method, Noise Reduction Program, Recording Medium Having Noise Reduction Program Recorded Thereon, and Noise Reduction Apparatus
US20080002063A1
