A method for denoising against outliers based on Kalman filtering and least squares fitting
Through the anti-field value noise reduction technology combined with Kalman filtering and least squares fitting method, the error problems caused by noise and field values in engineering tests are solved, and effective signal noise reduction is achieved and signal recognizability is improved.
Patent Information
- Application Number
- CN202010738170.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-07-28
AI Technical Summary
In engineering testing, noise and field values exist in the measurement data, resulting in large errors during signal noise reduction and may cause signal distortion. It is difficult for the prior art to effectively eliminate field values and reduce noise impact.
The anti-field value noise reduction method based on Kalman filtering and least squares fitting is used to calculate the new information through Kalman filtering and distinguish the field value using the Wright criterion. The field value is corrected by combining the least squares fitting method, and the adjustment factor is designed. Reduce the effect of continuous field value fitting error.
It effectively eliminates field value interference in the signal, reduces noise influence, and improves signal recognizability, especially when processing continuous field values, it has strong anti-interference ability and improves filtering accuracy.
Smart Images

Figure CN114004248B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of signal processing, and in particular relates to an anti-outlier denoising method based on Kalman filtering and least squares fitting. Background Art
[0002] In engineering tests, due to the interference of errors in the measuring equipment itself, random errors and various environmental factors, the measured data contains noise. Some measured data will also contain data that seriously deviate from the target true value. These data that seriously deviate from the target true value are usually called "wild values". Depending on whether they are continuous, wild values can be divided into isolated wild values and continuous wild values. Isolated wild values are manifested as isolated points, while continuous wild values appear in patches, also known as spotted wild values. When denoising the measured data, wild values will bring large errors to the denoising results and even cause serious distortion of the signal. Summary of the invention
[0003] The main technical problem solved by the present invention is to provide an anti-outlier denoising method based on Kalman filtering and least squares fitting, which can effectively reduce the influence of noise in the signal while eliminating outlier interference and improve the identifiability of the signal.
[0004] In order to achieve the above object, the present invention adopts the following technical solution:
[0005] A method for reducing outliers based on Kalman filtering and least squares fitting comprises the following steps:
[0006] Step 1: Perform Kalman filter calculation on the original signal data to obtain the new information at time k:
[0007] Construct the state equation of the system to be measured:
[0008] X(k)=A·X(k-1)+B·U(k)+w(k)
[0009] Construct the observation equation for the system that needs to be measured:
[0010] Z(k)=H·X(k)+y(k)
[0011] Among them, X(k) and X(k-1) are the states of the system at time k and k-1 respectively, A and B are the system state transfer equation and control equation respectively, w(k) is the system noise with covariance Q, Z(k) is the observation value of the system at time k, H is the observation state transfer matrix, and y(k) is the observation noise with covariance R;
[0012] Constructing the prediction equations that require measuring the system:
[0013] X(k|k-1)=AX(k-1|k-1)+BU(k)
[0014] P(k|k-1)=AP(k-1|k-1)A T +Q
[0015] Among them, X(k|k-1) is the prediction of the system state at time k at time k, P is the error covariance, P(k|k-1) is the prediction of the error covariance at time k at time k, and A T is the transpose of A;
[0016] Get the new information e(k) at time k:
[0017] e(k)=Z(k)-HX(k|k-1);
[0018] Step 2: Use the Wright criterion to judge the observation value at time k in step 1; suppose the new information sequence before time k is e1, e2, e3, ..., e k , then the mean of the new information sequence before time k-1 is:
[0019]
[0020] Its mean square error is:
[0021]
[0022] If the new information at time k satisfies -3δ≤e k ≤3δ, the k-time observation value is determined to be a normal value, and step 3 is executed; if the k-time new information does not satisfy -3δ≤e k ≤3δ, the observed value at time k is determined to be an outlier, and steps 4 to 7 are executed in sequence;
[0023] Step 3: Use the k-time observation value to correct the Kalman prediction value:
[0024] Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R)
[0025] X(k|k)=X(k|k-1)+Kg(k)(Z(k)-HX(k|k-1))
[0026] P(k|k)=(I-Kg(k)H)P(k|k-1)
[0027] Among them, Kg(k) is the Kalman gain at time k, X(k|k) is the state value after filtering at time k, that is, the final filtering result, and P(k|k) is the error covariance at time k. At this point, a complete Kalman filtering is completed and the final filtering result is obtained;
[0028] Step 4: Use the least squares method to obtain the k-time fitting estimate:
[0029] Assume that there are n+1 data (x i ,y i ), where i = k-1, k-2, ···, kn-1, we can find the satisfying point (x i ,y i )’s m-th-order least squares fitting polynomial:
[0030]
[0031] To obtain the above formula, first construct a point (x i ,y i ) of the orthogonal basis functions {Q j (x)(j=0,1,···,m)}, where the polynomial Q j (x) The function degree is not greater than m, then P m (x) is expressed as:
[0032] P m (x)=q0Q0(x)+q1Q1(x)+···+q m Q m (x)
[0033] in,
[0034]
[0035] At this time, the orthogonal basis function {Q j The recursive expression of (x)(j=0,1,···,m)} is:
[0036]
[0037] in,
[0038]
[0039] Substituting the coordinates of the m consecutive known points before the k-th moment into the least square fitting polynomial, obtaining the fitting coefficient, finding the fitting polynomial, and extrapolating the fitting estimate at the k-th moment through the fitting polynomial;
[0040] Step 5: Replace the wild value with the k-time fitting estimate in step 4, and update the k-time new information:
[0041] Assume that the fitting estimate at time k is Z'(k), then the new information at time k is updated as:
[0042] e(k)=Z'(k)-HX(k|k-1);
[0043] Step 6: Design the adjustment factor λ;
[0044] Step 7: Use the adjustment factor designed in step 6 to correct the k-time new information updated in step 5, and then obtain the final filtering result.
[0045] Furthermore, in step 6, the adjustment factor λ is designed as:
[0046]
[0047] Where n is the number of consecutive outliers.
[0048] Furthermore, the specific operations of step seven are:
[0049] Use the adjustment factor λ designed in step 6 to correct the k-time new information updated in step 5 to obtain the k-time new information
[0050] e(k)=λ(Z'(k)-HX(k|k-1))
[0051] Then use the corrected new information from the above formula to obtain the final filtering result:
[0052] Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R)
[0053] X(k|k)=X(k|k-1)+λKg(k)(Z'(k)-HX(k|k-1))
[0054] P(k|k)=(I-Kg(k)H)P(k|k-1)
[0055] Among them, Kg(k) is the Kalman gain at the kth moment, X(k|k) is the filtered state value, that is, the final filtering result, and P(k|k) is the error covariance at the kth moment. At this point, a complete Kalman filtering is completed and the final filtering result is obtained.
[0056] It can be seen that as the number of consecutive outliers increases, λ gradually decreases. When outliers appear continuously, the correction effect of the extrapolated fitting estimate on the predicted value gradually decreases. Therefore, it can effectively suppress the influence of the fitting error and improve the filtering accuracy.
[0057] The beneficial effect of the present invention is as follows: the present invention provides an anti-outlier denoising method based on Kalman filtering and least squares fitting. In the process of Kalman filtering, the Wright criterion is first used to distinguish the outliers, and then the least squares fitting method is used to correct the outliers. The influence of the continuous outlier fitting error is reduced by designing an adjustment factor. The present invention has a strong anti-interference ability for continuous outliers, and can effectively reduce the influence of noise in the signal while eliminating the interference of outliers in the signal, thereby improving the identifiability of the signal. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 The present invention is a flowchart of an outlier denoising method based on Kalman filtering and least squares fitting.
[0059] Figure 2 This is a comparison chart of the original signal containing a single outlier before and after denoising.
[0060] Figure 3 This is a comparison chart of the original signal containing continuous outliers before and after denoising. DETAILED DESCRIPTION
[0061] The following examples may enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.
[0062] like Figure 1 The figure shows a flow chart of a method for reducing outliers based on Kalman filtering and least squares fitting according to the present invention. Figure 1 A method for reducing outliers based on Kalman filtering and least squares fitting is described below.
[0063] The present invention proposes a method for reducing outliers based on Kalman filtering and least squares fitting, which specifically includes the following steps:
[0064] Step 1: Perform Kalman filter calculation on the original signal data to obtain the new information of the kth sampling point, which is specifically:
[0065] The state equation for constructing the original signal data measurement system is:
[0066] X(k)=A·X(k-1)+B·U(k)+w(k)
[0067] The observation equation for constructing the original signal data measurement system is:
[0068] Z(k)=H·X(k)+y(k)
[0069] Among them, X(k) and X(k-1) are the states of the system at the kth sampling point and the k-1th sampling point respectively; A and B are the system state transfer equation and control equation respectively; w(k) is the system noise with covariance Q; Z(k) is the observation value of the system at the kth sampling point, H is the observation state transfer matrix, and y(k) is the observation noise with covariance R.
[0070] The prediction equation is constructed as:
[0071] X(k|k-1)=AX(k-1|k-1)+BU(k)
[0072] P(k|k-1)=AP(k-1|k-1)A T +Q
[0073] Where X(k|k-1) is the prediction of the system state of the k-1th sampling point to the k-th sampling point; P is the error covariance, and P(k|k-1) is the prediction of the error covariance of the k-1th sampling point to the k-th sampling point; A T is the transpose of A.
[0074] The new information e(k) of the kth sampling point is obtained as:
[0075] e(k)=Z(k)-HX(k|k-1)
[0076] Step 2: Use the Wright criterion to judge the new information of the kth sampling point to determine whether the observation value of the sampling point is an outlier, specifically:
[0077] Assume that the new information sequence before the kth sampling point is e1, e2, e3, ..., e k , then the mean of the new information sequence before the k-1th sampling point is:
[0078]
[0079] The mean square error of the new information sequence before the k-1th sampling point is obtained as:
[0080]
[0081] If the kth sampling point innovation satisfies -3δ≤e k ≤3δ, the observation value of the kth sampling point is judged to be a normal value, otherwise the observation value of the kth sampling point is judged to be an outlier.
[0082] Step 3: If the observed value of the kth sampling point in step 2 is a normal value, the Kalman prediction value is corrected using the observed value, and then the final filtering result is obtained, which is specifically:
[0083] When the observation value of the kth sampling point is a normal value, the normal value is used to correct the Kalman prediction value. The specific process is as follows:
[0084] Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R)
[0085] X(k|k)=X(k|k-1)+Kg(k)(Z(k)-HX(k|k-1))
[0086] P(k|k)=(I-Kg(k)H)P(k|k-1)
[0087] Among them, Kg(k) is the Kalman gain corresponding to the kth sampling point, X(k|k) is the state value after filtering, that is, the final filtering result, and P(k|k) is the error covariance. At this point, the Kalman filtering of the kth sampling point is completed when there is no outlier.
[0088] Step 4: If the observed value of the kth sampling point in step 2 is an outlier, the estimated value of the sampling point is fitted using the least squares method, specifically:
[0089] When the observed value of the kth sampling point is an outlier, the specific process of fitting the estimated value of the sampling point using the least squares method is as follows:
[0090] Assume that there are n+1 data (x i ,y i ), i = k-1, k-2, ···, kn-1, we can find an m-order least squares fitting polynomial:
[0091]
[0092] To obtain the above formula, first construct a point (x i ,y i ) of the orthogonal basis functions {Q j (x)(j=0,1,···,m)}; where the polynomial Q j (x) The function degree is not greater than m, then P m (x) is expressed as:
[0093] P m (x)=q0Q0(x)+q1Q1(x)+···+q m Q m (x)
[0094] in,
[0095]
[0096] At this time, the orthogonal basis function {Q j The recursive expression of (x)(j=0,1,···,m)} is:
[0097]
[0098] in,
[0099]
[0100] Substitute the coordinates of the m sampling points before the k-th sampling point into the least square fitting polynomial to obtain the fitting coefficient and the fitting polynomial, and then use the fitting polynomial to extrapolate the estimated value of the k-th sampling point.
[0101] Step 5: Use the fitted estimated value in step 4 to replace the wild value and update the new information of the kth sampling point. Specifically:
[0102] Assume that the estimated value of the k-th sampling point obtained by fitting in step 4 is Z'(k), then the new information of the k-th sampling point is updated as:
[0103] e(k)=Z'(k)-HX(k|k-1)
[0104] Step 6: Design adjustment factors, specifically:
[0105] The design adjustment factor λ is:
[0106]
[0107] Where n is the number of consecutive outliers.
[0108] Step 7: Use the adjustment factor designed in step 6 to correct the new information updated in step 5, and then obtain the final filtering result, which is:
[0109] The specific steps for using adjustment factors to correct new information are:
[0110] e(k)=λ(Z'(k)-HX(k|k-1))
[0111] Then use the corrected new information from the above formula to obtain the final filtering result:
[0112] Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R)
[0113] X(k|k)=X(k|k-1)+λKg(k)(Z'(k)-HX(k|k-1))
[0114] P(k|k)=(I-Kg(k)H)P(k|k-1)
[0115] Among them, Kg(k) is the Kalman gain at the kth moment, X(k|k) is the final filtering result of the kth sampling point under the condition of outliers, and P(k|k) is the error covariance at the kth moment. Thus, a complete Kalman filtering under the condition of outliers is completed.
[0116] Then repeat the above steps to complete the filtering process for subsequent sampling points in sequence.
[0117] As the number of consecutive outliers increases, λ gradually decreases. It can be seen that when outliers appear continuously, the correction effect of the extrapolated fitting value on the predicted value gradually decreases, which can effectively suppress the influence of the fitting error and improve the filtering accuracy.
[0118] It should be understood by those skilled in the art that the above embodiments are merely exemplary embodiments and that various changes, substitutions and alterations may be made without departing from the spirit and scope of the present invention.
Claims
1. A method for reducing outliers based on Kalman filtering and least squares fitting. It is characterized in that The steps include: Step 1: Perform Kalman filter calculation on the original signal data to obtain the new information at time k: Construct the state equation of the system to be measured: X(k)=A·X(k-1)+B·U(k)+w(k) Construct the observation equation for the system that needs to be measured: Z(k)=H·X(k)+y(k) Among them, X(k) and X(k-1) are the states of the system at time k and k-1 respectively, A and B are the system state transfer equation and control equation respectively, w(k) is the system noise with covariance Q, Z(k) is the observation value of the system at time k, H is the observation state transfer matrix, and y(k) is the observation noise with covariance R; Constructing the prediction equations that require measuring the system: X(k|k-1)=AX(k-1|k-1)+BU(k) P(k|k-1)=AP(k-1|k-1)A T +Q Among them, X(k|k-1) is the prediction of the system state at time k at time k, P is the error covariance, P(k|k-1) is the prediction of the error covariance at time k at time k, and A T is the transpose of A; Get the new information e(k) at time k: e(k)=Z(k)-HX(k|k-1); Step 2: Use the Wright criterion to judge the observation value at time k in step 1; let the new information sequence before time k be e 1 ,e 2 ,e 3 ,...,e k , then the mean of the new information sequence before time k-1 is: Its mean square error is: If the new information at time k satisfies -3δ≤e k ≤3δ, the k-time observation value is determined to be a normal value, and step 3 is executed; if the k-time new information does not satisfy -3δ≤e k ≤3δ, the k-time observation value is determined to be an outlier, and steps 4 to 7 are executed in sequence; Step 3: Use the k-time observation value to correct the Kalman prediction value: Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R) X(k|k)=X(k|k-1)+Kg(k)(Z(k)-HX(k|k-1)) P(k|k)=(I-Kg(k)H)P(k|k-1) Among them, Kg(k) is the Kalman gain at time k, X(k|k) is the state value after filtering at time k, that is, the final filtering result, and P(k|k) is the error covariance at time k. At this point, a complete Kalman filtering is completed and the final filtering result is obtained; Step 4: Use the least squares method to obtain the k-time fitting estimate: Assume that there are n+1 data (x i ,y i ), where i = k-1, k-2, ..., kn-1, we can find the satisfying point (x i ,y i )’s m-th-order least squares fitting polynomial: To obtain the above formula, first construct a point (x i ,y i ) of the orthogonal basis functions {Q j (x)(j=0,1,…,m)}, where the polynomial Q j (x) The function degree is not greater than m, then P m (x) is expressed as: P m (x)=q 0 Q 0 (x)+q 1 Q 1 (x)+…+q m Q m (x) in, At this time, the orthogonal basis function {Q j The recursive expression of (x)(j=0,1,…,m)} is: in, Substituting the coordinates of the m consecutive known points before the k-th moment into the least square fitting polynomial, obtaining the fitting coefficient, finding the fitting polynomial, and extrapolating the fitting estimate at the k-th moment through the fitting polynomial; Step 5: Replace the wild value with the k-time fitting estimate in step 4 and update the k-time new information: Assume that the fitting estimate at time k is Z'(k), then the new information at time k is updated as: e(k)=Z'(k)-HX(k|k-1); Step 6: Design the adjustment factor λ; Step 7: Use the adjustment factor designed in step 6 to correct the k-time new information updated in step 5, and then obtain the final filtering result; In step 6, the adjustment factor λ is designed as: Where n is the number of consecutive outliers.
2. The method for reducing outliers based on Kalman filtering and least squares fitting according to claim 1, It is characterized in that The specific operations of step seven are: Use the adjustment factor λ designed in step 6 to correct the k-time new information updated in step 5 to obtain the k-time new information e(k)=λ(Z'(k)-HX(k|k-1)) Then use the corrected new information from the above formula to obtain the final filtering result: Kg(k)=P(k|k-1)H T / (HP(k|k-1)H T +R) X(k|k)=X(k|k-1)+λKg(k)(Z'(k)-HX(k|k-1)) P(k|k)=(I-Kg(k)H)P(k|k-1) Among them, Kg(k) is the Kalman gain at the kth moment, X(k|k) is the filtered state value, that is, the final filtering result, and P(k|k) is the error covariance at the kth moment. At this point, a complete Kalman filtering is completed and the final filtering result is obtained; It can be seen that as the number of consecutive outliers increases, λ gradually decreases. When outliers appear continuously, the correction effect of the extrapolated fitting estimate on the predicted value gradually decreases. Therefore, it can effectively suppress the influence of the fitting error and improve the filtering accuracy.
