A spatio-temporal video denoising method based on PID control

Abstract

This invention provides a spatiotemporal video denoising method based on PID control, comprising: S1. Spatial filtering of the current frame; S2. Spatial filtering of the reference frame; S3. Calculating the spatial filtering weights of the reference frame: calculating the optimal spatial filtering weights of the reference frame using a PID adjustment method; S4. Calculating the temporal filtering weights: calculating the optimal temporal fusion weights of the reference frame using a PID adjustment method; S5. Spatiotemporal fusion to obtain the final processing result. In this method, the PID adjustment method smoothly adjusts the weighted weights of the reference frames in moving and stationary regions, making the transition between the imaging effects in moving and stationary regions smoother and the visual effect more natural. The integral control part of the PID adjustment method can effectively eliminate the problem of inaccurate motion judgment caused by noise, and can effectively avoid the phenomenon of motion blur.

Description

Technical Field

[0001] This invention belongs to the field of video denoising technology, and specifically relates to a spatiotemporal video denoising method based on PID control. Background Technology

[0002] With the development of network technology, people's demand for video is increasing, most notably in live television broadcasts and security monitoring. However, video is inevitably affected by noise during acquisition and processing, leading to a deterioration in video quality. Therefore, it is necessary to research video denoising techniques to reduce the impact of noise on video quality.

[0003] Spatial domain denoising utilizes spatial correlation for spatial filtering. Commonly used methods include mean filtering, Gaussian filtering, median filtering, and bilateral filtering.

[0004] Temporal denoising leverages the temporal correlation of video data to perform filtering along the time dimension. It is generally divided into motion-adaptive temporal filtering methods and motion-compensated temporal filtering methods. Motion-adaptive temporal filtering methods only filter stationary regions, leaving moving regions unfiltered. Therefore, motion detection algorithms are needed to distinguish between moving and stationary regions in the video. Motion-compensated temporal filtering methods require a motion estimation unit, a motion compensation unit, and a temporal filtering unit. The motion estimation unit estimates the motion vector of each pixel in the reference frame relative to the current frame. The motion compensation unit uses the motion vector to correct each pixel in the reference frame, obtaining a motion-compensated frame that perfectly matches the current frame. The temporal filtering unit performs a weighted average of the motion-compensated reference frame and the current frame to obtain the final filtered result.

[0005] However, the shortcomings of existing technologies are:

[0006] 1. Motion-adaptive temporal filtering is prone to high noise in motion regions, so noise reduction in motion regions can only rely on spatial filtering. The noise and sharpness differences between motion and static regions are large, which can easily lead to uneven effects. In addition, inaccurate motion detection can easily cause video ghosting.

[0007] 2. Motion-compensated temporal filtering is highly complex and heavily dependent on the accuracy of motion vectors. In noisy real-world scenarios, motion vectors are generally difficult to estimate accurately. When motion vectors are inaccurate, problems such as motion blur or ghosting can easily occur, or even abnormal effects such as changes in the shape of objects can appear.

[0008] In addition, the terminology commonly used in this technology includes:

[0009] 1. PID Control: The PID (Proportional-Integral-Derivative) controller is a feedback control method widely used in automatic control systems. It continuously compares the deviation between the actual system output and the desired setpoint, and generates a control signal to reduce this deviation, thereby enabling the system to reach a stable state or track a set trajectory. The design of a PID controller is based on three basic operations: proportional (P), integral (I), and derivative (D). Combining these three operations forms a powerful controller capable of simultaneously and quickly responding to errors, eliminating steady-state errors, and predicting system behavior.

[0010] 2. Temporal Denoising: A video sequence can be viewed as a one-dimensional trajectory of pixels, that is, an ordered arrangement of multiple frames on the time axis. Specifically, at each spatial location, a pixel is a one-dimensional signal along the time axis. Therefore, one-dimensional temporal denoising plays a crucial role in digital video processing technology. It utilizes the temporal correlation of image-video sequences to avoid the blurring and motion blur phenomena caused by spatial filters when processing image-video sequences. Temporal filtering typically requires storing a weighted average of the previous frame and the current frame, where the previous frame is called the reference frame and the current frame is called the current frame.

[0011] 3. Spatial domain denoising: A denoising technique for single-frame images that utilizes the spatial correlation of images for two-dimensional noise reduction.

[0012] 4. Spatiotemporal domain denoising: This method considers both temporal and spatial correlations for denoising. Because it includes two-dimensional spatial denoising and one-dimensional temporal denoising, it is also called three-dimensional spatiotemporal domain denoising or 3D denoising.

[0013] 5. Sharpness, noise, and motion blur: These are the main evaluation indicators in video denoising. Sharpness reflects the clarity of the video, noise reflects the overall noise level of the video, and motion blur reflects the strength of the afterimage around moving objects after temporal denoising. Motion blur can also reflect the strength of the side effects of temporal filtering. Summary of the Invention

[0014] In order to solve the above problems, the purpose of this application is:

[0015] 1. By using the PID control method to smoothly adjust the weighting of reference frames in the moving and stationary regions, the imaging effects in the moving and stationary regions transition more smoothly and the visual effect is more natural; 2. The integral control part of the PID control method can effectively eliminate the problem of inaccurate motion judgment caused by noise, and can effectively avoid the phenomenon of ghosting.

[0016] Specifically, the present invention provides a spatiotemporal domain video denoising method based on PID control, the method comprising the following steps:

[0017] S1. Spatial filtering of the current frame;

[0018] S2. Spatial filtering of the reference frame;

[0019] S3. Calculate the reference frame spatial filtering weights: Calculate the optimal reference frame spatial filtering weights using the PID control method.

[0020] S4. Calculate the temporal filtering weights: Calculate the optimal temporal fusion weights for the reference frame using the PID control method;

[0021] S5. Spatiotemporal domain fusion includes the following steps:

[0022] S5.1, use the results calculated in steps S2 and S3 to obtain the optimal spatial domain denoising result of the reference frame; S5.2, use the temporal domain fusion weight of the reference frame obtained in step S4 to weight and fuse the results calculated in S1 and S5.1 to obtain the final processing result.

[0023] The spatial filtering in step S1 uses the nonlocal mean denoising algorithm (NLM).

[0024] The spatial filtering in step S2 uses the nonlocal mean denoising algorithm (NLM).

[0025] In step S3, the current frame and the reference frame are subtracted to obtain the deviation signal, and then the weight is obtained using the PID algorithm; that is, the method of calculating the spatial domain denoising weight of the reference frame using PID includes the deviation calculation method and the weight calculation method.

[0026] Step S3 further includes:

[0027] S3.1 Current frame mean filtering. Mean filtering can improve the convergence speed of weights, but if the mean filtering window is too large, it can easily lead to inaccurate weight convergence results. Assuming a 3x3 filtering window is set, when the overall noise of the image is very strong, the filtering window can be appropriately increased.

[0028] S3.2 Calculate the deviation:

[0029] e(i,j,t)=c(i,j,t)-r(i,j,t)

[0030] Where t represents the frame number, i and j represent the spatial coordinates, c(·) represents the pixel value of the current frame, and r(·) represents the pixel value of the reference frame;

[0031] S3.3 PID control calculation of reference frame spatial domain filter weights:

[0032] w s (i,j,t)=1-|u s (i,j,t)|

[0033] Where w s(i,j,t) represents the spatial filtering weights of the reference frame, u s (i,j,t) represents the result of PID control, and its calculation method is as follows:

[0034]

[0035] in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. This represents the differential adjustment parameter.

[0036] In step S4, the current frame and the reference frame are subtracted to obtain the deviation signal, and then the weight is obtained using the PID algorithm; that is, step S4 uses the PID algorithm to calculate the temporal fusion weight of the reference frame, including the deviation calculation method and the weight calculation method.

[0037] In step S4, the temporal fusion weight of the reference frame is calculated using PID control, and is expressed as follows:

[0038] w p (i,j,t)=1-|u p (i,j,t)|

[0039] Where w p (i,j,t) represents the temporal filtering weights of the reference frame, u p (i,j,t) represents the result of PID control, and its calculation method is as follows:

[0040]

[0041] in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. Let represent the differential adjustment parameter, and e(i,j,t) represent the deviation calculated by S3.2.

[0042] Step S5 further includes:

[0043] S5.1, using the denoising result calculated in step S2 and the weights calculated in step S3, fuse them with the original reference frame to obtain the final spatial domain denoising result of the reference frame:

[0044]

[0045] in, Indicates the fusion result. This represents the reference frame before denoising. This represents the result after processing by the spatial domain denoising algorithm in step S2, w s (i,j,t) represents the denoising weights obtained by the PID adjustment algorithm in step S3;

[0046] S5.2, using the temporal fusion weights of the reference frame obtained in step S4, the results calculated in S1 and S5.1 are weighted and fused to obtain the final spatiotemporal domain denoising result:

[0047]

[0048] Among them, y o (i,j,t) represents the final output of the spatiotemporal domain denoising. This represents the result of the reference frame processing obtained in step S5.1. w represents the spatial denoising result of the current frame obtained after step S1 (e.g., NLM algorithm). p (i,j,t) represents the time-domain fusion weights obtained by the PID control algorithm in step S4.

[0049] Therefore, the advantage of this application is:

[0050] 1. Better processing effect: By adopting the PID adjustment method, better spatiotemporal fusion weights can be dynamically calculated, the transition between moving and still areas of the video is more natural, and the motion blur problem of moving objects is reduced. At the same time, the PID dynamically adjusts the denoising weights of the reference frame, which can achieve better spatial denoising effect.

[0051] 2. Higher algorithm robustness: The PID control method can adapt to various complex application scenarios, especially in the fields of live TV broadcasting and security monitoring, avoiding the problem of algorithm failure in special scenarios such as strong noise scenes or moving objects similar to the background.

[0052] 3. Low complexity: Compared with traditional time-domain denoising algorithms based on motion compensation, the method used in this patent can effectively avoid the problem of abnormal results caused by motion estimation failure. The algorithm has low complexity but can achieve better denoising effect. Attached Figure Description

[0053] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0054] Figure 1 This is a schematic diagram of the macroscopic framework of the spatiotemporal video denoising method based on PID control proposed in this application.

[0055] Figure 2 This is a flowchart illustrating the method.

[0056] Figure 3 This is a schematic diagram of the PID control principle in steps S3 and S4 of this application. Detailed Implementation

[0057] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0058] The present invention provides a spatiotemporal video denoising method based on PID control, the macroscopic framework of which is illustrated as follows: Figure 1 As shown, the specific method and steps are as follows: Figure 2 As shown, it includes the following steps:

[0059] S1. Spatial filtering of the current frame; non-local mean denoising algorithm (NLM) can be used.

[0060] S2. Spatial filtering of the reference frame; non-local mean denoising algorithm (NLM) can be used.

[0061] S3. Calculate the spatial filtering weights of the reference frame: Calculate the optimal spatial filtering weights of the reference frame using the PID control method; this further includes the following steps:

[0062] S3.1 Current frame mean filtering. Mean filtering can improve the convergence speed of weights, but if the mean filtering window is too large, it can easily lead to inaccurate weight convergence results. Generally, a 3x3 filtering window is sufficient. When the overall noise of the image is very strong, the filtering window can be appropriately increased.

[0063] S3.2 Calculate the deviation:

[0064] e(i,j,t)=c(i,j,t)-r(i,j,t)

[0065] Where t represents the frame number, i and j represent the spatial coordinates, c(·) represents the pixel value of the current frame, and r(·) represents the pixel value of the reference frame.

[0066] S3.3 PID control calculation of reference frame spatial domain filter weights:

[0067] w s (i,j,t)=1-|u s (i,j,t)|

[0068] Where w s (i,j,t) represents the spatial filtering weights of the reference frame, u s (i,j,t) represents the result of PID control, and its calculation method is as follows:

[0069]

[0070] in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. This represents the differential adjustment parameter.

[0071] S4. Calculate the temporal filtering weights: Calculate the optimal temporal fusion weights for the reference frames using the PID control method; further, calculate the temporal fusion weights for the reference frames using PID control:

[0072] w p (i,j,t)=1-|u p (i,j,t)|

[0073] Where w p (i,j,t) represents the temporal filtering weights of the reference frame, u p (i,j,t) represents the result of PID control, and its calculation method is as follows:

[0074]

[0075] in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. Let represent the differential adjustment parameter, and e(i,j,t) represent the deviation calculated by S3.2.

[0076] In steps S3 and S4, the deviation signal can be obtained by subtracting the current frame from the reference frame. That is, the deviation signal is obtained by directly subtracting the current frame signal from the reference frame signal, and then the weight is obtained using the PID algorithm.

[0077] The PID control principles of steps S3 and S4 are as follows: Figure 3 As shown.

[0078] S5. Spatiotemporal domain fusion to obtain the final processing result; including the following steps:

[0079] S5.1 The optimal spatial domain denoising result of the reference frame is obtained by using the results calculated in steps S2 and S3; S5.2 The result calculated in S1 is weighted and fused with the result calculated in S5.1 by using the temporal domain fusion weight of the reference frame obtained in step S4 to obtain the final processing result.

[0080] Step S5 further includes:

[0081] S5.1 The denoising result calculated in step S2 and the weights calculated in step S3 are fused with the original reference frame to obtain the final spatial domain denoising result of the reference frame:

[0082]

[0083] in, Indicates the fusion result. This represents the reference frame before denoising. This represents the result after processing by the spatial domain denoising algorithm (such as the NLM algorithm) in step S2, w s(i,j,t) represents the denoising weights obtained by the PID adjustment algorithm in step S3.

[0084] S5.2 Using the temporal fusion weights of the reference frame obtained in step S4, the results calculated in S1 and S5.1 are weighted and fused to obtain the final spatiotemporal domain denoising result:

[0085]

[0086] Among them, y o (i,j,t) represents the final output of the spatiotemporal domain denoising. This represents the result of the reference frame processing obtained in step S5.1. w represents the spatial denoising result of the current frame obtained after step S1 (e.g., NLM algorithm). p (i,j,t) represents the time-domain fusion weights obtained by the PID control algorithm in step S4.

[0087] In summary, the key technical solutions of this application include:

[0088] 1. Step S3 uses a method to calculate the spatial domain denoising weights of the reference frame using PID, including a method for calculating the deviation and a method for calculating the weights;

[0089] 2. Step S4 uses a method to calculate the temporal fusion weight of the reference frame using PID, including a method for calculating the deviation and a method for calculating the weight.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal video denoising method based on PID control, characterized in that, The method includes the following steps: S1. Spatial filtering of the current frame; S2. Spatial filtering of the reference frame; S3. Calculate the reference frame spatial filtering weights: Calculate the optimal reference frame spatial filtering weights using the PID control method. S4. Calculate the temporal filtering weights: Calculate the optimal temporal fusion weights for the reference frame using the PID control method; S5. Spatiotemporal domain fusion includes the following steps: S5.1, use the results calculated in steps S2 and S3 to obtain the optimal spatial domain denoising result of the reference frame; S5.2, use the temporal domain fusion weight of the reference frame obtained in step S4 to weight and fuse the results calculated in S1 and S5.1 to obtain the final processing result.

2. The spatiotemporal video denoising method based on PID control according to claim 1, characterized in that, The spatial filtering, i.e., spatial denoising algorithm in step S1, adopts the nonlocal mean denoising algorithm (NLM).

3. The spatiotemporal video denoising method based on PID control according to claim 1, characterized in that, The spatial filtering, i.e., spatial denoising algorithm in step S2, adopts the nonlocal mean denoising algorithm (NLM).

4. The spatiotemporal video denoising method based on PID control according to claim 1, characterized in that, In step S3, the current frame and the reference frame are subtracted to obtain the deviation signal, and then the weight is obtained using the PID algorithm; that is, the method of calculating the spatial domain denoising weight of the reference frame using PID includes the deviation calculation method and the weight calculation method.

5. A spatiotemporal video denoising method based on PID control according to claim 3, characterized in that, Step S3 further includes: S3.1 Current frame mean filtering. Mean filtering can improve the convergence speed of weights, but if the mean filtering window is too large, it can easily lead to inaccurate weight convergence results. Assuming a 3x3 filtering window is set, if the overall noise of the image is very strong, the filtering window can be appropriately increased. S3.2 Calculate the deviation: e(i,j,t)=c(i,j,t)-r(i,j,t) Where t represents the frame number, i and j represent the spatial coordinates, c(·) represents the pixel value of the current frame, and r(·) represents the pixel value of the reference frame; S3.3 PID control calculation of reference frame spatial domain filter weights: w s (i,j,t)=1- | u s (i,j,t) Where w s (i,j,t) represents the spatial filtering weights of the reference frame, u s (i,j,t) represents the result of PID control, and its calculation method is as follows: in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. This represents the differential adjustment parameter.

6. The spatiotemporal video denoising method based on PID control according to claim 1, characterized in that, In step S4, the current frame and the reference frame are subtracted to obtain the deviation signal, and then the weight is obtained using the PID algorithm. That is, step S4 uses the method of calculating the temporal fusion weight of the reference frame using PID, including the deviation calculation method and the weight calculation method.

7. A spatiotemporal video denoising method based on PID control according to claim 6, characterized in that, In step S4, the temporal fusion weight of the reference frame is calculated using PID control, and is expressed as follows: w p (i,j,t)=1-|u p (i,j,t)| Where w p (i,j,t) represents the temporal filtering weights of the reference frame, u p (i,j,t) represents the result of PID control, and its calculation method is as follows: in, This indicates the proportional adjustment parameter. Indicates the integral adjustment parameter. Let represent the differential adjustment parameter, and e(i,j,t) represent the deviation calculated by S3.

2.

8. The spatiotemporal video denoising method based on PID control according to claim 1, characterized in that, Step S5 further includes: S5.1, using the denoising result calculated in step S2 and the weights calculated in step S3, fuse them with the original reference frame to obtain the final spatial domain denoising result of the reference frame: in, Indicates the fusion result. This represents the reference frame before denoising. This represents the result after processing by the spatial domain denoising algorithm in step S2, w s (i,j,t) represents the denoising weights obtained by the PID adjustment algorithm in step S3; S5.2, using the temporal fusion weights of the reference frame obtained in step S4, the results calculated in S1 and S5.1 are weighted and fused to obtain the final spatiotemporal domain denoising result: Among them, y o (i,j,t) represents the final output of the spatiotemporal domain denoising. This represents the result of the reference frame processing obtained in step S5.

1. w represents the spatial domain denoising algorithm result of the current frame obtained after step S1. p (i,j,t) represents the time-domain fusion weights obtained by the PID control algorithm in step S4.

Images