Video Deblurring Method Based on Alignment Motion Features and Online Re-Degradation Learning

Abstract

The present invention belongs to the field of computer vision and relates to a video deraining method based on aligned moving features and online heavy degradation learning. The present invention first utilizes newly designed feature alignment, feature movement and feature fusion modules to form a video deraining network, so that the redundant temporal information in the video can be fully utilized. Among them, the feature alignment module utilizes multiple modules such as deformation convolution, optical flow and dilated convolution to align the features of adjacent frames of the video, which is conducive to the effective fusion of features in the subsequent stages. After the feature alignment module, the present invention creatively introduces a feature movement module, which can explicitly exchange features of different time positions, so that the features of each frame contain the features of adjacent frames, which is conducive to the integration of temporal features. Finally, the present invention uses the attention mechanism and residual structure to design a compact feature fusion module to effectively integrate spatiotemporal features and reconstruct the video frames after deraining.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a video deraining method based on aligned motion features and online heavy degradation learning. Background Art

[0002] In recent years, the demand for high-quality videos has rapidly increased. However, rain streaks inevitably affect video quality. Therefore, removing rain streaks from videos is a crucial task. In recent years, numerous algorithms specifically designed for video deraining have been proposed. In general, existing video deraining methods can be roughly categorized into the following types: model-based methods and learning-based methods.

[0003] Among the representative model-based methods, some algorithms analyze different prior knowledge of rain streaks and background in the gradient domain, or construct a sparse coding model for video deraining. However, model-based methods have limitations when manually constructing priors and can only remove stripes of certain shapes. In addition, they usually spend a lot of time performing optimization iterations. Recently, a large number of deep learning-based video deraining methods have emerged. Some algorithms use superpixel segmentation technology for alignment operations and then use additional convolutional neural networks to compensate for lost details. In addition, some algorithms propose a continuous deep expansion framework to solve the deraining model, which includes an optical flow estimation network and a multi-frame fusion network. Recently, some algorithms use a self-learning mechanism to train the network using unlabeled real rain streak data.

[0004] However, these methods still suffer from problems such as incomplete rain streak removal and severe loss of background details in the restored image. Furthermore, most of these learning-based methods utilize optical flow estimation techniques to align adjacent frames, but this often requires a significant amount of time to estimate optical flow. Recently, some video understanding algorithms have adopted a strategy of shifting features along the temporal direction to explicitly fuse temporal features. To address these issues, the present invention proposes temporal alignment, shifting, and fusion at the feature level, effectively integrating redundant temporal information in the video and achieving more effective video deraining. Summary of the Invention

[0005] This paper proposes a video deraining method based on an online re-degradation learning strategy that combines temporal feature alignment, shifting, and fusion to remove rain streaks from videos and restore a clear background in each frame. The first component of the video deraining network designed in this paper is the temporal feature alignment module, which utilizes multiple modules such as deformable convolution, optical flow, and dilated convolution to align features of adjacent video frames, thereby facilitating effective feature fusion in subsequent stages. Following the feature alignment module, this paper innovatively introduces a feature shifting module that explicitly swaps features from different temporal positions, ensuring that the features of each frame incorporate those of adjacent frames, facilitating the integration of temporal features. Finally, this paper utilizes an attention mechanism and a residual structure to design a compact feature fusion module that effectively integrates spatiotemporal features and reconstructs the derained video frames. To enhance the network's adaptability to real-world scenarios, this paper further proposes an online re-degradation learning strategy that utilizes a large number of real rainy videos without ground-truth labels to train the video deraining network designed in this paper, thereby improving its generalization capability. Finally, features that fully integrate temporal cues are passed through a reconstruction layer consisting of two convolutional layers to restore a clean background in each frame.

[0006] The technical solution adopted in the present invention is:

[0007] The video deraining method based on aligned motion features and online re-degradation learning has the following steps:

[0008] Step 1: Temporal feature alignment module based on deformable convolution, dilated convolution and optical flow.

[0009] First, the convolutional layer is used to extract features from the target frame and its adjacent frames:

[0010] F t =Conv(R t ),

[0011] F t+n =Conv(R t+n ),

[0012] Where: R t and R t+n represents the input target frame with rain marks and its adjacent frames, n takes the value of -1 or +1, representing the previous and next frames, Conv represents the convolution layer, its convolution kernel size is 3, the number of input channels is 3, the number of output channels is 64, and F t and F t+n The extracted features are 64 channels. Then the optical flow between the two frames is used to preliminarily align the adjacent frames to the target frame to obtain the preliminarily aligned adjacent frames:

[0013]

[0014] of t→t+nRepresents the estimated optical flow between two frames, and Warp represents the operation of aligning adjacent frames using optical flow. Represents adjacent frames after preliminary alignment.

[0015] Then the present invention proposes to use a dilated convolution module with multiple dilation rates to estimate the offset for deformable convolution alignment:

[0016]

[0017] DilatedConv represents the dilated convolution block proposed in this invention, whose input is the features of the adjacent frames and the target frame after preliminary alignment, and ΔP represents the estimated offset.

[0018] Unlike traditional deformable convolution alignment schemes, this paper adds the estimated offset and optical flow for deformable convolution alignment. By guiding the deformable convolution through optical flow, the instability of the deformable convolution during training is avoided:

[0019]

[0020] Where DeFormConv represents the standard deformable convolution operation, Represents the features after deformation convolution alignment.

[0021] Step 2: Timing feature movement module.

[0022] The present invention proposes using a feature shifting module to further explicitly utilize temporal information. The feature shifting module shifts the first [0:f] channels of features extracted from each video frame forward along the time axis and the last [cf:c] channels backward, where c represents the number of features extracted from a video frame. The value of c can range from a minimum of 32 to a maximum of 128. Furthermore, the present invention sets f = c / 8. Through this shifting approach, the network of the present invention can mix features of adjacent frames with features of the current frame, thereby explicitly utilizing temporal features.

[0023] Step 3: Time series feature fusion module.

[0024] To effectively fuse features containing spatiotemporal information, this paper designs a temporal feature fusion module using a residual structure and a channel attention mechanism. This module consists of a cascade of residual blocks. To effectively remove rain streaks, the number of cascaded residual blocks is at least five. Each residual block consists of two convolutional layers with activation functions and a channel attention layer. A residual connection directly adds the output of the residual block to the input to prevent the gradient of the deep network from gradually disappearing during backpropagation. The convolution kernels of the convolutional layers within the residual block are all unified to 3x3.

[0025] Step 4: Online re-degradation learning strategy.

[0026] To improve the generalization capability of the proposed video rain removal network, this paper proposes an online re-degradation learning strategy to utilize real data to assist in training the rain removal network. The paper first trains the network using labeled synthetic data until convergence. Next, the network removes rain marks from real videos to obtain rain-free videos, which serve as pseudo-labels for this data. The estimated real rain marks are then augmented with the rain-free videos to form a new training set. This training set and pseudo-labels are then used to fine-tune the rain removal network.

[0027] The present invention has excellent rain removal performance for scenes with continuous heavy rain and large movements in videos. It not only effectively removes rain streaks from every frame, but also preserves detailed information. It also enables end-to-end video rain removal and significantly accelerates inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an overall schematic diagram of the deep video deraining network based on temporal feature alignment-movement-fusion proposed in the present invention;

[0029] Figure 2 It is a temporal feature alignment module that uses deformable convolution, dilated convolution and optical flow;

[0030] Figure 3 It is a temporal feature fusion module;

[0031] Figure 4 It is a temporal feature fusion module composed of residual blocks and channel attention;

[0032] Figure 5 (a) Real video frame with rain marks in scene A;

[0033] Figure 5(b) shows the rain removal result of scenario A of the present invention;

[0034] Figure 6 (a) Real video frame with rain marks in scene B;

[0035] Figure 6(b) shows the rain removal result of scenario B of the present invention;

[0036] Figure 7 (a) Real video frame with rain marks in scene C;

[0037] Figure 7(b) shows the rain removal result of scenario C of the present invention;

[0038] Figure 8 (a) Synthesized heavy rain video frame;

[0039] Figure 8(b) shows the deraining result of synthetic heavy rain;

[0040] Figure 9(a) Synthesized light rain video frame;

[0041] Figure 9(b) shows the deraining result of synthetic light rain. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0043] The specific implementation process of the present invention is as follows Figure 1 As shown in the figure, three consecutive video frames first pass through the convolutional layer to extract features, then pass through the temporal feature alignment module, feature shift module, and feature fusion module in sequence to fully align and integrate the temporal information. Finally, two layers of convolutional layers are used to reconstruct the continuous video frames without rain.

[0044] In order to balance the deraining effect and inference speed, the present invention sets the number of extracted features c to 64 and the number of residual blocks constituting the temporal feature fusion module to 10.

[0045] The video deraining network proposed in this paper is end-to-end trainable. During training, the Adam optimizer is used to optimize the model parameters. The number of training iterations on the synthetic dataset is approximately 10,000, and the number of training iterations on the mixed real and synthetic data is approximately 5,000.

[0046] The present invention belongs to the video enhancement algorithm in the field of computer vision. The loss function consists of two parts: reconstruction loss and loss function for maintaining video temporal consistency:

[0047]

[0048] The present invention selects the L1 loss function, which is often used in various image and video enhancement algorithms, as the reconstruction loss function. As shown below:

[0049]

[0050] in represents the label corresponding to the t-th frame, Indicates the deraining result corresponding to the t-th frame. In addition, this method also uses a temporal consistency loss function Used to constrain adjacent video frames after reconstruction to keep the results continuous and smooth:

[0051]

[0052] in Indicates the mask about occlusion between the target frame and the adjacent frames that needs to be pre-calculated, L1 loss is also used. It should be mentioned that during the training phase, the network's recovery effect on each frame of the video is relatively poor. Therefore, the temporal consistency loss function Use the rain-free version of the adjacent frame as labels instead of the deraining results of neighboring frames.

Claims

1. A video deraining method based on aligned motion features and online heavy degradation learning, characterized by: Here are the steps: Step 1: Temporal feature alignment module based on deformable convolution, dilated convolution and optical flow; First, the convolutional layer is used to extract features from the target frame and its adjacent frames: F t =Conv(R t ), F t+n =Conv(R t+n ), Where: R t and R t+n Represents the input target frame with rain marks and its adjacent frames, n takes the value of -1 or +1, representing the two frames before and after, Conv represents the convolution layer, F t and F t+n Represents the extracted features; then the optical flow between the two frames is used to preliminarily align the adjacent frames to the target frame to obtain the preliminarily aligned adjacent frames: of t→t+n Represents the estimated optical flow between two frames, and Warp represents the operation of aligning adjacent frames using optical flow. Represents adjacent frames after preliminary alignment; Then, a dilated convolution module with multiple dilation rates is used to estimate the offset for deformable convolution alignment: Where DilatedConv represents the dilated convolution block, its input is the features of the adjacent frames and the target frame after preliminary alignment, and ΔP represents the estimated offset; The estimated offset and optical flow are added together for deformation convolution alignment. The deformation convolution is guided by the optical flow to avoid the occurrence of deformation convolution instability during training: Where DeFormConv represents the standard deformable convolution operation, Represents the features after deformation convolution alignment; Step 2: Time series feature moving module; We propose to use a feature shift module to further explicitly utilize temporal information. The feature shift module shifts the first [0:f] channels of the features extracted from each video frame forward and the last [cf:c] channels backward along the time axis, where c represents the number of features extracted from a video frame. Step 3: Time series feature fusion module; A temporal feature fusion module is designed using a residual structure and a channel attention mechanism. The temporal feature fusion module consists of a cascade of at least five residual blocks. Each residual block consists of two convolutional layers with activation functions and a channel attention layer. The output and input of the residual block are directly added via a residual connection to prevent the gradient of the deep network from gradually disappearing during backpropagation. The convolution kernels of the convolutional layers within the residual block are all unified to 3×3. Step 4: Online re-degradation learning strategy; To improve the generalization ability of the proposed video deraining network, an online re-degradation learning strategy is used to utilize real data to assist in training the deraining network. First, the network is trained on labeled synthetic data until convergence. Next, the network is used to remove rain marks from real videos to obtain rain-free videos, which serve as pseudo-labels for this data. The estimated real rain marks are then augmented with the rain-free videos as a new training set. This training set and pseudo-labels are then used to fine-tune the deraining network.

2. The video deraining method based on aligned motion features and online heavy degradation learning according to claim 1, characterized in that: In the step 2, the value range of c is 32 to 128.

3. The video deraining method based on aligned motion features and online heavy degradation learning according to claim 1 or 2, characterized in that: In the step 2, f=c / 8.

4. The video deraining method based on aligned motion features and online heavy degradation learning according to claim 1 or 2, characterized in that: In the step 1, the Conv layer represents a convolutional layer, the convolution kernel size is 3, the number of input channels is 3, the number of output channels is 64, and the F t and F t+n Represents the extracted features, which are 64 channels.

5. The video deraining method based on aligned motion features and online heavy degradation learning according to claim 3, characterized in that: In the step 1, the Conv layer represents a convolutional layer, the convolution kernel size is 3, the number of input channels is 3, the number of output channels is 64, and the F t and F t+n Represents the extracted features, which are 64 channels.

Images