Video deblurring method based on dynamic weight mechanism and spatiotemporal feature enhancement

Abstract

The present invention belongs to the technical field of intelligent processing of digital images, and is specifically a video deblurring method based on a dynamic weight mechanism and spatiotemporal feature enhancement. The method of the present invention comprises: extracting features from a blurred image through a feature extraction module to obtain image features: fusing the extracted image features forward and backward in time sequence by using a UNet structure; enhancing the image features after propagation fusion by constructing a spatial feature enhancement module; reconstructing the enhanced image features through a reconstruction module; and deblurring the reconstructed image features. Experimental results show that the present invention can dynamically restore severely blurred areas, areas with rich texture details, and high-frequency parts such as object edges in the video, thereby improving the subjective visual effect.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image intelligent processing, and in particular relates to a video deblurring method. Background Art

[0002] In recent years, with the rise of the short video industry, as well as the rise of smart devices, smartphones and other devices, more and more people are shooting videos and sharing their lives. However, due to object movement, camera shake, and depth change, motion blur will occur, affecting people's visual experience. Therefore, deblurring blurred images or videos and restoring clarity can not only greatly improve people's subjective experience, but more importantly, it can better promote computer high-level visual tasks such as face recognition, target detection, and object tracking.

[0003] At present, the main reason for video blur is motion blur caused by dynamic scenes. In order to better restore blurred video frames, the restoration effect of blurred images can be improved by using adjacent frames and motion compensation. For motion compensation, it is essentially to learn the spatiotemporal features of video sequences, which contain motion clues. The existing methods can be divided into the following three main categories: the first category, by using optical flow estimation or deformation convolution to estimate the motion information in the video frame, the optical flow can be used to align the frame to be repaired with the reference frame. However, the estimation of optical flow from the blurred image input itself has the problem of inaccurate optical flow estimation, resulting in the degradation of the final model performance; the second category implicitly explores spatiotemporal information through the recurrent neural network structure, avoiding the problem of inaccurate optical flow estimation, but the recurrent neural network structure also has the problem of not being able to model long-distance temporal dependencies, and there will be information attenuation during the information transmission process. The third method uses 3D convolution to extract spatiotemporal features, but due to the limitation of the receptive field or the situation of large motion, the limited receptive field of 3D convolution cannot guarantee the same content in time sequence in one convolution. In addition, 3D convolution has high computational requirements. More importantly, existing methods all use L1 or L2 loss functions, which penalize the deblurring results with the same weight, and the model cannot better focus on repairing these areas. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a video deblurring method based on dynamic weight and spatiotemporal feature enhancement to better restore blurred video.

[0005] The video deblurring method based on dynamic weight and spatiotemporal feature enhancement provided by the present invention comprises the following specific steps:

[0006] (1) The feature extraction module is used to extract features from the blurred image and obtain image features:

[0007] (2) The extracted image features are fused by using the UNet structure for forward and backward propagation in time sequence;

[0008] (3) By constructing a spatial feature enhancement module, the image features after propagation fusion are enhanced;

[0009] (4) Reconstructing the enhanced image features through a reconstruction module;

[0010] (5) Deblur the reconstructed image features.

[0011] In step (1), the feature extraction module has a structure as follows: Figure 3 As shown in the figure, each blurred input image is sequentially subjected to convolution operation (Conv), 2 channel attention blocks (CAB, Channel Attention Block), and convolution operation (Conv) to obtain the features of the blurred image; the specific process of the channel attention block is: sequentially subjected to convolution operation (Conv), activation function (ReLU), and convolution operation (Conv), and then divided into two sub-branches; one of the sub-branches is used as the residual and is used later, and the other sub-branch sequentially undergoes: global average pooling, convolution operation (Conv), activation function (ReLU), convolution operation (Conv), activation function (ReLU), and then passes through the sigmoid function to obtain the attention map; it is point-by-point multiplied with the previous sub-branch; finally, the original features are added point-by-point to the output of the current processing to obtain the same new image features as the input.

[0012] In step (2), the use of the UNet structure includes using a forward UNet structure and a reverse UNet structure to propagate feature information between adjacent frames; the UNet structure is as follows: Figure 4 As shown in the figure, the UNet structure outputs three different scale features in the encoder and decoder parts, with the number of channels being 96, 144, and 192 respectively. At the same time, the encoder part at time t will fuse the features of the encoder and decoder of the corresponding scale at time t-1, and pass the features of the encoder and decoder at time t to time t+1, which is realized through a recurrent neural network.

[0013] In step (3), the features of each scale created by the forward and backward UNet structures in the encoder and decoder parts enter the spatiotemporal feature enhancement module respectively. The operation of the spatiotemporal feature enhancement module is described by the following formula:

[0014]

[0015]

[0016] In the formula, They represent the forward and backward features of the i-th scale of the Encoder at time t, respectively. SFB(·) represents the spatiotemporal feature enhancement network. Represents the enhanced features.

[0017] The constructed spatial feature enhancement module (SFB) is a three-branch residual network block;

[0018] The first branch uses Fourier transform to extract the real and imaginary parts of the input features. The real part of the feature is sequentially subjected to the following operations: convolution operation (Conv), activation function (ReLU), convolution operation (Conv), and then the imaginary part and the processed real part are concatenated (Concat) in the feature channel dimension. After inverse Fourier transform, the feature with the same scale as the input feature is obtained.

[0019] The second branch first passes the input features through local convolution, and then goes through convolution operation (Conv), activation function (ReLU), and convolution operation (Conv) in sequence. The result is divided into two sub-branches: one sub-branch is used as the residual for later use; the other sub-branch goes through: global average pooling, convolution operation (Conv), activation function (ReLU), convolution operation (Conv), activation function (ReLU), and then through the sigmoid function to obtain the attention map, which is multiplied point by point with the previous sub-branch; finally, the original feature is added point by point with the output of the current processing to obtain a new feature that is the same as the input;

[0020] The third branch is the original characteristics;

[0021] Finally, the results of the three branches are summed point by point to obtain the enhanced features, whose spatial dimension is the same as the input dimension.

[0022] In step (4), the output of the enhancement module is input into the reconstruction module. Figure 4 As shown, the reconstruction module uses a multi-scale fusion method, and the specific fusion method can be expressed by the formula:

[0023]

[0024]

[0025]

[0026] In the above formula, Represents the reconstruction features of different scales, i = 1, 2, 3; Up() represents the upsampling operation, Conv represents the convolution operation, Φ(B t ) represents the blurred frame B t Extracted features.

[0027] In step (5), the features reconstructed in step (4) are divided into two branches, where:

[0028] One branch is to deblur, specifically, to reconstruct the final features After the convolution operation Conv, the input blurred image B t Add them together to get the final deblurring result, as shown in formula (6).

[0029]

[0030] The other branch is the uncertainty map estimation branch, which obtains the uncertainty map δ i , as shown in formula (7), through exp(-lnδ i ) calculates the dynamic weight of each pixel during the training process, ||S i -F(B i )||1 means using the same weight, that is, using L1 loss, exp(-lnδ i )||S i -F(B i )||1 represents the dynamic penalty obtained according to the uncertainty graph. Finally, the total loss function is shown in formula (7), which uses uncertainty loss and L1 loss to jointly optimize the training of the model.

[0031]

[0032] Among them, δ i Indicates the uncertainty map obtained through the uncertainty estimation module, S i represents a blurred image, B i represents the real clear image corresponding to the blurred image, F(.) represents the deblurring network, and N is the number of blur images. The uncertainty branch can obtain the uncertainty estimation map during the training phase, which is used to calculate the loss function and train the model.

[0033] During the testing phase, the model will also output the results of the two branches. Since only the deblurred results are needed and there is no need to calculate the loss function value, the results of the uncertainty branch can be ignored.

[0034] The uncertainty map is obtained by the uncertainty estimation module. The specific operation process is as follows: the input features are sequentially subjected to Conv convolution, ELU activation function, and then 8 identical channel attention modules (CAB), wherein each channel attention module sequentially performs the following operations: the input features are first subjected to local convolution, and then sequentially subjected to convolution operation (Conv), activation function (ReLU), and convolution operation (Conv); the processed results are divided into two sub-branches; wherein one sub-branch is used as a residual and is used later, and the other sub-branch sequentially undergoes: global average pooling, convolution operation (Conv), activation function (ReLU), convolution operation (Conv), activation function (ReLU), and then the attention map is obtained through the sigmoid function, and the attention map is multiplied point by point with the previous sub-branch; finally, the original features are added point by point with the output of the current processing to obtain new features identical to the input; thus completing one process; after being processed by 8 identical modules, the result is subjected to the final convolution to output an uncertainty estimation map having the same length and width dimensions as the input and an output channel number of 3.

[0035] The present invention can dynamically restore severely blurred areas, areas with rich texture details, and high-frequency parts such as object edges in the video, thereby improving the subjective visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a network framework diagram of the present invention.

[0037] Figure 2 It is the spatiotemporal feature enhancement module of the present invention.

[0038] Figure 3 Channel attention module.

[0039] Figure 4 Feature extraction module.

[0040] Figure 5 UNet structure diagram.

[0041] Figure 6 Uncertainty estimation module.

[0042] Figure 7 Rebuild the module.

[0043] Figure 8 Input blurred image.

[0044] Fig. 9 The present invention is Figure 7 The result of deblurring the blurred image.

[0045] Fig.10 Uncertainty map estimated by the uncertainty module of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0047] use Figure 1 The network structure in is trained using the GoPro dataset, which contains 2103 pairs of blurry and clear images for training.

[0048] The specific steps are:

[0049] (1) Before training, use the lmdb tool to process the training set and test set to speed up the model's access to files;

[0050] (2) During training, 5 consecutive frames of images are input, image blocks of size 200×200 are randomly cropped, and the cropped image blocks are randomly flipped and rotated for data enhancement; the initial learning rate in the network is set to 0.0001, and the learning rate decay uses the cosine annealing scheme, which is automatically adjusted according to the set number of training epochs; the Adam stochastic gradient descent method is used to minimize the loss function; the batch size is set to 1; during the training process, the weights of the spatial feature enhancement module are first fixed, and the entire network and the uncertainty estimation module are trained separately; then, the feature extraction part of the network and the bidirectional recurrent neural network structure are frozen, the weights of the spatiotemporal feature enhancement module are unfrozen, and the uncertainty estimation module and the spatiotemporal feature enhancement module are trained again; finally, the learning rate is adjusted to 0.00001, and the entire network is fine-tuned; when optimizing the loss function, the calculation of the loss function requires the input of the two branches mentioned in step (5);

[0051] (3) During the test, the model parameters are frozen, and blurred images of five consecutive frames in the video are input into the model. According to the steps in the manual, a forward reasoning is performed. Finally, the model will output two branches, of which the deblurring branch is used as the final deblurring result. The uncertainty map estimated by the uncertainty estimation module is not used in the test phase.

[0052] Fig. 9 This is the result of repairing the blurred image invented in this article. It can be seen that the method of the present invention can achieve a good deblurring effect.

[0053] References

[0054] [1]Nah S,Hyun Kim T,Mu Lee K.Deep multi-scale convolutional neuralnetwork for dynamic scene deblurring[C] / / Proceedings of the IEEE conferenceon computer vision and pattern recognition.2017:3883-3891.

[0055] [2]Zhu C,Dong H,Pan J,et al.Deep Recurrent Neural Network with Multi-scale Bi-directional Propagation for Video Deblurring[J].arXiv preprintarXiv:2112.05150,2021.

[0056] [3]Ning Q,Dong W,Li X,et al.Uncertainty-Driven Loss for Single ImageSuper-Resolution[J].Advances in Neural Information Processing Systems,2021,34.

[0057] [4]Zhang X,Dong H,Hu Z,et al.Gated fusion network for joint imagedeblurring and super-resolution[J].arXiv preprint arXiv:1807.10806,2018.

[0058] [5]Son H,Lee J,Lee J,et al.Recurrent video deblurring with blur-invariant motion estimation and pixel volumes[J].ACM Transactions on Graphics(TOG),2021,40(5):1-18.

[0059] [6]Ren W,Pan J,Cao X,et al.Video deblurring via semantic segmentationand pixel-wise non-linear kernel[C] / / Proceedings of the IEEE InternationalConference on Computer Vision.2017:1077-1085.[7]Ning T,Li W,Li Z,et al.Dual-stream spatio-temporal decoupling network for video deblurring[J].AppliedSoft Computing,2022,116:108342.

[0060] [8]Pan J,Bai H,Tang J.Cascaded deep video deblurring using temporalsharpness prior[C] / / Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2020:3043-3051.

[0061] [9]Su S,Delbracio M,Wang J,et al.Deep video deblurring for hand-heldcameras[C] / / Proceedings of the IEEE Conference on Computer Vision and PatternRecognition.2017:1279-1288.

[0062]

[10] Wang T,Zhang X,Jiang R,et al.Video deblurring via spatiotemporalpyramid network and adversarial gradient prior[J].Computer Vision and ImageUnderstanding,2021,203:103135.

[0063]

[11] Ning T,Li W,Li Z,et al.Dual-stream spatio-temporal decouplingnetwork for video deblurring[J].Applied Soft Computing,2022,116:108342。

Claims

1. A video deblurring method based on dynamic weight mechanism and temporal and spatial feature enhancement, characterized in that: The specific steps are: (1) The feature extraction module is used to extract features from the blurred image and obtain image features: (2) The extracted image features are fused by using the UNet structure for forward and backward propagation in time sequence; (3) By constructing a spatial feature enhancement module, the image features after propagation fusion are enhanced; (4) Reconstructing the enhanced image features through a reconstruction module; (5) Deblurring the reconstructed image features; The deblurring of the reconstructed image features in step (5) is specifically performed as follows: the features reconstructed in step (4) are divided into two branches, wherein: One branch is to deblur, specifically, to reconstruct the final features After the convolution operation Conv, it is then combined with the input blurred image B t Add together to get the final deblurring result, as shown in formula (6): The other branch is the uncertainty map estimation branch, which obtains the uncertainty map δ i ; By exp(-lnδ i ) calculates the dynamic weight of each pixel during the training process, using ||S i -F(B i )||1 means using the same weight, that is, using L1 loss, with exp(-lnδ i )||S i -F(B i )||1 represents the dynamic penalty obtained according to the uncertainty graph; finally, the total loss function is as shown in formula (7), using uncertainty loss and L1 loss to jointly optimize the training of the model; Among them, δ i Indicates the uncertainty map obtained through the uncertainty estimation module, S i represents a blurred image, B i represents the real clear image corresponding to the blurred image, F(.) represents the deblurring network, and N is the number of blur images. The uncertainty branch obtains the uncertainty estimation map in the training stage, which is used to calculate the loss function and train the model.

2. The video deblurring method according to claim 1, characterized in that: In step (1), the feature extraction module is used to extract features from the blurred image to obtain image features. The specific process is as follows: each blurred frame of the input image is subjected to a convolution operation, a 2-channel attention block, and a convolution operation in sequence to obtain the features of the blurred image; The specific process of the channel attention block is as follows: it undergoes convolution operation, activation function, and convolution operation in sequence, and then is divided into two sub-branches; one of the sub-branches is used as the residual and is used later, and the other sub-branch undergoes: global average pooling, convolution operation, activation function, convolution operation, activation function, and then passes through the sigmoid function to obtain the attention map; it is multiplied point by point with the previous sub-branch; finally, the original features are added point by point with the output of the current processing to obtain the same new image features as the input.

3. The video deblurring method according to claim 2, characterized in that: In step (2), the use of the UNet structure includes using a forward UNet structure and a reverse UNet structure to propagate feature information between adjacent frames; the UNet structure outputs features of three different scales in the encoder and decoder parts, respectively, with the number of channels being 96, 144, and 192, respectively; at the same time, the encoder part at time t fuses the features of the encoder and decoder of the corresponding scale at time t-1, and passes the features of the encoder and decoder at time t to time t+1, which is achieved through a recurrent neural network.

4. The video deblurring method according to claim 3, characterized in that: In step (3), the image features after propagation fusion are enhanced by constructing a spatial feature enhancement module. The specific process is as follows: the features of each scale are created in the encoder part and the decoder part of the forward and backward UNet structures, and then enter the spatiotemporal feature enhancement module respectively. The operation of the spatiotemporal feature enhancement module is described by the following formula: In the formula, They represent the forward and backward features of the i-th scale of the Encoder at time t, respectively. SFB(·) represents the spatiotemporal feature enhancement network. Indicates the enhanced features; The spatial feature enhancement module (SFB) is a three-branch residual network block; The first branch uses Fourier transform to extract the real and imaginary parts of the input features. The real part of the feature is sequentially subjected to the following operations: convolution operation, activation function, convolution operation, and then the imaginary part and the processed real part are connected in the feature channel dimension. After inverse Fourier transform, the image feature with the same scale as the input feature is obtained. The second branch first passes the input image features through local convolution, and then passes through convolution operation, activation function, and convolution operation in sequence; the result is divided into two sub-branches: one sub-branch is used as residual for later use; the other sub-branch passes through: global average pooling, convolution operation, activation function, convolution operation, activation function, and then through the sigmoid function to obtain the attention map; it is multiplied point by point with the previous sub-branch; finally, the original feature is added point by point with the current processed output to obtain the new image feature that is the same as the input; The third branch is the original characteristics; Finally, the results of the three branches are added point by point to obtain the enhanced image features, whose spatial dimension is the same as the input dimension.

5. The video deblurring method according to claim 4, characterized in that: In step (4), the enhanced image features are reconstructed by a reconstruction module. The specific process is to input the output result of the enhancement module into the reconstruction module. The reconstruction module reconstructs the features using a multi-scale fusion method. The multi-scale fusion method is expressed by the following formulas (3)-(5): In the formula, Represents the reconstruction features of different scales, i = 1, 2, 3; Up() represents the upsampling operation, Conv represents the convolution operation, Φ(B t ) represents the blurred frame B t Extracted features.

6. The video deblurring method according to claim 1, characterized in that: The uncertainty map is obtained through the uncertainty estimation module, and the specific operation process is: the input features are sequentially subjected to convolution, activation function, and then 8 identical channel attention modules; wherein each channel attention module sequentially performs the following operations: the input features are first subjected to local convolution, and then sequentially subjected to convolution operation, activation function, and convolution operation processing; the processed results are divided into two sub-branches; wherein one sub-branch is used as a residual and is used later, and the other sub-branch sequentially undergoes: global average pooling, convolution operation, activation function, convolution operation, activation function, and then the sigmoid function to obtain the attention map; the attention map is multiplied point by point with the previous sub-branch; finally, the original features are added point by point with the output of the current processing to obtain the same new features as the input; thus completing one process; after being processed by 8 identical modules, the result is subjected to the final convolution to obtain an uncertainty estimation map with the same length and width dimensions as the input and an output channel number of 3.

Images