Video deblurring method and system

Abstract

The present invention provides a video deblurring method and system, firstly, the feature extraction network is used to extract the image features in the blurred video frame sequence to obtain the first image feature; then the spatiotemporal context self-attention network is used to extract the static context information and dynamic context information in the first image feature, and the static context information and the dynamic context information are fused to obtain the second image feature; then the deep feature propagation network is used to perform optical flow and correction on the second image feature according to the optical flow value of the blurred video frame sequence to obtain the third image feature; finally, the video reconstruction network is used to reconstruct the third image feature to obtain a clear video frame sequence. The video deblurring method provided by the present invention uses the spatiotemporal context self-attention network to enable the video deblurring model to better pay attention to the global information in adjacent frames, and the deep feature propagation network can pay attention to the longer-term information in all video frames, thereby improving the effect of video deblurring.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video deblurring method and system. Background Art

[0002] With the widespread use of many high-definition shooting devices in daily life, people are in urgent need of using deblurring technology on these devices to obtain clear videos. Video deblurring methods require that multiple frames of video be input to obtain restored multiple frames of video. The video may contain unknown motion blur and brightness noise, so it has high requirements on the robustness of the hardware and algorithm of the device. Since video deblurring is a highly pathological problem, traditional methods are mainly based on the Bayesian framework and use various priors to solve this type of problem. However, this assumption based on motion blur and the degradation method of the potential frame are usually not matched. In addition, the prior constructed by this type of method usually brings very complex optimization problems, which are often difficult to solve. At present, with the development of convolutional neural networks, people usually directly stack adjacent frame videos to deal with video deblurring problems, but this does not fully pay attention to the information of the time dimension. Subsequently, methods such as 3D convolution and RNN have been proposed to solve the problem of insufficient utilization of time dimension information, but these methods cannot effectively utilize global information. Summary of the invention

[0003] The purpose of the present invention is to provide a video deblurring method and system, which can improve the effect of video deblurring.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A video deblurring method, the method comprising: performing deblurring processing on a blurry video frame sequence based on a trained video deblurring model to obtain a clear video frame sequence, the trained video deblurring model comprising a feature extraction network, a spatiotemporal context self-attention network, a deep feature propagation network and a video reconstruction network;

[0006] The specific steps of deblurring the blurred video frame sequence based on the trained video deblurring model to obtain a clear video frame sequence are as follows:

[0007] The feature extraction network is used to extract image features in the blurred video frame sequence to obtain a first image feature;

[0008] Extracting static context information and dynamic context information from the first image feature using a spatiotemporal context self-attention network, and fusing the static context information and the dynamic context information to obtain a second image feature;

[0009] According to the optical flow value of the blurred video frame sequence, the deep feature propagation network is used to perform optical flow correction on the second image feature to obtain a third image feature;

[0010] A video reconstruction network is used to reconstruct the third image feature to obtain a clear video frame sequence.

[0011] Optionally, the deblurring the blurred video frame sequence based on the trained video deblurring model to obtain a clear video frame sequence may also include:

[0012] Using the training sample set, the video deblurring model is trained using the following loss function to obtain a trained video deblurring model;

[0013]

[0014] Where loss represents the loss function, HEM(·,·) is the loss function for difficult sample mining, L is the fuzzy video frame sequence sample in the training sample set, and N k (L) represents the clear video frame sequence output by the video deblurring model, and GT is the clear video sequence sample corresponding to the blurred video frame sequence sample in the training sample set.

[0015] Optionally, the spatiotemporal contextual self-attention network includes a self-attention layer, a dynamic region-aware convolutional layer, and a feature fusion layer;

[0016] The method of extracting static context information and dynamic context information from the first image feature by using the spatiotemporal context self-attention network, and fusing the static context information and the dynamic context information to obtain the second image feature specifically includes:

[0017] Performing convolution processing on the first image feature based on the self-attention layer to obtain a query item query, a key key and a value value of the first image feature, wherein the key key is static context information;

[0018] The similarity weight matrix is ​​calculated using the following formula according to the query item query and the key key;

[0019] W=CONV3D(QK)×2,kernel size=3×1×1

[0020] Where W is the similarity weight matrix, Q is the query item, K is the key, kernel size is the three-dimensional convolution parameter, and CONV3D(*) is the three-dimensional convolution;

[0021] Based on the dynamic region-aware convolutional layer, the dynamic context information is calculated using the following formula according to the value value and the similarity weight matrix;

[0022] guidance=argmax(CONV(V)),kernelsize=1×3×3

[0023] W i =g(W) i=0···c

[0024]

[0025] Where V is the value, CONV(·) is the convolution operation, argmax(·) is the argmax function, kernelsize is the three-dimensional convolution parameter, and W is i represents the weight of the i channels, guidance is the index value, c is the channel dimension, Dynamic is the dynamic context information, g(·) is the convolution kernel generation operation, It is a standard two-dimensional convolution;

[0026] Based on the feature fusion layer, the static context information and the dynamic context information are fused using the following formula to obtain the second image feature;

[0027] weight=softmax(CONV3D(mean(Dynamic+K))×2)

[0028] out=weight*Dynamic+(1-weight)*K

[0029] Where K is the key, representing the static context information, mean(·) is the mean operation, CONV3D(·)w is a 1×1 three-dimensional convolution operation, softmax(·) is the softmax function, weight is the fusion weight matrix, and out is the second image feature.

[0030] Optionally, performing optical flow correction on the second image feature using the deep feature propagation network according to the optical flow value of the blurred video frame sequence to obtain a third image feature specifically includes:

[0031] Performing optical flow correction on initial bidirectional propagation features using the optical flow values ​​of the blurred video frame sequence, wherein the bidirectional propagation features include a first forward propagation feature and a first reverse propagation feature;

[0032] The first forward propagation feature and the first reverse propagation feature after the optical flow correction are stacked with the second image feature respectively, and the dimension is reduced to obtain a second forward propagation feature and a second reverse propagation feature;

[0033] The second forward propagation feature and the second backward propagation feature are stacked and dimensionally reduced to obtain a third image feature.

[0034] Optionally, a video reconstruction network is used to reconstruct the third image feature to obtain a clear video frame sequence, which specifically includes:

[0035] Using a video reconstruction network, the third image feature is reconstructed using the following formula to obtain a clear video frame sequence;

[0036] R=N recons (bidirectional_feature)

[0037] R is the clear video frame sequence obtained after image reconstruction, N recons (·) is the video reconstruction network, and bidirectional_feature is the third image feature.

[0038] A video deblurring system, wherein the video deblurring system is a trained video deblurring system, and the trained video deblurring system is used for deblurring a blurred video frame sequence to obtain a clear video frame sequence;

[0039] The video deblurring system comprises a feature extraction unit, a spatiotemporal contextual self-attention unit, a deep feature propagation unit and a video reconstruction unit;

[0040] The feature extraction unit is used for extracting the image feature in the blurred video frame sequence to obtain the first image feature;

[0041] The spatiotemporal context self-attention unit is used to extract static context information and dynamic context information from the first image feature, and fuse the static context information and the dynamic context information to obtain a second image feature;

[0042] The deep feature propagation unit is used to perform optical flow correction on the second image feature according to the optical flow value of the blurred video frame sequence to obtain a third image feature;

[0043] The video reconstruction unit is used to perform image reconstruction according to the third image feature to obtain a clear video frame sequence.

[0044] Optionally, the video deblurring system further includes a training module, and the training module is used to train the video deblurring system using a training sample set, a regression loss function and a difficult sample mining loss function.

[0045] Optionally, the spatiotemporal context self-attention unit includes a self-attention module, a dynamic region-aware convolution module and a feature fusion module;

[0046] The self-attention module is used to perform convolution processing on the first image feature to obtain a query item query, a key key and a value value of the first image feature, wherein the key key is static context information, and a similarity weight matrix W is calculated according to the query item query and the key key using the following formula;

[0047] W=CONV3D(QK)×2,kernel size=3×1×1

[0048] Where W is the similarity weight matrix, Q is the query item, K is the key, kernel size = 3 × 1 × 1 is the three-dimensional convolution parameter, and CONV3D(·) is the three-dimensional convolution;

[0049] The dynamic region-aware convolution module is used to calculate the dynamic context information according to the value value and the similarity weight matrix W using the following formula;

[0050] guidance=argmax(CONV(V)),kernelsize=1×3×3

[0051] W i =g(W) i=0···c

[0052]

[0053] Where V is the value, CONV(·) is the convolution operation, argmax(·) is the argmax function, kernelsize is the three-dimensional convolution parameter, and W is i represents the weight of the i channels, guidance is the obtained index value, c is the channel dimension, Dynamic is the dynamic context information, g(·) is the convolution kernel generation operation, It is a standard two-dimensional convolution;

[0054] The feature fusion module is used to obtain the second image feature by fusing the static context information and the dynamic context information;

[0055] weight=softmax(CONV3D(mean(Dynamic+K))×2)

[0056] out=weight*Dynamic+(1-weight)*K

[0057] Where K is the key, representing the static context information, mean(·) is the mean operation, CONV3D(·) is a 1×1 three-dimensional convolution operation, softmax(·) is the softmax function, weight is the fusion weight matrix, and out is the second image feature.

[0058] Optionally, the deep feature propagation unit includes an optical flow correction module, a feature update module and a feature fusion module;

[0059] The optical flow correction module is used to perform optical flow correction on the initial two-way propagation feature according to the optical flow value of the blurred video frame sequence, and the two-way propagation feature includes a first forward propagation feature and a first reverse propagation feature;

[0060] The feature updating module is used to stack the first forward propagation feature and the first reverse propagation feature after the optical flow correction with the second image feature, and reduce the dimension to obtain a second forward propagation feature and a second reverse propagation feature;

[0061] The feature fusion module is used to stack the second forward propagation feature and the second backward propagation feature, and reduce the dimension to obtain a third image feature.

[0062] Optionally, the video reconstruction unit performs image reconstruction on the third image feature to obtain a clear video frame sequence, specifically comprising:

[0063] The video reconstruction unit reconstructs the third image feature using the following formula to obtain a clear video frame sequence;

[0064] R=N recons (bidirectional_feature)

[0065] R is the clear video frame sequence obtained after image reconstruction, N recons (·) is the video reconstruction network, and bidirectional_feature is the third image feature.

[0066] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0067] The present invention discloses a video deblurring method and system, the method comprising: performing deblurring processing on a blurred video frame sequence based on a trained video deblurring model to obtain a clear video frame sequence, the trained video deblurring model comprising a feature extraction network, a spatiotemporal context self-attention network, a deep feature propagation network and a video reconstruction network; firstly, the feature extraction network is used to extract image features in the blurred video frame sequence to obtain a first image feature; then, the spatiotemporal context self-attention network is used to extract static context information and dynamic context information in the first image feature, and the static context information and the dynamic context information are fused to obtain a second image feature; then, according to the optical flow value of the blurred video frame sequence, the deep feature propagation network is used to perform optical flow and correction on the second image feature to obtain a third image feature; finally, the video reconstruction network is used to perform image reconstruction on the third image feature to obtain a clear video frame sequence. The video deblurring method provided by the present invention uses a spatiotemporal context self-attention network to enable the video deblurring model to better pay attention to the global information in adjacent frames, and the deep feature propagation network can pay attention to longer-term information in all video frames, thereby improving the effect of video deblurring. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0069] Figure 1 A flowchart of a video deblurring method provided by the present invention

[0070] Figure 2 Flowchart for implementing video deblurring by the video deblurring model provided by the present invention

[0071] Figure 3 The structural diagram of the feature extraction network provided by the present invention

[0072] Figure 4 The structure diagram of the spatiotemporal contextual self-attention network provided by the present invention

[0073] Figure 5 The structure diagram of the dynamic area perception convolution layer provided by the present invention

[0074] Figure 6 The structure diagram of the deep feature propagation network provided by the present invention DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] The purpose of the present invention is to provide a video deblurring method and system, which can better focus on the global information in adjacent frames and can focus on the longer-term information in all video frames, thereby improving the effect of video deblurring.

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] The present invention provides a video deblurring method, which utilizes a spatiotemporal context self-attention network to simultaneously focus on information in the time and space dimensions during video deblurring, and utilizes a deep feature propagation network to focus on longer-term information in video frames, which is more conducive to the restoration of clear videos. The present invention is based on a spatiotemporal context self-attention network and a deep feature propagation network, and integrates the model into an end-to-end lightweight network, while taking into account both computational complexity and accuracy.

[0079] like Figure 1 As shown, the present invention provides a video deblurring method, comprising deblurring a blurred video frame sequence based on a trained video deblurring model to obtain a clear video frame sequence, wherein the trained video deblurring model comprises a feature extraction network, a spatiotemporal context self-attention network, a deep feature propagation network and a video reconstruction network, Figure 1 Here, Feature extractor represents the feature extraction network, Spatial-temporal contextual transformer represents the spatiotemporal contextual self-attention network, Feature propagation represents the deep feature propagation network, Videoreconstruction represents the video reconstruction network, Blurryvideo represents the blurred video frame sequence, and Restoerd video represents the clear video frame sequence obtained after image reconstruction.

[0080] like Figure 2 As shown in FIG, the specific steps of deblurring the blurred video frame sequence based on the trained video deblurring model to obtain a clear video frame sequence are as follows:

[0081] Step 101: Use the feature extraction network to extract image features in a blurred video frame sequence to obtain a first image feature.

[0082] like Figure 3 As shown, the feature extraction network specifically includes 1 three-bit convolution operation and 3 residual operations, and the formula used is:

[0083] X=N f (V)

[0084] Where N f is the feature extraction network, V is the blurred video frame sequence, and X is the first image feature.

[0085] Furthermore, before step 101, the method also includes obtaining a sample blurred video used for training and a corresponding real clear video on a benchmark data set for video deblurring (GOPRO, DVD, BSD, etc.) to establish a training sample set, wherein the training sample set includes a sample blurred video and a real clear video, and the sample blurred video corresponds to the real clear video one-to-one; and obtaining N consecutive video image frames from the sample blurred video to obtain a sample blurred video frame sequence V.

[0086] Further, using the training sample set to train the video deblurring model specifically includes: using the training sample set, using the following loss function to train the video deblurring model, and obtaining a trained video deblurring model;

[0087]

[0088] Where loss represents the loss function, HEM(·,·) is the loss function for difficult sample mining, L is the fuzzy video frame sequence sample in the training sample set, and N k (L) represents the clear video frame sequence output by the video deblurring model, and GT is the clear video sequence sample corresponding to the blurred video frame sequence sample in the training sample set.

[0089] During the training process, the batch size was set to 2. There were two training stages, each with 300,000 iterations. The initial learning rate for the first 300,000 iterations was set to 2.0×10 -4 , the initial learning rate for the last 300,000 is set to 1.0×10 -4 , all use the cosine annealing learning strategy; the learning rate of the optical flow estimation network is set to 0.125 times of that of the main network, the weight of the optical flow estimation network is fixed in the first 5000 iterations, and then trained together with the video deblurring model, and the above operation is repeated until the experimental results converge when the loss function loss in the above formula is minimized, the video deblurring model training is completed, and the blurred video is deblurred using the video deblurring model.

[0090] Step 102: Use a spatiotemporal context self-attention network to extract static context information and dynamic context information from the first image feature, and fuse the static context information and the dynamic context information to obtain a second image feature.

[0091] like Figure 4 The structure and parameters of the spatiotemporal contextual self-attention network are shown. Its purpose is to calculate the attention features based on the information of the time dimension and the spatial dimension, and to fuse the obtained static context information and dynamic context information.

[0092] The spatiotemporal context self-attention network includes a self-attention layer, a dynamic area perception convolution layer and a feature fusion layer. Based on the self-attention layer, the first image feature is calculated by different convolution methods to obtain a query item query, a key key and a value value. The query item query, the key key and the value value can respectively characterize the properties of the feature X itself. The similarity of blocks between different frames is implicitly calculated between the query item query and the key key, and the calculated similarity weight is applied to the value value to obtain the second image feature out.

[0093] Specifically, the query item query and key key are calculated using the following formula to obtain a similarity weight matrix;

[0094] W=CONV3D(QK)×2,kernel size=3×1×1

[0095] Where W is the similarity weight matrix, Q is the query term, K is the key, kernel size is the three-dimensional convolution parameter, and CONV3D(·) is the three-dimensional convolution.

[0096] After obtaining the weight matrix W, a dynamic region-aware convolutional layer is used to fuse the value and the weight matrix W to obtain dynamic context information.

[0097] Specifically, dynamic region-aware convolution is as follows Figure 5 As shown, it first obtains the guiding feature map after several convolution operations on the sample blurred video frame sequence V, and calculates the index guidance of the maximum value of each pixel point in the channel dimension of the guiding feature map through the argmax function. The specific formula is:

[0098] guidance=argmax(CONV(V)),kernelsize=1×3×3

[0099] Where V is the value, CONV(·) is the convolution operation, argmax(·) is the argmax function, kernelsize is the three-dimensional convolution parameter, and W is irepresents the weight of the i channels, guidance is the index value, and c is the channel dimension;

[0100] Then the feature map is divided into different regions according to the size of the index position value. Assuming that the channel dimension of the guide map is c, this method divides the guide feature map into c different regions. Using formula W i =g(W)i=0…c convolution kernel generation module g(x) dynamically generates c convolution kernels from weight matrix W. The convolution kernel generation module is composed of several convolution networks. During the convolution process, the i-th convolution kernel performs convolution operation on all positions divided into region i in the guide feature map, thereby realizing a dynamic convolution process in which different positions use different convolution kernels for convolution operation.

[0101] Specifically, the dynamic context information is obtained by the following formula:

[0102]

[0103] Where Dynamic is the dynamic context information, g(·) is the convolution kernel generation operation, is a standard two-dimensional convolution.

[0104] Further, based on the feature fusion layer, the static context information and the dynamic context information are fused using the following formula to obtain a second image feature;

[0105] weight=softmax(CONV3D(mean(Dynamic+K))×2)

[0106] out=weight*Dynamic+(1-weight)*K

[0107] Where K is the key, representing the static context information, mean(·) is the mean operation, CONV3D(·) is a 1×1 three-dimensional convolution operation, softmax(·) is the softmax function, weight is the fusion weight matrix, and out is the second image feature.

[0108] Step 103, performing optical flow correction on the second image feature using the deep feature propagation network according to the optical flow value of the blurred video frame sequence to obtain a third image feature;

[0109] like Figure 6 The structure and parameters of the deep feature propagation network are shown. The deep feature propagation network uses the optical flow value flow on the blurred video frame sequence, and uses the optical flow to obtain the corrected video feature bidirectional_feature based on the second image feature out.

[0110] Specifically, firstly, a SpyNet optical flow estimation network is used to obtain an optical flow value of the blurred video frame sequence, wherein the optical flow value is an offset value of each pixel in both horizontal and vertical directions, and then the optical flow value is used to perform optical flow correction on the initial bidirectional propagation feature, wherein the bidirectional propagation feature includes a first forward propagation feature and a second reverse propagation feature;

[0111] The first forward propagation feature and the first reverse propagation feature after the optical flow correction are stacked with the second image feature respectively, and then the feature channel latitude is halved by using the convolution operation. Finally, the second forward propagation feature and the second reverse propagation feature are obtained after 20 residual operations, and the formula used is as follows:

[0112]

[0113]

[0114] In the formula, feat_forward_propagation is the first forward propagation feature, and its initial value is 0. feat_backward_propagation is the first backward propagation feature, and its initial value is 0. They are continuously updated by optical flow correction with the current frame during the forward propagation and backward propagation processes. feature_forward_propagation is the second forward propagation feature, feature_backward_propagation is the second backward propagation feature, and flow_warp(·) represents the process of correcting the bidirectional propagation feature to the current frame using the predicted optical flow value. represents a stacking operation, conv_down(·) represents a convolution operation that halves the dimension of the feature channel, and resblock(·) represents 20 residual operations.

[0115] Further, the use of the optical flow value to correct the bidirectional propagation feature to the current frame is specifically as follows:

[0116] For the i-th frame image, during the forward propagation process, we need to correct the forward propagation features of the i-1 frame image to the i-th frame image according to the optical flow value of the i-1 frame image, fuse the corrected forward propagation features with the forward propagation features of the i-th frame through the concat function, and then obtain the reconstructed i-th frame information through several residual operations. Similarly, during the backward propagation process, the backward propagation features of the i+1 frame need to be corrected to the i-th frame.

[0117] Finally, the second forward propagation feature and the second back propagation feature are stacked and convolved to reduce the feature channel latitude by half to obtain the third image feature, and the formula is as follows:

[0118]

[0119] Where bidirectional_feature is the third image feature.

[0120] Step 104: Reconstruct the third image feature using a video reconstruction network to obtain a clear video frame sequence.

[0121] The video reconstruction network uses the encoder-decoder structure in the existing scale-recurrent-network. This encoding structure first doubles the channel dimension of the feature and halves the spatial dimension, then passes through several residual networks and repeats the above operation once to obtain a feature map with a spatial dimension of one-fourth of the original size and a channel dimension quadrupled. Then, a deconvolution is performed to halve the channel dimension of the feature and double the spatial dimension. After repeating this twice, it is restored to the original size, and finally a convolution operation is performed to obtain the restored video sequence.

[0122] Specifically, a video reconstruction network is used to reconstruct the third image feature using the following formula to obtain a clear video frame sequence:

[0123] R=N recons (bidirectional_feature)

[0124] R is the clear video frame sequence obtained after image reconstruction, N recons (·) For video reconstruction network, bidirectional_feature is the third image feature.

[0125] The present invention provides another specific embodiment:

[0126] A video deblurring system, wherein the video deblurring system is a trained video deblurring system, and the trained video deblurring system is used to deblur a blurred video frame sequence to obtain a clear video frame sequence; the video deblurring system comprises a feature extraction unit, a spatiotemporal context self-attention unit, a deep feature propagation unit and a video reconstruction unit; the feature extraction unit is used to extract image features in the blurred video frame sequence to obtain a first image feature; the spatiotemporal context self-attention unit is used to extract static context information and dynamic context information in the first image feature, and fuse the static context information and the dynamic context information to obtain a second image feature; the deep feature propagation unit is used to perform optical flow correction on the second image feature according to the optical flow value of the blurred video frame sequence to obtain a third image feature; the video reconstruction unit is used to perform image reconstruction according to the third image feature to obtain a clear video frame sequence.

[0127] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video deblurring method, characterized in that: The method comprises performing deblurring processing on a blurred video frame sequence based on a trained video deblurring model to obtain a clear video frame sequence, wherein the trained video deblurring model comprises a feature extraction network, a spatiotemporal context self-attention network, a deep feature propagation network and a video reconstruction network; The method further includes: performing deblurring processing on the blurred video frame sequence based on the trained video deblurring model to obtain a clear video frame sequence. Using the training sample set, the video deblurring model is trained using the following loss function to obtain a trained video deblurring model; loss=||N k (L)-GT||1+2*HEM(N k (L),GT) Where loss represents the loss function, HEM(·,·) is the loss function for difficult sample mining, L is the fuzzy video frame sequence sample in the training sample set, and N k (L) represents the clear video frame sequence output by the video deblurring model, and GT is the clear video sequence sample corresponding to the blurred video frame sequence sample in the training sample set; The specific steps of performing deblurring processing on the blurred video frame sequence based on the trained video deblurring model to obtain a clear video frame sequence are as follows: The feature extraction network is used to extract image features in the blurred video frame sequence to obtain a first image feature; Extracting static context information and dynamic context information from the first image feature using a spatiotemporal context self-attention network, and fusing the static context information and the dynamic context information to obtain a second image feature; According to the optical flow value of the blurred video frame sequence, the deep feature propagation network is used to perform optical flow correction on the second image feature to obtain a third image feature; A video reconstruction network is used to reconstruct the third image feature to obtain a clear video frame sequence.

2. The video deblurring method according to claim 1, characterized in that: The spatiotemporal contextual self-attention network includes a self-attention layer, a dynamic region-aware convolutional layer, and a feature fusion layer; The method of extracting static context information and dynamic context information from the first image feature by using the spatiotemporal context self-attention network, and fusing the static context information and the dynamic context information to obtain the second image feature specifically includes: Performing convolution processing on the first image feature based on the self-attention layer to obtain a query item query, a key key and a value value of the first image feature, wherein the key key is static context information; The similarity weight matrix is ​​calculated using the following formula according to the query item query and key key; W=CONV3D(QK)×2,kernel size=3×1×1 Where W is the similarity weight matrix, Q is the query item, K is the key, kernel size is the three-dimensional convolution parameter, and CONV3D(·) is the three-dimensional convolution; Based on the dynamic region-aware convolutional layer, the dynamic context information is calculated using the following formula according to the value value and the similarity weight matrix; Guidance=argmax(CONV(V)),kernelsize=1×3×3 W i =g(W)i=0…c Where V is the value, CONV(·) is the convolution operation, argmax(·) is the argmax function, kernelsize is the three-dimensional convolution parameter, and W is i represents the weight of the i channels, guidance is the index value, c is the channel dimension, Dynamic is the dynamic context information, g(·) is the convolution kernel generation operation, It is a standard two-dimensional convolution; Based on the feature fusion layer, the static context information and the dynamic context information are fused using the following formula to obtain the second image feature; Weight=softmax(CONV3D(mean(Dynamic+K))×2) Out=weight*Dynamic+(1-weight)*K Where K is the key, representing the static context information, mean(·) is the mean operation, CONV3D(·) is a 1×1 three-dimensional convolution operation, softmax(·) is the softmax function, weight is the fusion weight matrix, and out is the second image feature.

3. The video deblurring method according to claim 1, characterized in that: The step of performing optical flow correction on the second image feature using the deep feature propagation network according to the optical flow value of the blurred video frame sequence to obtain the third image feature specifically includes: Performing optical flow correction on initial bidirectional propagation features using the optical flow values ​​of the blurred video frame sequence, wherein the bidirectional propagation features include a first forward propagation feature and a first reverse propagation feature; The first forward propagation feature and the first reverse propagation feature after the optical flow correction are stacked with the second image feature respectively, and the dimension is reduced to obtain a second forward propagation feature and a second reverse propagation feature; The second forward propagation feature and the second backward propagation feature are stacked and dimensionally reduced to obtain a third image feature.

4. The video deblurring method according to claim 1, characterized in that: The third image feature is reconstructed using a video reconstruction network to obtain a clear video frame sequence, which specifically includes: Using a video reconstruction network, the third image feature is reconstructed using the following formula to obtain a clear video frame sequence; R=N recons (bidirectional_feature) R is the clear video frame sequence obtained after image reconstruction, N recons (·) is the video reconstruction network, and bidirectional_feature is the third image feature.

5. A video deblurring system, characterized in that: The video deblurring system is a trained video deblurring system, and the trained video deblurring system is used for deblurring a blurred video frame sequence to obtain a clear video frame sequence; The video deblurring system comprises a feature extraction unit, a spatiotemporal contextual self-attention unit, a deep feature propagation unit and a video reconstruction unit; The video deblurring system further comprises a training module, wherein the training module is used to train the video deblurring system using a training sample set with a regression loss function and a difficult sample mining loss function; The feature extraction unit is used to extract image features in the blurred video frame sequence to obtain a first image feature; The spatiotemporal context self-attention unit is used to extract static context information and dynamic context information from the first image feature, and fuse the static context information and the dynamic context information to obtain a second image feature; The deep feature propagation unit is used to perform optical flow correction on the second image feature according to the optical flow value of the blurred video frame sequence to obtain a third image feature; The video reconstruction unit is used to perform image reconstruction according to the third image feature to obtain a clear video frame sequence.

6. The video deblurring system according to claim 5, characterized in that: The spatiotemporal context self-attention unit includes a self-attention module, a dynamic region-aware convolution module and a feature fusion module; The self-attention module is used to perform convolution processing on the first image feature to obtain a query item query, a key key and a value value of the first image feature, wherein the key key is static context information, and a similarity weight matrix W is calculated according to the query item query and the key key using the following formula; W=CONV3D(QK)×2,kernelsize=3×1×1 Where W is the similarity weight matrix, Q is the query item, K is the key, kernel size = 3 × 1 × 1 is the three-dimensional convolution parameter, and CONV3D(·) is the three-dimensional convolution; The dynamic region-aware convolution module is used to calculate the dynamic context information according to the value value and the similarity weight matrix W using the following formula; W i =g(W)i=0…c standard 2D convolution Where V is the value, CONV(·) is the convolution operation, argmax(·) is the argmax function, kernelsize is the three-dimensional convolution parameter, Wi represents the weight of the i channels, guidance is the obtained index value, c is the channel dimension, Dynamic is the dynamic context information, g(·) is the convolution kernel generation operation, It is a standard two-dimensional convolution; The feature fusion module is used to obtain the second image feature by fusing the static context information and the dynamic context information.

7. The video deblurring system according to claim 5, characterized in that: The deep feature propagation unit includes an optical flow correction module, a feature update module and a feature fusion module; The optical flow correction module is used to perform optical flow correction on the initial two-way propagation feature according to the optical flow value of the blurred video frame sequence, and the two-way propagation feature includes a first forward propagation feature and a first reverse propagation feature; The feature updating module is used to stack the first forward propagation feature and the first reverse propagation feature after the optical flow correction with the second image feature, and reduce the dimension to obtain a second forward propagation feature and a second reverse propagation feature; The feature fusion module is used to stack the second forward propagation feature and the second backward propagation feature, and reduce the dimension to obtain a third image feature.

8. The video deblurring system according to claim 5, characterized in that: The video reconstruction unit performs image reconstruction on the third image feature to obtain a clear video frame sequence, which specifically includes: The video reconstruction unit reconstructs the third image feature using the following formula to obtain a clear video frame sequence; R=N recons (bidirectional_feature) R is the clear video frame sequence obtained after image reconstruction, N recons (·) is the video reconstruction network, and bidirectional_feature is the third image feature.

Images