A novel video deblurring codec network based on multi-domain information fusion

Abstract

This invention relates to the field of video recording technology, and more particularly to a method for a novel video deblurring codec network based on multi-domain information fusion. The steps are as follows: S1: Extract features from every 5 consecutive blurred video frames using a backbone; S2: Pass the features extracted in S1 to a feature fusion network to form a hybrid feature; S3: Restore the sharp image to the hybrid feature using a frequency domain converter. The novel video deblurring codec network based on multi-domain information fusion provided by this invention, through a spatiotemporal frequency domain fusion network (STFFNet), ensures that the decoded features contain both global and local information. Simultaneously, the introduction of a frequency domain converter enables the utilization of the differences between blurred and sharp images in the frequency domain, and the computation of these differences in the frequency domain, thereby achieving state-of-the-art performance on benchmark datasets; achieving optimal visual effects and quantitative results.

Description

Technical Field

[0001] This invention relates to the field of video recording technology, and in particular to a method for a novel video deblurring codec network based on multi-domain information fusion. Background Technology

[0002] Video recording often encounters quality issues due to various factors, such as defocusing and camera shake, which severely impact the performance of downstream tasks like detection and tracking. Therefore, improving the quality of model-reconstructed images has become a pressing problem in video deblurring. In deblurring tasks, a blurred image is typically considered the result of a blur kernel acting on a sharp image. Previously, researchers needed to manually design features to estimate the sharp image. While image deblurring research is necessary, it performs poorly in more complex scenarios.

[0003] Deep learning technology has brought new solutions and proposed some novel methods. Using deep learning, researchers can save the overhead of manually designing features and give network models full autonomy. In video deblurring tasks, utilizing consecutive frames becomes crucial. CNN-based methods address this problem by stacking adjacent frames as input to the model. This approach often makes learning the temporal information of the input data challenging, leading to limited modeling effectiveness. RNN-based methods use a recurrent neural network architecture to process input information frame by frame, making better use of the spatiotemporal information between frames. However, due to the limited information exchange between frames in this method, information extraction lacks feedback and interaction. Furthermore, blurred and sharp images have significant differences in their frequency domain representations, as shown in the figure. Video deblurring algorithms typically focus on utilizing spatial and temporal information, neglecting the use of frequency domain information. Therefore, researching how to utilize frequency domain information is very important.

[0004] To address this issue, a novel video deblurring codec network based on multi-domain information fusion is designed to provide an alternative technical solution. Summary of the Invention

[0005] Therefore, it is necessary to provide a novel video deblurring codec network based on multi-domain information fusion to address the aforementioned technical problems.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] The method for a novel video deblurring codec network based on multi-domain information fusion comprises the following steps:

[0008] S1: Extract the features of each frame from every 5 consecutive blurred video frames using the backbone.

[0009] S2: The features extracted in S1 are passed to the feature fusion network to form hybrid features;

[0010] S3: Restore the mixed features to a clear image using a frequency domain converter.

[0011] In a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, in step S1, features of each frame are extracted using the backbone from every 5 consecutive blurred video frames, as follows:

[0012] Features are extracted from blurred video frames using DEF decoding to obtain the features. The expression is as follows:

[0013] ;

[0014] ;

[0015] in, For memorizing information, Representing the A video sequence is set to 5 frames. Generated in the last frame, denoted as .

[0016] As a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, the DEF decoding extracts features from the blurred video frames, and the steps are as follows:

[0017] Video frames are sequentially encoded to obtain global features, and only each video frame is decoded to obtain local features containing global information.

[0018] In a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, in step S2, the features extracted in S1 are passed to the feature fusion network to form hybrid features, and the steps are as follows:

[0019] The features extracted by S1 are combined into mixed features by using amplitude blocks.

[0020] As a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, the step of forming a hybrid feature from the features extracted by S1 through difference blocks is as follows:

[0021] The central frame features and the other four frame features are combined into four feature sets;

[0022] Learn the differences between each set of features in each Block;

[0023] The four sets of feature differences are concatenated and convolved to the corresponding dimension to obtain the mixed feature F.

[0024] As a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, a differential amplifier is used to concentrate the blurring differences between frames. The specific steps are as follows:

[0025] The difference between a pair of frames is represented as a feature of a pair of non-center frames. Subtracting the features of the center frame yields the difference information, expressed as follows:

[0026] ;

[0027] in, For frames, Indicates the first For frames, ∈[1,4], This is the difference information;

[0028] The result of convolution with the GAP branch forms a block, which is then processed by the four pairs of feature groups of the DA block. The overall characteristic F is expressed as follows:

[0029] .

[0030] In a preferred embodiment of the novel video deblurring codec network based on multi-domain information fusion provided by the present invention, in step S3, the mixed features are restored to a clear image through a frequency domain converter, and the steps are as follows:

[0031] The predicted image and ground truth values ​​are converted into amplitude and phase information respectively using FDC;

[0032] The final difference is obtained by calculating the Manhattan distance between the predicted image and the ground truth in terms of amplitude and phase, as expressed below:

[0033]

[0034] Where FDC(·) represents the frequency domain converter and L1(·) represents the loss function. Represents the ground truth image, Represents the predicted image. and These represent the amplitude and phase information of the predicted image, respectively. and These represent the amplitude and phase information of the ground truth image, respectively.

[0035] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0036] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0037] The present invention provides a novel video deblurring codec network method based on multi-domain information fusion. Through the spatiotemporal frequency domain fusion network (STFFNet), the decoded features contain both global and local information. At the same time, a frequency domain converter is introduced to utilize the difference between blurred and sharp images in the frequency domain and to calculate this difference in the frequency domain. As a result, it can achieve a state-of-the-art level on benchmark datasets, achieving the best visual effect and quantitative results. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram illustrating the differences in amplitude and phase information between the blurred image and the true ground image of the present invention.

[0040] Figure 2 This is a schematic diagram of the overall architecture of STFFNet of the present invention;

[0041] Figure 3 This is a schematic diagram of the differential amplifier (DA) of the present invention;

[0042] Figure 4 This is a schematic diagram of the frequency domain information of the present invention;

[0043] Figure 5 This is a visual schematic diagram of the different feature fusion network of the present invention;

[0044] Figure 6 This is a diagram showing the impact of the DA block of this invention on the PSNR metric during training.

[0045] Figure 7 This is a schematic diagram comparing the visual results of the method of the present invention with those of other methods. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0050] Example 1

[0051] Reference Figures 1-7 A novel video deblurring codec network based on multi-domain information fusion.

[0052] 1. Video recovery

[0053] To better recover video frames, a spatiotemporal frequency domain fusion network (STFFNet) was proposed. It consists of three parts, and its framework is as follows: Figure 2 As shown. First, every 5 consecutive blurred video frames are processed using the backbone to extract the features f(x) of each frame. Figure 2 (a)). Next, f is passed to the feature fusion network ( Figure 2 (b) , mix the 5 features f into a mixed feature f, and use the reconstructor ( Figure 2 (c) The mixed features f are recovered into a sharp image. Finally, the loss is passed back to guide subsequent processes.

[0054] 1.1 Novel Backbone Network

[0055] In video processing, temporal information between frames is crucial. With the development of deep learning technology, RNN-based methods have replaced CNN-based methods as the mainstream approach. However, global features can be more clearly incorporated during RNN propagation to improve the globality of generated features. Inspired by Seq2Seq and Bi-LSTM, a new backbone network—DEF Decoding (Decoding Every Frame)—is proposed. It adopts the encoder-decoder idea, such as... Figure 2 As shown in Figure a, its architecture is designed as follows: video frames are sequentially encoded to obtain global features, and only the global features of each video frame are decoded to obtain local features containing global information. More specifically, sequential frames are first sequentially encoded in memory information h by the RDB unit, and then passed to subsequent frames. Then, the RDB unit combines the video frames to sequentially decode the final memory information h to obtain the features. Within this framework, more advanced features are extracted.

[0056] (1)

[0057] (2)

[0058] In this architecture, memory information ;Generated during the encoder stage (where Representing the A video sequence is set to 5 frames. In the first frame of the video... =0, If it is empty, then It is the encoder for each frame of the image. Generated in the last frame, denoted as .

[0059] The global information containing the frame sequence is given by formula (1): Global information generated by the encoder In the decoder stage, respectively for the first Frame characteristics Decode, no longer perform the first The frames are decoded; generated, and formulated into equations, each decoding process is independent and does not communicate with each other.

[0060] Compared to Seq2Seq networks, it eliminates the need for communication between frames in the decoder stage. The encoder stage remains unchanged, while the decoder only retains the output features. This change was experiment-driven; however, Seq2Seq passed too much redundant information in this task, further reducing network performance and increasing the difficulty of fitting.

[0061] 1.2 Feature Fusion Module

[0062] After obtaining the features of each frame image Subsequently, video deblurring is viewed as a problem of recovering a clear frame from a series of blurred frames. Different frames have different effects on the restoration effect, and different considerations should be taken into account during the processing.

[0063] A novel data fusion method, called Difference Amplitude Blocks (dab), is proposed. It is composed of multiple fused blocks, and its structure is as follows: Figure 2As shown in (b), the five features generated by DEF are fused here. The central frame feature and the other four frame features are combined into four feature sets, and the differences between each feature set are learned in the Block. Then, the differences of the four feature sets are concatenated and convolved to the corresponding dimension to obtain the mixed feature F, which is then passed to the next step.

[0064] In this embodiment, Block refers to Figure 2 The first module shown in (b) is identical to the first one. Specifically, a Block refers to the combination of the initial central feature ft and the frame features. The concat operation is performed, and the results are fed into the global average pooling layer GPA, the differential amplifier DA module, and the convolutional layer respectively. The GPA result is multiplied element-wise with the result obtained from the convolution, and the result is then convolved again to obtain ft'. At the same time, the differential amplifier DA module is multiplied element-wise with the result obtained from the convolution to obtain fi'. ft' and fi' are concatted and then convolved again.

[0065] In contrast to the GSA structure, DABs (Difference Amplitude Blocks) divide each block into a branch parallel to the GAP (Global Average Pooling) layer. This branch uses a differential amplifier (DA) to concentrate blur differences between frames. The differential amplifier (DA) structure is as follows: Figure 3 As shown. Specifically, a pair of frames (in Indicates the first For frames, The difference between ∈[1,4]) is represented as a feature of a pair of non-center frames. Subtracting the features of the center frame yields the difference information (represented as...). )and The convolution product is formalized as Formula 4. The result of convolution with the GAP branch forms a block, which is then processed by the four pairs of feature groups of the DA block. The overall characteristic F is formulated as Equation 5.

[0066] (3)

[0067] (4)

[0068] (5)

[0069] In the formula, Indicates the center frame of the frame sequence. ∈[3,N﹢﹣2], where, Indicates the number of frames that are different from the center frame. ∈[1,2].

[0070] 1.3 Frequency Domain Guidance

[0071] Blurred or unambiguous is represented by significant differences in the frequency domain, such as Figure 1 As shown. It is generally believed that frequency domain information can be used to guide image restoration from another perspective. In image processing, amplitude typically represents the contribution of each spatial frequency in an image. High amplitude usually corresponds to the high-frequency components of an image, such as edges or textures, which often contain detailed information. On the other hand, low amplitude corresponds to smooth regions or low-frequency components. Compared to sharp images, blurred images tend to have weaker high-frequency information. Therefore, reducing the influence of low-frequency blurring information while increasing the high-frequency information of the restored image is another focus of this invention.

[0072] Information is introduced into the frequency domain using a frequency domain converter (FDC), and its framework is as follows: Figure 4 As shown, the predicted image and the ground truth are converted into amplitude (amp) and phase (pha) information, respectively, using FDC. The final difference is obtained by calculating the Manhattan distance between the predicted image and the ground truth in terms of amplitude and phase.

[0073] Formula 6 specifically utilizes amplitude and phase information extracted from the predicted image and the ground truth, respectively. , , , Then, the magnitude and distance of the difference between the predicted image and the ground truth are calculated. .

[0074] (6)

[0075] In the formula, FDC(·) represents the frequency domain converter, and L1(·) represents the loss function. Represents the ground truth image, Represents the predicted image. and These represent the amplitude and phase information of the predicted image, respectively. and These represent the amplitude and phase information of the ground truth image, respectively.

[0076] 1.4 Loss Function

[0077] It includes two main types of loss: MSE loss and FDC loss. The entire loss function can be expressed as Equation 7:

[0078]

[0079] in, and Two weight parameters are set to 1 and 0.1 respectively. Loss function. It is the mean square error (in pixels) between the actual value and the predicted value. It is the frequency domain Manhattan distance between the actual ground value and the predicted value.

[0080] 2. Experimental Evaluation

[0081] 2.1 Dataset

[0082] The Beam Splitter Deblurring Dataset (BSD) employs a beam splitter acquisition system with two synchronized cameras to obtain sharp and blurred images by controlling exposure time and intensity. Unlike methods that obtain blur from abrupt frame degradation, BSD is a truly real-world dataset containing three different frame rate datasets: 1ms-8ms, 2ms-16ms, and 3ms-24ms. The training video dataset contains 60 videos, the test video dataset contains 20 videos, and the validation video dataset contains 20 videos, each dataset containing sharp frames and their corresponding blurred frames.

[0083] The GoPro dataset (GoPro) is a dataset used for deblurring tasks. It consists of 3214 blurred images, each 1280 × 720 pixels in size, with 2103 being training images and 1111 being test images. The dataset provides a one-to-one correspondence between real-blurred images and ground-truth images, both captured by high-speed cameras.

[0084] The Realistic and Diverse Scenes Dataset (red) is a dataset that provides realistic and dynamic scenes for video deblurring and super-resolution. The dataset consists of 300 video sequences at a resolution of 720 × 1280, with 240 as training videos, 30 as validation videos, and 30 as test videos.

[0085] 2.2 Experimental Setup

[0086] All of this was done on a PyTorch instance equipped with two 3090 GPUs. The code in this invention is based on ESTRNN. [Zhong , Z., Gao, Y., Zheng, Y., Zheng, B., Sato, I.: Real-world video deblurring: A benchmark dataset and an efficient rec urrent neural network. International Journal of Computer Vision 131(1), 284-301 (2023)] The model was not pre-trained on the rest of the dataset. It was trained 500 times using the Adam optimizer with an initial learning rate of 5 × 10⁻⁴. The model was trained using RGB patches of size 256 × 256 as input to subsequences of 5 frames. Furthermore, horizontal and vertical flipping was implemented for each subsequence to augment the data. The batch size was set to 8.

[0087] 2.3 Ablation Study

[0088] Ablation studies were conducted to demonstrate the effectiveness of DEF, cab, and frequency domain information. As shown in Tables 1(a) and 1(b), a series of comparative experiments demonstrated the effectiveness of DEF. DEF shows significant improvements compared to RNNs and Bi-LSTMs.

[0089] Table 1: Ablation Study Table

[0090]

[0091] (a) compares the proposed DEF(Ours) backbone with the RNN in ESTRNN. (b) compares the DEF and Bi-LSTM backbones. (c) compares networks with different feature fusions. (d) shows the effect of using frequency domain information with or without it. For quantitative experimental comparisons, other parameters remained constant when comparing a specific parameter. All ablation studies were performed on BSD, with times ranging from 2ms to 16ms. (√ indicates no frequency information used, √ indicates frequency information used. Best and second best scores are highlighted and underlined).

[0092] A comparison was established, with one group replacing the GSA network with DABs and the other group replacing the DA in the DABs with Global Max Pooling (GMP) operations. The results are shown in Table 1(c), demonstrating that the multi-branch structure improves network performance. Furthermore, the network shows a more significant improvement only when DA is used instead of GMP. Additionally, from... Figure 5 Visual analysis reveals that the text generated using the DABs module has a clearer left side and less color difference. Overall, the visual effect of DABs is superior to the other two modules.

[0093] DABs outperform GSA and GMP because the GSA structure focuses more on the results after various transformations between two frames. Furthermore, the transformations involve many elements, and learning the differences between two frames takes more time. DABs can accelerate the convergence process while focusing on inter-frame differences, such as... Figure 6 As shown.

[0094] Finally, through ablation experiments, it was found that the introduction of frequency domain information significantly improved network performance, as shown in Table 1 (d).

[0095] 2.4 Quantitative and Visual Results

[0096] Table 2: A pictorial comparison of quantitative results with other methods on the BSD dataset (highlighting the best and second-best scores).

[0097]

[0098] Side-by-side comparative experiments were conducted on the real-world BSD dataset using multiple methods. The visualization results are as follows: Figure 7 As shown in Table 2, more visually appealing results were achieved in BSD. Quantitative results are presented in Table 2. Compared to all other methods, the method of this invention achieves state-of-the-art performance. Both visual and quantitative results validate the effectiveness of the method of this invention in practical video deblurring tasks.

[0099] Furthermore, the method of this invention was compared with the GoPro and Reds datasets, and the quantitative results are shown in Table 3. The method of this invention also reaches the state-of-the-art level.

[0100] Table 3: Quantitative results of the method of this invention compared with other methods on the GoPro and Reds datasets (highlighting the best and second-best scores).

[0101]

[0102] 3. Conclusion

[0103] An efficient video deblurring method (STFFNet) is proposed for perceptually guided and metric-favorable enhancement. Specifically, it first explores the use of encoder-decoder architecture to construct a new backbone. While generating current frame features, it combines global features to extract deeper and broader information, achieving better reconstruction. Furthermore, a novel feature fusion module is developed to accelerate fitting and improve modeling results. Finally, frequency domain information is incorporated into the network, allowing it to focus more on high-frequency information and obtain more explicit images. Experimental results show that STFFNet achieves state-of-the-art performance on benchmark datasets.

[0104] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for a novel video deblurring codec network based on multi-domain information fusion, characterized in that, The steps are as follows: S1: Extract the features of each frame from every 5 consecutive blurred video frames using the backbone. S2: The features extracted in S1 are passed to the feature fusion network to form hybrid features; S3: Recover the blended features into a clear image using a frequency domain converter; In step S2, the features extracted in S1 are passed to the feature fusion network to form hybrid features. The steps are as follows: The features extracted by S1 are combined into mixed features by using amplitude blocks; The steps for forming mixed features from the features extracted by S1 through the difference block are as follows: The central frame features and the other four frame features are combined into four feature sets; Learn the differences between each set of features in each Block; The four sets of feature differences are concatenated and convolved to the corresponding dimensions to obtain the mixed feature F; The blur differences between frames are concentrated using a differential amplifier. The specific steps are as follows: The difference between a pair of frames is represented as a feature of a pair of non-center frames. Subtracting the features of the center frame yields the difference information, expressed as follows: ； in, For frames, Indicates the first For frames, ∈[1,4], This is the difference information; The result of convolution with the GAP branch forms a block, which is then processed by the four pairs of feature groups of the DA block. The overall characteristic F is expressed as follows: ； In step S3, the mixed features are restored to a clear image using a frequency domain converter, and the steps are as follows: The predicted image and ground truth values ​​are converted into amplitude and phase information respectively using FDC; The final difference is obtained by calculating the Manhattan distance between the predicted image and the ground truth in terms of amplitude and phase, as expressed below: Where FDC(·) represents the frequency domain converter and L1(·) represents the loss function; Represents the ground truth image, Represents the predicted image. and These represent the amplitude and phase information of the predicted image, respectively. and These represent the amplitude and phase information of the ground truth image, respectively.

2. The method for a novel video deblurring codec network based on multi-domain information fusion according to claim 1, characterized in that, In step S1, features of each frame are extracted using the backbone from every 5 consecutive blurred video frames. The steps are as follows: Features are extracted from blurred video frames using DEF decoding to obtain the features. The expression is as follows: ； ； in, For memorizing information, Representing the A video sequence is set to 5 frames. Generated in the last frame, denoted as .

3. The method for a novel video deblurring codec network based on multi-domain information fusion according to claim 2, characterized in that, The DEF decoding extracts features from blurred video frames, and the steps are as follows: Video frames are sequentially encoded to obtain global features, and only each video frame is decoded to obtain local features containing global information.

Images