Video deblurring method based on spectral attention and feature shift

Abstract

The application discloses a video deblurring method based on spectrum attention and feature shift, relates to the field of video deblurring technology in computer vision, and comprises a downsampling module, a spectrum propagation module, a feature shift aggregation module and an upsampling module in the method. Main steps are: (1) selecting a public image dataset and obtaining smaller size downsampling features through the downsampling module; (2) a spectrum propagation (SP) module utilizes spectrum information in the downsampling features and extracts forward and reverse hidden features from the downsampling features; (3) a feature shift aggregation (FSA) module integrates the hidden features in the two directions to obtain corresponding aggregated features; (4) an upsampling module takes the initial blurred image and the aggregated features as input, upsamples the aggregated features and adds the aggregated features to the initial blurred image to obtain a final deblurring result. The application can efficiently utilize multi-scale features and spectrum information, and realizes fast and efficient video deblurring while controlling the model size.

Description

Technical Field

[0001] This invention relates to the field of video deblurring technology in computer vision, and in particular to a video deblurring method based on spectral attention and feature shifting. Background Technology

[0002] Videos can become blurry due to uncontrollable factors such as camera shake, object movement, shutter speed, and inaccurate focusing. Video blurring not only significantly impacts video quality and visual effects but can also lead to the loss of image information. The development of video deblurring technology provides an effective means to improve the quality and appearance of blurry videos. Video deblurring is an important computer vision task, its main goal being to recover sharp video from blurry video sequences. Video deblurring is also a fundamental problem in the field of computer vision. It has a positive impact on many advanced tasks, including object detection, image segmentation, and object tracking.

[0003] Common deep learning methods can be divided into two categories: sliding window-based methods and recurrent neural network (RNN)-based methods. Both methods have some limitations. Sliding window-based methods are relatively inefficient in feature utilization and may fail to fully capture important information in the video; while RNN-based methods may perform poorly when processing long video sequences because they cannot effectively utilize information from frames far from the target frame. Furthermore, the Transformer model improves performance by introducing self-attention mechanisms and positional encoding. However, this also introduces significant computational complexity, making model training and inference more time-consuming. In recent years, many deep learning-based video deblurring algorithms have achieved good deblurring results, but most require large model sizes and high hardware requirements. Analyzing these methods reveals several problems, such as insufficient feature utilization efficiency and an unreasonable stacking structure of modules. These methods rely too heavily on increasing model size to improve deblurring results, leaving considerable room for improvement. For example, multi-scale structures, which are widely used in video deblurring, require constructing multiple identical structures to process input images at different scales in conventional multi-scale methods. This leads to a large amount of repetitive computation and parameters in the model, increasing its complexity and training difficulty. Most existing video deblurring models are also very large and suffer from inadequate structural design. Summary of the Invention

[0004] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide a video deblurring method based on spectral attention and feature shifting. It proposes a video deblurring method with a small model size and number of parameters, yet achieving fast and effective results. The core of this method lies in designing an SP module (Spectrum Propagation Module) and an FSA module (Feature Shift Aggregation Module). The SP module utilizes spectral information and multi-scale structures within the module to propagate and align inter-frame features. The FSA module uses feature shifting to increase the receptive field of the module and combines frequency domain information to assist in intra-frame feature aggregation. The combination of these two modules enables the model to achieve good video deblurring results on a relatively small scale.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A video deblurring method based on spectral attention and feature shifting proposed in this invention includes the following steps:

[0007] Step S1: Select a public image dataset as the ground truth, and use bilinear interpolation algorithm to downsample the input blurred image to obtain downsampled features;

[0008] Step S2: For the downsampled features obtained in step S1, construct the spectrum propagation SP module to further extract the features useful for deblurring from the downsampled features, and obtain the forward and reverse hidden features;

[0009] Step S3: Construct a Feature Shift Aggregation (FSA) module to aggregate forward and reverse hidden features to obtain aggregated features;

[0010] Step S4: Input the aggregated features obtained in step S3 into the upsampling module and fuse them with the original input to obtain the final deblurring result.

[0011] As a further optimization scheme of the video deblurring method based on spectral attention and feature shift described in this invention, after steps S1-S4, a video deblurring network based on spectral attention and feature shift is obtained. After step S4, step S5 is also included: setting the loss function and training the video deblurring network based on spectral attention and feature shift to obtain the mapping model.

[0012] As a further optimization of the video deblurring method based on spectral attention and feature shift described in this invention, step S1 uses a downsampling module to downsample the original blurred image to obtain features with smaller dimensions, and preliminarily extracts features from the blurred frames of the input blurred image, as shown in the following formula:

[0013]

[0014] Among them, I i Let i be the i-th blurred frame of the input, where i is the frame index. Indicates the downsampling module, x i Let be the downsampled features of the i-th blurred frame.

[0015] As a further optimization of the video deblurring method based on spectral attention and feature shift described in this invention, the SP module in step S2 includes a forward SP module and a reverse SP module. Both the forward SP module and the reverse SP module include 3 CAB U-Net structures and 2 SFM modules. The SP module can obtain a hidden feature of the current frame that retains the information of adjacent frames. The formula of the forward SP module is as follows:

[0016]

[0017] Where, φ C () represents CAB U-Net, φ S () represents the SFM module, u i , u i 'and The SP module outputs positive hidden features, representing variables that represent the results of intermediate operations. This is the positive hidden feature of the previous frame;

[0018] The formula for the reverse SP module in backpropagation is as follows:

[0019] u i =φ C (x i )

[0020]

[0021] in, It is the inverse hidden feature of the next frame, and the final output of the inverse SP module is denoted as

[0022] As a further optimization scheme of the video deblurring method based on spectral attention and feature shift described in this invention, the CAB U-Net structure uses multiple channel attention blocks (CABs) as the basic structure, and forms a U-shaped structure with downsampling and upsampling operations. It also uses two CABs as jump connections to connect the upsampling path and the downsampling path.

[0023] The SFM module first concatenates the two input features along the channel dimension, merging the features together. Then, this mixed feature is divided into two parts and processed by the channel attention layer respectively. Two spectral branches are added to the SFM module, using Discrete Fourier Transform (DFT) to obtain frequency domain feature information and process it. The spectral information captures fuzzy information that is not easily found in the time domain, assisting in feature alignment.

[0024] As a further optimization of the video deblurring method based on spectral attention and feature shifting described in this invention, the FSA module in step S3 includes a shift block, an aggregation block, a CAB U-Net, and a convolutional layer, wherein...

[0025] The specific method of the FSA module is as follows:

[0026] The forward and reverse hidden features obtained in step S2 and Input into the FSA module, and First, the feature is copied and then segmented by channel, dividing each channel into two features; for Will Divide the feature into two channels, each with half the number of channels as the original feature: the first positive feature. Second positive feature Reverse hidden features After being cut Divide the feature into two features based on the channel dimension, with the number of channels being half that of the original feature: the first inverse feature. Second reverse feature The subsequent processing procedure is as follows:

[0027]

[0028] Where [·] represents a tensor concatenation operation along the channel dimension, Shift() represents a shift block, and φ A () represents an aggregate block, φ C () indicates CAB U-Net, Conv n () indicates a convolution with an n×n kernel. The superscripts f1, f2, b1, and b2 are used to distinguish different data at different stages of the processing flow.

[0029] and These represent the first positive feature and the first negative feature after the shift operation, respectively. and These are the first forward aggregation feature, the second forward aggregation feature, the first reverse aggregation feature, and the second reverse aggregation feature, respectively, after processing by the aggregation block. and They are respectively The third and fourth positive features obtained after segmentation and They are respectively The third and fourth inverse features obtained after segmentation and These represent the fourth positive feature and the fourth negative feature after shifting operations, respectively;

[0030] and Segmentation process along channel dimension and and The segmentation process is the same, and the output of the FSA module is the aggregated feature g. i .

[0031] As a further optimization of the video deblurring method based on spectral attention and feature shifting described in this invention, the operation method of the shift block is as follows: Each shift operation is performed only on the feature slice of half the channel dimension of the hidden feature. The FSA module performs a total of 4 shifts to ensure that the hidden features in both the forward and reverse directions have undergone the shift operation. After the shift operation, the shifted feature slice and the original feature slice are respectively subjected to a 3×3 convolution operation. Next, the two convolutioned feature slices are concatenated with the previously backed-up hidden features in the other direction along the channel and passed as input to the aggregation block for subsequent processing.

[0032] The aggregation block consists of two branches. The first branch includes layer normalization, residual blocks, super convolutional kernel blocks, and the Simple Gate module. This branch aggregates the shifted feature slices and the original feature slices. The second branch utilizes spectral information, converts the original features into frequency domain features using Discrete Fourier Transform (DFT), and processes them through multiple convolutional layers and activation functions. Then, the frequency domain features processed by multiple convolutional layers and activation functions are multiplied element-wise with the original features. Finally, the results of the two branches are added together to effectively fuse the hidden features and the shifted features.

[0033] The features processed by shift blocks, aggregation blocks, and convolution blocks are finally summed and aggregated features are obtained through CAB U-Net.

[0034] As a further optimization of the video deblurring method based on spectral attention and feature shift described in this invention, in step S4, the upsampling module will use the aggregated feature g obtained in step S3. i Upsampling is performed and added to the original blurred input image to obtain the final deblurred result R. i :

[0035]

[0036] in, This indicates the upsampling module.

[0037] As a further optimization of the video deblurring method based on spectral attention and feature shift described in this invention, in step S5, the loss function is the L1 Charbonnier loss function.

[0038] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0039] (1) This invention provides a method for constructing a video deblurring network based on spectral attention and feature shifting. This method improves the network's efficiency in utilizing feature information by designing a U-shaped attention mechanism structure and introducing spectral information and shifting operations. The network uses a bidirectional propagating RNN network as its basic architecture and mainly consists of a downsampling module, a SP module, an FSA module, and an upsampling module. The core modules are the SP module and the FSA module;

[0040] (2) The SP module is responsible for further feature extraction of the current frame features output by the downsampling module, and for collecting inter-frame information from adjacent frames to effectively align with the current frame. The SP module consists of CAB U-Net (Channel Attention Block U-Net) and SFM module (Spectrum Feature Match Module). CAB U-Net uses a U-shaped attention mechanism structure to replace the conventional multi-scale structure that repeats the same structure multiple times, integrating information from different scales into the same structure, which can efficiently leverage the advantages of features at different scales. The SFM module uses a simple structure to preserve and enhance associated features, and introduces frequency domain information. It uses the amplified ambiguity information in the spectrum to reduce the impact of ambiguity on feature alignment, retaining useful and correct information, and discarding invalid and inaccurate information.

[0041] (3) The FSA module can aggregate the bidirectional hidden features output by the SP module, including shift blocks and aggregation blocks. The shift blocks shift features according to different channels, increasing the richness of feature representation and expanding the receptive field of the model, thereby enhancing the network's ability to handle diverse and complex blurs; the aggregation blocks focus on the relationship between different features, using frequency domain feature information as a guide, and can efficiently aggregate shifted and unshifted hidden features, and optimize and adjust them to make them suitable for reconstructing clear frames. The FSA module avoids heavy computational costs through structural design, and can obtain excellent video deblurring results at a relatively fast processing speed. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the entire system framework of the invention.

[0043] Figure 2 This is a schematic diagram of the SP module.

[0044] Figure 3 This is a schematic diagram of the FSA module.

[0045] Figure 4 The results are the deblurred results of some blurry video frames in the test set. (a), (c), (e), and (f) are consecutive blurry input frames, and (b), (d), (f), and (h) are the deblurred results of (a), (c), (e), and (f), respectively. Detailed Implementation

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0047] PyTorch was chosen as the deep learning framework for network implementation under the Linux operating system. The hardware environment consisted of an NVIDIA GeForce RTX 4090 graphics card and an Intel Core i7-13700F CPU. The deep learning framework and hardware environment were only used to implement and test the proposed method, verifying the effectiveness of the video deblurring method proposed in this patent application. It should be understood that these examples are for illustrative purposes only and not for limiting the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0048] A video deblurring method based on spectral attention and feature shifting, the system framework diagram is as follows: Figure 1 As shown.

[0049] Specifically, the following steps are included:

[0050] Step 1: Select the public datasets GORRO and BSD (2ms-16ms) as the training and testing datasets. First, use the downsampling module to downsample the input blurred frames, reducing the width and height to 1 / 4 of the original size, and then perform preliminary feature extraction.

[0051]

[0052] Among them, I i The input is a blurred frame, where i is the frame index. Indicates the downsampling module, x i This is a downsampling feature.

[0053] Step 2: As Figure 2 The SP module is constructed to further extract features useful for deblurring from the downsampled features, thus obtaining hidden features. The SP module is divided into forward and backward propagation SP modules, both of which consist of three CAB U-Nets and two SFM modules. After passing through the SP module, a hidden feature of the current frame that retains information from adjacent frames can be obtained. During forward propagation, the hidden feature passes through the three CAB U-Nets, and between the CAB U-Nets, it also passes through the SFM module for feature matching with the hidden feature of the previous frame, as shown in the following formula:

[0054]

[0055] Where, φ C Represents CAB U-Net, φ S Represents the SFM module, u i , u i 'and Variables representing the results of intermediate operations; module outputs positive hidden features. For the positive hidden features of the previous frame, during backpropagation, Replace with It is the inverse hidden feature of the next frame, and the final output of the module is denoted as... The specific formula is as follows:

[0056]

[0057] Among them, u i , u i 'and The variable representing the result of the intermediate process of the operation is similar to that in equation (2), but the specific values ​​are different;

[0058] The CAB U-Net structure uses multiple CABs as the basic structure, and combines downsampling and upsampling operations to form a U-shaped structure. It also uses two CABs as jump connections to connect the left upsampling path and the right downsampling path.

[0059] The SFM module first concatenates the two input features along the channel dimension, merging the features together. An additional spectral branch is used to learn information in the frequency domain, enhancing the extraction and blending of effective features in the blurred image through attention. Then, this blended feature is divided into two parts and processed through a structure similar to the channel attention layer in CAB. Before the hidden features are output by SFM, a spectral branch is also added to supplement the information that is not obvious in the time domain with spectral information, assisting in feature alignment.

[0060] Step 3: As Figure 3 Construct an FSA module to aggregate the forward and reverse hidden features obtained in step 2 to obtain aggregated features;

[0061] The FSA module consists of a shift block, an aggregation block, the CAB U-Net described in step 2, and convolutional layers. The specific method of the FSA module is to use the forward and backward hidden features obtained in step 2... and As input, the two hidden features are first processed by channel segmentation, each being split into two features along the channel dimension. For example, it can be divided into two features based on the channel dimension, with the number of channels being half that of the original feature. and The subscripts (1) and (2) are used to distinguish the operations, and then the shift and aggregation operations are performed. The specific processing flow can be formulated as follows:

[0062]

[0063] Where [·] represents a tensor concatenation operation along the channel dimension, Shift represents a shift block, and φ A Represents an aggregate block, φ C Indicates CAB U-Net, Conv n This indicates a convolution with an n×n kernel. The superscripts f1, f2, b1, and b2 are used to distinguish different data at different stages of the processing flow. and Segmentation process along channel dimension and and The segmentation process is the same, and the output of the FSA module is the aggregated feature g. i ;

[0064] The shift block divides the input features into 24 groups along the channel dimension, and spatially shifts each feature slice along its width and height directions by ΔX.n ΔY n ) pixels, the empty pixels after shifting are set to 0, the subscript n is the group index, ΔX n =k×l,ΔY n = k × l, where k takes the value {-2, -1, 0, 1, 2} and l takes the value 4. Therefore, ΔX and ΔY take the values ​​{-8, -4, 0, 4, 8}. Let ΔX... n and ΔY n Different combinations of values, excluding the combination (0, 0) that does not produce pixel shift, yield 24 different combinations, which represent the shift distance of each feature slice.

[0065] After the shift operation, convolution operations are performed on the shifted feature slice and the original feature slice respectively. Next, the two convolutional feature slices are concatenated with another hidden feature along the channel and passed as input to the aggregation block for subsequent processing.

[0066] The aggregation block is mainly divided into two branches. The first branch consists of layer normalization, residual blocks, super convolutional kernel blocks, and Simple Gate. This branch aggregates the original features and the shifted features, avoiding heavy computation. The second branch utilizes spectral information, using DFT to convert the original features into frequency domain features, followed by convolutional layers and activation functions. This integrates temporal and frequency domain features, effectively determining the importance of different information and improving the module's utilization of effective information. Finally, the results of the two branches are added together, aggregating the inter-frame and intra-frame features to obtain the aggregated feature g. i .

[0067] Step 4: Combine the aggregated features g obtained in Step 3 i The input is fed into the upsampling module and added to the original input to obtain the final deblurred result R. i :

[0068]

[0069] in, This indicates the upsampling module.

[0070] Step 5: Train a video deblurring network based on spectral attention and feature shifting.

[0071] 5.1 Setting the Loss Function

[0072] The loss function is the L1 Charbonnier loss function.

[0073] 5.2 Setting Training Parameters

[0074] The training epochs were set to 600. In each epoch, all images in the training dataset were cropped into smaller images with a resolution of 256×256 and used as input to the network model. The initial learning rate was set to 0.0004, and it was halved at epochs 150, 250, 350, 450, 500, and 550. The network was iteratively trained using the Adam optimizer, with optimizer parameters β1 = 0.9, β2 = 0.999, and ε = 10. -8 .

[0075] 5.3 Evaluation Criteria

[0076] The clear image obtained after deblurring is compared with the true value, and the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values ​​are calculated.

[0077] The experimental results are as follows:

[0078] The PSNR result on the GOPRO dataset test set is 32.85, and the SSIM result is 0.938; the PSNR result on the BSD (2ms-16ms) dataset test set is 32.91, and the SSIM result is 0.932.

[0079] Figure 4 This is the result after deblurring some blurry video frames in the test set. Figure 4 In the example, (a), (c), (e), and (f) represent consecutive fuzzy input frames. Figure 4 In the diagram, (b), (d), (f), and (h) are the results after deblurring (a), (c), (e), and (f), respectively.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video deblurring method based on spectral attention and feature shifting, characterized in that, Includes the following steps: Step S1: Select a public image dataset as the ground truth, and use bilinear interpolation algorithm to downsample the input blurred image to obtain downsampled features; Step S2: For the downsampled features obtained in step S1, construct the spectrum propagation SP module to further extract the features useful for deblurring from the downsampled features, and obtain the forward and reverse hidden features; Step S3: Construct a Feature Shift Aggregation (FSA) module to aggregate forward and reverse hidden features to obtain aggregated features; Step S4: Input the aggregated features obtained in step S3 into the upsampling module and fuse them with the original input to obtain the final deblurring result; In step S2, the SP module includes a forward SP module and a reverse SP module. Both the forward SP module and the reverse SP module include 3 CAB U-Net structures and 2 SFM modules. The CAB U-Net structure uses multiple channel attention blocks (CABs) as the basic structure, and combines downsampling and upsampling operations to form a U-shaped structure. It also uses two CABs as jump connections to connect the upsampling path and the downsampling path. The SFM module first concatenates the two input features along the channel dimension, merging the features together. Then, this mixed feature is divided into two parts and processed by the channel attention layer. Two spectral branches are added to the SFM module to obtain frequency domain feature information using Discrete Fourier Transform (DFT) and process it. The spectral information is used to capture fuzzy information that is not easy to find in the time domain, which helps with feature alignment. In step S3, the FSA module includes a shift block, an aggregation block, a CAB U-Net, and a convolutional layer; The shift block is used to implement the shift operation, and the aggregation block includes two branches. The features processed by the shift block, aggregation block and convolution block are finally added together and the aggregated features are obtained through CAB U-Net.

2. The video deblurring method based on spectral attention and feature shifting according to claim 1, characterized in that, After steps S1-S4, a video deblurring network based on spectral attention and feature shift is obtained. Step S5 is included after step S4. Step S5: Set the loss function and train the video deblurring network based on spectral attention and feature shift to obtain the mapping model.

3. The video deblurring method based on spectral attention and feature shifting according to claim 1, characterized in that, In step S1, the downsampling module downsamples the original blurred image to obtain features with smaller dimensions, and initially extracts features from the blurred frames of the input blurred image, as shown in the following formula: ； in, For the input of the first Blur frame, It is a frame index. Indicates the downsampling module. For the first Downsampling features of blurred frames.

4. The video deblurring method based on spectral attention and feature shifting according to claim 3, characterized in that, In step S2, the SP module can obtain a hidden feature of the current frame that retains information from adjacent frames. The formula for the forward SP module is as follows: ； in, Representing CAB U-Net, Represents the SFM module. , , and The SP module outputs positive hidden features, representing variables that represent the results of intermediate operations. , This is the positive hidden feature of the previous frame; The formula for the reverse SP module in backpropagation is as follows: ； in, It is the inverse hidden feature of the next frame, and the final output of the inverse SP module is denoted as .

5. A video deblurring method based on spectral attention and feature shifting according to claim 4, characterized in that, In step S3, The specific method of the FSA module is as follows: The forward and reverse hidden features obtained in step S2 and Input into the FSA module, and First, the feature is copied and then segmented by channel, dividing each channel into two features; for ,Will Divide the feature into two channels, each with half the number of channels as the original feature: the first positive feature. Second positive feature Reverse hidden features After being cut Divide the feature into two features based on the channel dimension, with the number of channels being half that of the original feature: the first inverse feature. Second reverse feature The subsequent processing procedure is as follows: ； in, This represents a tensor splicing operation along the channel dimension. ( ) represents a shift block. ( ) represents an aggregate block. ( ) represents CAB U-Net, () indicates a convolution with an n×n kernel, and the superscript indicates a convolution. , , and Used to distinguish different data at different stages of the processing flow. and These represent the first positive feature and the first negative feature after the shift operation, respectively. , , and These are the first forward aggregation feature, the second forward aggregation feature, the first reverse aggregation feature, and the second reverse aggregation feature, respectively, after processing by the aggregation block. and They are respectively The third and fourth positive features obtained after segmentation and They are respectively The third and fourth inverse features obtained after segmentation and These represent the fourth positive feature and the fourth negative feature after shifting operations, respectively; and Segmentation process along channel dimension and and The segmentation process is the same, but the output of the FSA module is the aggregated feature. .

6. The video deblurring method based on spectral attention and feature shifting according to claim 5, characterized in that, The specific operation method of the shift block is as follows: Each shift operation is performed only on the feature slice of half the channel dimension of the hidden feature. The FSA module performs a total of 4 shifts to ensure that the hidden features in both the forward and reverse directions have undergone the shift operation. After the shift operation, the shifted feature slice and the original feature slice are convolved with a 3×3 kernel respectively. Next, the two convolved feature slices are concatenated with the previously backed-up hidden features in the other direction along the channel and passed as input to the aggregation block for subsequent processing. The aggregation block consists of two branches. The first branch includes layer normalization, residual blocks, super convolutional kernel blocks, and the SimpleGate module. This branch aggregates the shifted feature slices and the original feature slices. The second branch utilizes spectral information, converts the original features into frequency domain features using Discrete Fourier Transform (DFT), and processes them through multiple convolutional layers and activation functions. Then, the frequency domain features processed by multiple convolutional layers and activation functions are multiplied element-wise with the original features. Finally, the results of the two branches are added together to effectively fuse the hidden features and the shifted features. The features processed by shift blocks, aggregation blocks, and convolution blocks are finally summed and aggregated features are obtained through CAB U-Net.

7. The video deblurring method based on spectral attention and feature shifting according to claim 1, characterized in that, In step S4, the upsampling module will use the aggregated features obtained in step S3. Upsampling is performed and then added to the original blurred input image to obtain the final deblurred result. : ； in, () indicates the upsampling module.

8. A video deblurring method based on spectral attention and feature shifting according to claim 2, characterized in that, In step S5, the loss function is the L1 Charbonnier loss function.

Images