Video deblurring method based on wavelet perception and space-frequency attention

Abstract

The invention provides a video deblurring method based on wavelet perception and space-frequency attention, and the method comprises the steps: carrying out the preprocessing of a fuzzy video frame through 3D convolution, carrying out the wavelet transformation, and obtaining first-stage low-frequency and high-frequency features; performing wavelet transform on the first-level low-frequency features to obtain second-level low-frequency and high-frequency features; the second-level low-frequency features pass through three feature shift coding modules to obtain second-level low-frequency enhanced features, and meanwhile, second-level high-frequency enhanced features are obtained through two-layer convolution enhancement; performing inverse wavelet transform on the secondary low-frequency and high-frequency enhancement characteristics, and sending the secondary low-frequency and high-frequency enhancement characteristics into a gating dynamic filtering module to obtain primary low-frequency and high-frequency primary enhancement characteristics; inputting the primary low-frequency primary enhanced features into a forward propagation module and a backward propagation module to obtain primary low-frequency final enhanced features; the first-stage high-frequency primary enhancement feature is enhanced through two-layer convolution, and a first-stage high-frequency final enhancement feature is obtained. And performing inverse wavelet transform on the first-level low-frequency and high-frequency final enhanced features to obtain reconstructed clear frame features, and fusing the reconstructed clear frame features with an original fuzzy frame through 3D convolution to obtain a final deblurring result.

Description

Technical Field

[0001] The present invention relates to the technical field of video deblurring, and in particular to a video deblurring method based on wavelet perception and space-frequency attention. Background Art

[0002] Video deblurring is a key research topic in computer vision. Blur is ubiquitous in real-world videos, primarily due to various sources, including object motion, camera shake, and scene depth variations. These intertwined blurring factors make it difficult to accurately capture the complex spatiotemporal relationships in blurry videos, posing a significant challenge to subsequent video deblurring. Accurately restoring clear video frames has important applications in a wide range of fields, including video analysis, surveillance, and film and television production. Therefore, researching effective video deblurring methods is of great practical significance.

[0003] Currently, a variety of deep learning-based video deblurring methods have been proposed to utilize the complex spatiotemporal information in the input frame sequence from different aspects. On the one hand, models based on convolutional neural networks (CNNs) usually feed the current frame and adjacent consecutive blurred frames into an encoder-decoder structure to directly restore the clear frame. On the other hand, models based on recurrent neural networks (RNNs) maintain hidden states between video frames to sequentially propagate features from the first frame to the last frame. In addition, there has been a gradual increase in recent work using Transformers to deal with the complexity of video blur (such as the spatiotemporal inhomogeneity of blur). These networks, with their powerful long-range dependency extraction capabilities of the self-attention mechanism, are able to explore better spatial features for deblurring.

[0004] However, these existing video deblurring methods still have some shortcomings. For CNN and RNN-based methods, frame alignment modules are usually introduced to achieve better deblurring effects, which leads to high model complexity. In addition, the existing RNN propagation mechanism has limited ability to temporarily transmit useful information for potential frame recovery. If the features of non-local frames are not correctly estimated, errors will accumulate during the cyclic propagation process, thereby affecting the effect of video deblurring. As for existing Transformer-based methods, they usually act in the spatial domain and their capabilities are also limited. It is difficult to better capture high-frequency information, which in turn limits the further improvement of video deblurring effects. Summary of the Invention

[0005] According to the technical problems raised above, a video deblurring method based on wavelet perception and space-frequency attention is provided. The present invention organically combines a variety of innovative technical means such as a feature shift fusion module, an effective propagation method based on wavelet perception, and a space-frequency transformer block based on wavelet perception to form an efficient and accurate video deblurring method. These technical means cooperate and work together with each other to give full play to their respective advantages, overcome the shortcomings of the existing technology, and thus achieve better video deblurring performance as a whole. Compared with existing methods, the present invention can restore clear video frames more quickly and accurately in more complex blurred scenes, has higher practicality and application value, and provides new technical ideas and solutions for the development of the field of video deblurring.

[0006] The technical means adopted in the present invention are as follows:

[0007] A video deblurring method based on wavelet perception and space-frequency attention, comprising:

[0008] S1. Select a public video deblurring dataset as a benchmark and use 3D convolution to input blurred video frame I i Perform preprocessing to obtain preprocessing features x i ;

[0009] S2, preprocess the feature x i Perform a wavelet transform to obtain the first-level low-frequency features and the first-level high-frequency features;

[0010] S3, performing wavelet transform on the first-level low-frequency features again to obtain second-level low-frequency features and second-level high-frequency features;

[0011] S4, the obtained secondary low-frequency features are continuously passed through three feature shift coding modules to obtain secondary low-frequency enhanced features, and the obtained secondary high-frequency features are enhanced by two layers of convolution to obtain secondary high-frequency enhanced features;

[0012] S5, the second-level low-frequency enhancement feature and the second-level high-frequency enhancement feature are subjected to inverse wavelet transform, and then sent to the gated dynamic filtering module to obtain the first-level low-frequency primary enhancement feature and the first-level high-frequency primary enhancement feature;

[0013] S6. Input the primary enhanced first-level low-frequency features into the forward propagation module and the backward propagation module in sequence to obtain the first-level low-frequency final enhanced features. The obtained first-level high-frequency primary enhanced features are enhanced through two layers of convolution to obtain the first-level high-frequency final enhanced features.

[0014] S7, performing inverse wavelet transform on the obtained first-level low-frequency final enhancement feature and the first-level high-frequency final enhancement feature to obtain reconstructed clear frame features;

[0015] S8, the obtained reconstructed clear frame features are subjected to a 3D convolution and then combined with the original input blurred video frame I i Fusion to obtain the final deblurred result R i .

[0016] Furthermore, the method also includes the step of jointly training the entire network using L1 Charbonnier loss and L1 loss in the frequency domain. The loss function is as follows:

[0017]

[0018] L fre =||FFT(R t )-FFT(G t )||1,

[0019] L all =L char +λL fre

[0020] Among them, R t is the recovered deblurred frame, G t is the corresponding real clear frame, λ represents the weight coefficient, which is empirically taken as 0.1, ε is a constant used to stabilize training, and FFT stands for Fourier transform.

[0021] Furthermore, the feature shift encoding module in step S4 is composed of a feature shift fusion block consisting of a series of temporal and spatial feature shift operations and a channel attention module. The feature shift fusion block uses lightweight and effective temporal feature shift and spatial feature shift to capture adjacent frame features without adopting the previous complex feature alignment module, and performs deep feature extraction through the channel attention module.

[0022] Furthermore, the feature shift fusion block utilizes lightweight and effective temporal feature shift and spatial feature shift to capture adjacent frame features, specifically including:

[0023] For temporal feature shifting, half of the feature channels (one feature group) are retained in each frame to characterize the visual appearance of the current time i, and the other half of the channels (another feature group) are moved to propagate the aggregated information between frames;

[0024] For spatial feature shifting, different sizes of displacement are used to adapt to various degrees of inter-frame motion, thereby effectively handling inter-frame misalignment.

[0025] Furthermore, the gated dynamic filtering module in step S5 includes a gating mechanism and dynamic convolution, and the generated dynamic weights are used to differentially explore useful spatial features to promote video deblurring.

[0026] Furthermore, the forward propagation module and the backward propagation module in step S6 are both composed of space-frequency transformer blocks based on wavelet perception, and both first use cross-attention operations to further fuse adjacent frame features, and then send them to a series of space-frequency self-attention blocks (SFSA) for deep feature extraction.

[0027] Furthermore, the principle of the cross attention operation (CrossAttention) is shown in the following formula:

[0028]

[0029] in, T represents the scaling factor and transposition operation; is the feature X of the current frame i Mapped by the convolutional layer, is composed of adjacent frame features X i-1 Obtained through convolutional layer mapping.

[0030] Furthermore, the space-frequency self-attention block includes spatial self-attention (SSA) and frequency self-attention (FSA), wherein the introduced frequency self-attention converts features to the frequency domain through Fourier transform, which is used to better decouple frequency information to capture local details, thereby enhancing the video deblurring effect.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. This invention provides a video deblurring method based on wavelet perception and space-frequency attention. It introduces a feature shift fusion module that uses lightweight and efficient spatiotemporal feature shifting to capture inter-frame features, eliminating the need for a complex feature alignment module. This not only simplifies the model structure and reduces its complexity, but also improves its operational efficiency, enabling faster video deblurring. It also ensures the effective capture of inter-frame feature information, providing more accurate input information for subsequent deblurring operations, thereby improving the overall performance of video deblurring.

[0033] 2. This invention provides a video deblurring method based on wavelet perception and space-frequency attention. The effective propagation method based on wavelet perception can effectively reduce this error accumulation. The wavelet perception method can better process multi-scale information in the video, enabling more accurate estimation of non-local frame features during feature propagation. This avoids error accumulation during the cyclic propagation process, thereby achieving higher-quality video restoration and improving the accuracy and reliability of video deblurring.

[0034] 3. The present invention provides a video deblurring method based on wavelet perception and space-frequency attention. A space-frequency transformer block based on wavelet perception is designed to better capture high-frequency information and thus enhance the video deblurring effect. Wavelet perception can convert video information from the spatial domain to the frequency domain, enabling better processing and analysis of high-frequency information in the video in the frequency domain. The space-frequency transformer block can fully leverage the advantages of the transformer's self-attention mechanism to better capture the complex spatiotemporal relationships and high-frequency detail information in the video in the space-frequency domain, thereby significantly enhancing the video deblurring effect, making the restored video frames clearer, more natural, and richer in details, thereby improving the overall quality of video deblurring.

[0035] Based on the above reasons, the present invention can be widely promoted in fields such as video deblurring. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 This is a flowchart of the method of the present invention.

[0038] Figure 2 This is a structural diagram of the gated dynamic filtering module of the present invention.

[0039] Figure 3 This is the structure diagram of the space-frequency self-attention (SFSA) of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0042] like Figure 1 As shown, the present invention provides a video deblurring method based on wavelet perception and space-frequency attention, comprising:

[0043] S1. Select a public video deblurring dataset as a benchmark and use 3D convolution to input blurred video frame I i Perform preprocessing to obtain preprocessing features x i ;

[0044] S2, preprocess the feature x i Perform a wavelet transform to obtain the first-level low-frequency features and the first-level high-frequency features;

[0045] S3, performing wavelet transform on the first-level low-frequency features again to obtain second-level low-frequency features and second-level high-frequency features;

[0046] S4, the obtained secondary low-frequency features are continuously passed through three feature shift coding modules to obtain secondary low-frequency enhanced features, and the obtained secondary high-frequency features are enhanced by two layers of convolution to obtain secondary high-frequency enhanced features;

[0047] S5, the second-level low-frequency enhancement feature and the second-level high-frequency enhancement feature are subjected to inverse wavelet transform, and then sent to the gated dynamic filtering module to obtain the first-level low-frequency primary enhancement feature and the first-level high-frequency primary enhancement feature;

[0048] S6. Input the primary enhanced first-level low-frequency features into the forward propagation module and the backward propagation module in sequence to obtain the first-level low-frequency final enhanced features. The obtained first-level high-frequency primary enhanced features are enhanced through two layers of convolution to obtain the first-level high-frequency final enhanced features.

[0049] S7, performing inverse wavelet transform on the obtained first-level low-frequency final enhancement feature and the first-level high-frequency final enhancement feature to obtain reconstructed clear frame features;

[0050] S8, the obtained reconstructed clear frame features are subjected to a 3D convolution and then combined with the original input blurred video frame I i Fusion to obtain the final deblurred result R i .

[0051] In specific implementation, as a preferred embodiment of the present invention, the method further includes the step of jointly training the entire network using L1 Charbonnier loss and L1 loss in the frequency domain. The loss function is as follows:

[0052]

[0053] L fre =||FFT(R t )-FFT(G t )||1,

[0054] L all =L char +λL fre

[0055] Among them, R t is the recovered deblurred frame, G t is the corresponding real clear frame, λ represents the weight coefficient, which is empirically taken as 0.1, ε is a constant used to stabilize training, and FFT stands for Fourier transform.

[0056] When implementing the invention, please refer to the preferred embodiment of the invention. Figure 1 The feature shift encoding module in step S4 is composed of a feature shift fusion block consisting of a series of temporal and spatial feature shift operations and a channel attention module. The feature shift fusion block uses lightweight and effective temporal feature shift and spatial feature shift to capture adjacent frame features without the need to use the previous complex feature alignment module, and performs deep feature extraction through the channel attention module.

[0057] When specifically implemented, as a preferred embodiment of the present invention, Figure 1 As shown in (a), the feature shift fusion block uses lightweight and effective temporal feature shift and spatial feature shift to capture adjacent frame features, specifically including:

[0058] For temporal feature shifting, half of the feature channels (one feature group) are retained in each frame to characterize the visual appearance of the current time i, and the other half of the channels (another feature group) are moved to propagate the aggregated information between frames;

[0059] For spatial feature shifting, different sizes of displacement are used to adapt to various degrees of inter-frame motion, thereby effectively handling inter-frame misalignment.

[0060] When specifically implemented, as a preferred embodiment of the present invention, Figure 2 As shown, the gated dynamic filtering module in step S5 includes a gating mechanism and dynamic convolution, and the generated dynamic weights are used to differentially explore useful spatial features to promote video deblurring.

[0061] In specific implementation, as a preferred embodiment of the present invention, the forward propagation module and the backward propagation module in step S6 are both composed of space-frequency transformer blocks based on wavelet perception, such as Figure 1 As shown in (b), the cross-attention operation is first used to further fuse the features of adjacent frames, and then sent to a series of space-frequency self-attention blocks (SFSA) for deep feature extraction.

[0062] In specific implementation, as a preferred embodiment of the present invention, the principle of the cross attention operation (CrossAttention) is shown in the following formula:

[0063]

[0064] in, T represents the scaling factor and transposition operation; is the feature X of the current frame i Mapped by the convolutional layer, is composed of adjacent frame features X i-1 Obtained through convolutional layer mapping.

[0065] In specific implementation, as a preferred embodiment of the present invention, the space-frequency self-attention block includes spatial self-attention (SSA) and frequency self-attention (FSA), such as Figure 3 As shown in the figure, the introduced frequency domain self-attention converts the features to the frequency domain through Fourier transform, which is used to better decouple the frequency information to capture local details, thereby enhancing the video deblurring effect.

[0066] In summary, the present invention designs a video deblurring method based on wavelet perception and space-frequency attention. It captures feature information between adjacent frames by utilizing lightweight and efficient spatiotemporal feature shifting, eliminating the need for the traditional complex feature alignment module. At the same time, by introducing a feature propagation method based on wavelet perception to reduce the error accumulation caused by the RNN propagation process, better video deblurring is achieved. In addition, the present invention further designs a space-frequency transformer block based on wavelet perception to better capture high-frequency information, thereby enhancing the video deblurring effect. Compared with some existing methods, the method proposed in the present invention achieves superior performance.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video deblurring method based on wavelet perception and space-frequency attention, characterized in that: include: S1. Select a public video deblurring dataset as a benchmark and use 3D convolution to input blurred video frame I i Perform preprocessing to obtain preprocessing features x i ; S2, preprocess the feature x i Perform a wavelet transform to obtain the first-level low-frequency features and the first-level high-frequency features; S3, performing wavelet transform on the first-level low-frequency features again to obtain second-level low-frequency features and second-level high-frequency features; S4, the obtained secondary low-frequency features are continuously passed through three feature shift coding modules to obtain secondary low-frequency enhanced features, and the obtained secondary high-frequency features are enhanced by two layers of convolution to obtain secondary high-frequency enhanced features; S5, the second-level low-frequency enhancement feature and the second-level high-frequency enhancement feature are subjected to inverse wavelet transform, and then sent to the gated dynamic filtering module to obtain the first-level low-frequency primary enhancement feature and the first-level high-frequency primary enhancement feature; S6. Input the obtained first-level low-frequency primary enhancement features into the forward propagation module and the backward propagation module in sequence to obtain the first-level low-frequency final enhancement features, and enhance the obtained first-level high-frequency primary enhancement features through two layers of convolution to obtain the first-level high-frequency final enhancement features; S7, performing inverse wavelet transform on the obtained first-level low-frequency final enhancement feature and the first-level high-frequency final enhancement feature to obtain reconstructed clear frame features; S8, the obtained reconstructed clear frame features are subjected to a 3D convolution and then combined with the original input blurred video frame I i Fusion to obtain the final deblurred result R i .

2. The video deblurring method based on wavelet perception and space-frequency attention according to claim 1 is characterized in that: It also includes the steps of jointly training the entire network using L1 Charbonnier loss and L1 loss in the frequency domain. The loss function is as follows: L fre =||FFT(R t )-FFT(G t )||1, THE all =L char +λL fre Among them, R t is the recovered deblurred frame, G t is the corresponding real clear frame, λ represents the weight coefficient, ε is a constant used to stabilize training, and FFT stands for Fourier transform.

3. The video deblurring method based on wavelet perception and space-frequency attention according to claim 1 is characterized in that: The feature shift encoding module in step S4 is composed of a feature shift fusion block consisting of a series of temporal and spatial feature shift operations and a channel attention module. The feature shift fusion block uses lightweight and effective temporal feature shift and spatial feature shift to capture adjacent frame features, and performs deep feature extraction through the channel attention module.

4. The video deblurring method based on wavelet perception and space-frequency attention according to claim 3 is characterized in that: The feature shift fusion block uses lightweight and effective temporal feature shift and spatial feature shift to capture inter-frame features, specifically including: For temporal feature shifting, half of the feature channels in each frame are retained to characterize the visual appearance of the current time i, and the other half of the channels are moved to propagate inter-frame aggregate information; For spatial feature shifting, different sizes of displacement are used to adapt to various degrees of inter-frame motion, thereby effectively handling inter-frame misalignment.

5. The video deblurring method based on wavelet perception and space-frequency attention according to claim 1 is characterized in that: The gated dynamic filtering module in step S5 includes a gating mechanism and dynamic convolution, and the generated dynamic weights are used to differentially explore useful spatial features to promote video deblurring.

6. The video deblurring method based on wavelet perception and space-frequency attention according to claim 1, characterized in that: The forward propagation module and the backward propagation module in step S6 are both composed of space-frequency transformer blocks based on wavelet perception, and both first use cross-attention operations to further fuse adjacent frame features, and then send them to a series of space-frequency self-attention blocks for deep feature extraction.

7. The video deblurring method based on wavelet perception and space-frequency attention according to claim 6, characterized in that: The principle of the cross attention operation is shown in the following formula: in, T represents the scaling factor and transposition operation; is the feature X of the current frame i Mapped by the convolutional layer, is composed of adjacent frame features X i-1 Obtained through convolutional layer mapping.

8. The video deblurring method based on wavelet perception and space-frequency attention according to claim 6, characterized in that: The spatial-frequency self-attention block includes spatial self-attention and frequency domain self-attention. The introduced frequency domain self-attention converts features to the frequency domain through Fourier transform, which is used to decouple frequency information to capture local details, thereby enhancing the video deblurring effect.

Images