A lightweight deblurring method based on the interaction of frequency domain and spatial domain features

Abstract

This invention relates to the technical field of image deblurring, and particularly to a lightweight deblurring method based on the interaction of frequency and spatial domain features. The method includes: obtaining low-level feature embeddings of the image to be restored using a 1×1 convolutional layer; converting the low-level feature embeddings into deep features through a U-shaped network composed of multiple symmetric encoders and decoders, where each level of the multi-level symmetric encoder-decoder consists of multiple FSTBlocks; refining the deep features at high spatial resolution; applying convolutional layers to the refined features to generate a residual image; and adding the residual image to the image to be restored to obtain the final restored image. This invention achieves efficient image restoration while successfully lightweighting the network, significantly improving computational efficiency and deployment feasibility, thus possessing advantages in practical applications.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, computer vision, and image deblurring, and particularly to a lightweight deblurring method based on the interaction of frequency domain and spatial domain features. Background Technology

[0002] Because convolution is a spatially invariant local operation, it does not model the spatial variation properties of image content, resulting in a limited local perceptual range. Most existing CNN-based methods compensate for the limitations of convolution by increasing network depth, but they still struggle to capture long-range dependencies and involve a large number of parameters and computational complexity. Existing Transformer-based methods mostly use self-attention with non-overlapping windows for single-image deblurring. While patching strategies reduce some computation, the scaling dot-product attention in these methods typically requires complex matrix multiplications, and their space and time complexity increases quadratically with the number of image pixels.

[0003] In image restoration tasks, details such as edges and textures are crucial for the quality of the restored image. Edges and textures in an image typically contain key information such as the shape, contour, and structure of an object; accurately restoring these details can significantly improve the realism and clarity of the restored image. However, most existing deep learning-based methods focus on extracting local features in the spatial domain, neglecting the capture of global information. This leads to the loss of detail, which in turn affects local blurring during image restoration. Furthermore, relying solely on spatial domain convolutional operations to model long-range dependencies in an image usually requires increasing network depth and expanding the convolutional kernel size to increase the receptive field, significantly increasing the number of model parameters and computational complexity, thus negatively impacting computational efficiency and restoration accuracy. Summary of the Invention

[0004] To address the challenges of capturing both global and detailed information and the excessive number of model parameters in image restoration tasks, this invention proposes a lightweight deblurring method based on the interaction of frequency and spatial domain features, specifically including the following steps:

[0005] A 1×1 convolutional layer is used to obtain the low-level feature embedding of the image to be restored;

[0006] The low-level feature embeddings are transformed into deep features through a U-shaped network consisting of a 3-level symmetric encoder-decoder. Each level of the 3-level symmetric encoder-decoder consists of multiple FSTBlocks.

[0007] Deep features are refined at high spatial resolution, and convolutional layers are applied to the refined features to generate residual images.

[0008] The residual image is added to the image to be restored to obtain the final restored image.

[0009] Furthermore, the U-shaped network composed of multi-level symmetric encoder-decoder employs skip connections, that is, the shallow feature information D from the encoder of the i-th layer... i The low-resolution latent features E of the decoder output of layer i i splicing to form composite feature F i The concatenated features are fused using 1×1 convolution, and then the composite features are processed using an FSTBlock block. Next, the features are refined using 1×1 convolution, and the refined features are divided into two feature vectors. The two feature vectors are added element by element, and the feature obtained after addition is used as the input of the next level decoder.

[0010] Furthermore, the data processing procedure of an FSTBlock block includes:

[0011] After performing layer normalization on the input feature map, frequency domain feature map is extracted through frequency domain attention, and spatial feature map is extracted through multi-head pyramid channel attention.

[0012] The frequency domain features and spatial features are interacted, that is, the pixel value with the larger pixel value between the frequency domain features and spatial features is selected as the pixel value of the output feature map after interaction.

[0013] The output feature map after interaction is connected to the input feature map via a skip connection and then fed into the pyramid feedforward network. The output of the pyramid feedforward network is used as the output of the FSTBlock block.

[0014] Furthermore, the process of extracting frequency domain feature maps through frequency domain attention includes:

[0015] After performing layer normalization on the input feature map, a channel average pooling layer is used to perform pooling operations in the channel dimension, and the channel features at each spatial location are averaged and aggregated to obtain a tensor of shape H×W×1.

[0016] Divide the H×W×1 tensor into multiple 8×8 pixel windows, and apply a Fast Fourier Transform to each window to transform it from the spatial domain to the frequency domain.

[0017] Introduce a learnable quantization matrix W f The matrix is ​​then learned using the inverse method of Joint Image Experts Group compression to selectively retain important frequency information in the frequency domain, thus obtaining the filtered frequency domain features.

[0018] The selected frequency domain features are subjected to inverse Fourier transform and the windows are merged to convert them into spatial domain features. Then, the spatial domain features are nonlinearly mapped using an activation function to obtain the mapped feature weights.

[0019] The input feature map is weighted by using feature weights after layer normalization. The weighted feature map is then fused with the input feature map by adding them together. The fused feature map is then used as the frequency domain feature map.

[0020] Furthermore, the process of extracting spatial feature maps through multi-head pyramid channel attention includes:

[0021] After performing layer normalization on the input feature map, the feature map is split into four vectors in the channel dimension. Each vector is simulated by a convolutional layer with different convolutional kernels to capture local information at different scales. The local information at different scales is then concatenated into a vector X.

[0022] Each feature map with different convolutions and processing is processed through a global average pooling layer and a fully connected layer, and the processed feature maps are concatenated into a vector Y;

[0023] The vector Y is fed into the classifier and mapped to the probability space. The probability distribution output by the classifier is then refined through a 1×1 convolutional layer.

[0024] The vector X is weighted using the processed probabilities, and the weighted feature map is fused with the input feature map by addition. The fused feature map is then used as the spatial feature map.

[0025] Furthermore, the pyramid feedforward network's processing of the input feature map includes:

[0026] After layer normalization of the input feature map, linear transformation is performed through the first linear transformation layer and the second linear transformation layer respectively.

[0027] The output of the first linear transformation layer is processed by the pyramid convolutional layer and activation function, and then fused with the output of the second linear transformation layer by multiplication.

[0028] The fused feature map is processed sequentially through a random deactivation layer and a third linear layer, and then fused with the input feature map by addition. The fused feature map is used as the output feature map of the pyramid feedforward network.

[0029] Compared with existing technologies, the method proposed in this invention significantly reduces the number of parameters and computational complexity while recovering high-quality, clear images. Comparison with Restormer, a representative Transformer-based deblurring method, shows that with minimal performance impact, the Peak Signal-to-Noise Ratio (PSNR) decreases by only 0.50 dB, and the Structural Similarity Index (SSIM) decreases by only 0.005. Furthermore, the method demonstrates remarkable lightweightness, with only 5.9M parameters, a 77% reduction compared to Restormer (approximately one-fifth of its size); computational complexity is also reduced by 97.43 MACs, a 69% reduction. In summary, this invention achieves efficient image restoration while successfully lightweighting the network, significantly improving computational efficiency and deployment feasibility, thus possessing advantages for practical applications. Attached Figure Description

[0030] Figure 1 This is a flowchart of a lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to the present invention.

[0031] Figure 2 This is a schematic diagram of the frequency domain attention structure in this invention;

[0032] Figure 3 This is a schematic diagram of the multi-headed pyramid channel attention structure in this invention;

[0033] Figure 4 This is a schematic diagram of the pyramid feedforward network structure in this invention;

[0034] Figure 5 This is a schematic diagram of the deblurring network composed of FSTBlock blocks according to the present invention;

[0035] Figure 6 This is a comparison diagram between the present invention and the prior art. Figure 1 ;

[0036] Figure 7 This is a comparison diagram between the present invention and the prior art. Figure 2 ;

[0037] Figure 8 This is a comparison diagram between the present invention and the prior art. Figure 3 ;

[0038] Figure 9 This is a comparison diagram between the present invention and the prior art. Figure 4 ;

[0039] Figure 10 This is a comparison diagram between the present invention and the prior art. Figure 5 . Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention proposes a lightweight deblurring method based on the interaction of frequency domain and spatial domain features, such as... Figure 1 Specifically, it includes the following steps:

[0042] A 1×1 convolutional layer is used to obtain the low-level feature embedding of the image to be restored;

[0043] The low-level feature embedding is transformed into deep features through a U-shaped network composed of multi-level symmetric encoder-decoder. Each level of the multi-level symmetric encoder-decoder consists of multiple FSTBlocks.

[0044] Deep features are refined at high spatial resolution, and convolutional layers are applied to the refined features to generate residual images.

[0045] The residual image is added to the image to be restored to obtain the final restored image.

[0046] To address the challenges of capturing both global and detailed information and the excessive number of model parameters, this invention first proposes a Frequency-Spatial Domain-based Feature-interaction Attention (FSA) module to compensate for the loss of spatial domain detail information. By utilizing compact frequency domain feature vectors to reduce computation, it establishes long-range dependencies for blurred images, enabling interaction between high-frequency detail information and local spatial features. Inspired by multilayer perceptrons, this invention proposes a Pyramid FeedForward Network (PFFN), which uses pyramidal convolutions to perform multi-scale convolutional processing on images, fusing multi-level convolutional features to achieve strong feature extraction capabilities with lower parameter count and computational complexity. Finally, this invention proposes a novel skip connection, inserting a Frequency-Spatial Domain-based Transformer Block (FSTBlock) module to dynamically filter feature information transmitted from the encoder's intermediate layers, strengthen useful features, guide the network to focus more on key detail information, and improve image restoration quality.

[0047] The frequency domain-based attention (FA) proposed in this invention for extracting frequency domain feature maps is as follows: Figure 1 As shown, the process of extracting frequency domain feature maps through frequency domain attention includes:

[0048] First, the input frequency domain attention tensor x1 is processed by a normalization layer to obtain the normalized tensor x. 1L ;

[0049] Then, using a channel average pooling layer, pooling operations are performed along the channel dimension to average and aggregate the channel features at each spatial location, resulting in a tensor x of shape H×W×1. p Tensor x p It significantly reduces the number of network parameters and computational complexity, and aggregates global information;

[0050] Subsequently, the image is divided into multiple 8×8 pixel windows, and a Fourier fast transform is applied to each window to transform it from the spatial domain to the frequency domain, thereby obtaining a frequency domain feature representation.

[0051] Introduce a learnable quantization matrix W fThe inverse method of Joint Photographic Experts Group (JPEG) compression is used to learn the frequency domain features x, selectively preserving important frequency information in the frequency domain. f ;

[0052] For x f Applying the inverse Fourier transform and merging the windows, it is converted into spatial domain features x. s Then, an activation function is used to apply the spatial domain features x. s Perform a nonlinear mapping to obtain the mapped feature weights;

[0053] Then, the normalized features x are weighted and averaged. 1L Multiplying the result of the nonlinear transformation of the spatial domain features enhances the key information in the spatial domain features while also preserving the important frequency components selected in the frequency domain.

[0054] Finally, a residual connection is introduced and added to the original input x1 to generate the final output x1. out ;

[0055] The above process can be represented by the following formula:

[0056] x 1L =LN(x1)

[0057]

[0058]

[0059]

[0060]

[0061] Where LN(·) represents the layer normalization operation; C represents the number of channels in the feature map; B represents the batch size of the feature map; H represents the height of the feature map; and W represents the width of the feature map. Indicates window division, Indicates window merging; Represents the Fast Fourier Transform. The asterisk (*) indicates the inverse fast Fourier transform; the asterisk (*) indicates element-wise multiplication. This represents the GELU activation function.

[0062] The Multi-Head Pyramid Channel Attention (MPCA) proposed in this invention for extracting spatial feature maps is as follows: Figure 2 As shown, the process of extracting spatial feature maps through multi-head pyramid channel attention includes:

[0063] The input tensor x2 of the multi-head pyramid channel attention is first processed by a normalization layer to obtain the normalized tensor x. 2L Then, it is split into four vectors in the channel dimension: vector 1y1, vector 2y2, vector 3y3, and vector 4y4;

[0064] The pyramid structure is simulated by using convolutional layers with different convolutional kernels to capture local information at different scales. In this embodiment, vector 1 is processed by a 1×1 convolutional kernel, vector 2 is processed by a 3×3 convolutional kernel, vector 3 is processed by a 3×3 convolutional kernel, and vector 4 is processed by a 5×5 convolutional kernel.

[0065] Next, these vectors processed by different convolution kernels are concatenated into a unified vector X and stored as intermediate parameters;

[0066] To further improve the expressive power of features and reduce redundant information, a simple channel attention mechanism is applied to each vector. A global average pooling layer is used to compress spatial information onto the channels, and a fully connected layer is used to interact with channel information, which improves computational efficiency and reduces model complexity.

[0067] Then, the four vectors are concatenated into a new feature vector Y;

[0068] Next, the vector Y is fed into the classifier and mapped to the probability space. The probability distribution output by the classifier is then refined through a 1×1 convolutional layer.

[0069] Finally, the output weights are multiplied by the concatenated vector X to weight and filter out important local information. After residual connection with the original input x2, the final output x2 is obtained. out ;

[0070] The above process can be expressed by the following formula:

[0071]

[0072] X = Cat(Conv) ki×ki (y i ), ki∈{1, 3, 3, 5}, i∈{1, 2, 3, 4}

[0073] Y = Cat(w(Avgpool(Conv)) ki×ki (y i ))), ki∈{1, 3, 3, 5}, i∈{1, 2, 3, 4}

[0074] x2 out =Conv 1×1 (Softmax(Y))*X+x2

[0075] Where Cat represents the concatenation operation, w represents the fully connected layer, Avgpool represents global average pooling, and Conv... ki*ki This represents a convolution operation with a kernel size of ki×ki, and Softmax represents the classifier.

[0076] Let the frequency domain feature x1 out Provided global information and spatial domain features x2 out By interacting with the provided local information, the frequency domain and spatial domain information are complemented to obtain the FSA output:

[0077] FSA out =MAX(x1) out x2 out )

[0078] Where MAX represents the maximum value operation, i.e., FSA. out Each pixel in the image has a feature map x1. out Feature map x2 out The larger value.

[0079] In the FSA proposed in this invention, by extracting features from the frequency domain and spatial domain respectively, global and local information can be effectively fused, thereby improving the accuracy and detail recovery capability of image restoration. To further optimize model performance, FSA employs a channel splitting and compression strategy, reducing redundant computations and the number of parameters, and significantly lowering the model's complexity.

[0080] The pyramid feedforward network proposed in this invention is as follows: Figure 3 As shown, simply applying a general feedforward network (FFN) in the Transformer fails to significantly improve the deblurring effect. Therefore, this invention proposes a pyramid feedforward network structure, which, by introducing pyramid convolution and gating mechanisms into the general feedforward network, can more effectively filter and process multi-level features.

[0081] The data processing procedure of the pyramid feedforward network structure proposed in this invention is as follows: Figure 3 As shown, it specifically includes:

[0082] The input x3 of the pyramid feedforward network structure is first processed by a normalization layer, and then the number of channels is expanded to twice the original number by a linear transformation layer to obtain intermediate vectors z1 and z2.

[0083] Subsequently, z1 is fed into a Pyramid Convolution Layer (PyConv) to extract multi-level features of the image using different receptive fields. Simultaneously, the concept of depthwise separable convolution is applied within the pyramid convolution to encode information about the spatially adjacent pixel positions at the same level, effectively learning the local image structure and significantly reducing computational overhead. After processing with an activation function, it is then transformed into a non-linear feature z1. out This enhances the expressive power and flexibility of the network;

[0084] Next, the feature z1 after nonlinear transformation out The feature z2 after linear transformation is multiplied element-wise to enhance feature correlation. Then, it enters a random deactivation layer to effectively suppress overfitting. Finally, a linear transformation layer is applied, restoring the number of output channels to their original value, and this value is added to the original input x3 to obtain the filtered feature information PFFN. out ;

[0085] The above process can be expressed by the following formula:

[0086]

[0087] z2=l(LN(x3))

[0088] PFFN out =x3+l(z1) out *z2)

[0089] Where l(·) represents a linear transformation and PyConv(·) represents a pyramid convolution.

[0090] In the Pyramid Feedforward Network (PFFN) proposed in this invention, the ability to capture local features is significantly enhanced by introducing receptive fields of different scales, thereby effectively extracting multi-level information from images. Simultaneously, the pyramid convolutional layers, combined with depthwise separable convolution, significantly reduce the network's computational complexity and improve computational efficiency. The gating mechanism enhances the model's nonlinear expressive power, dynamically and selectively transmitting useful information, while the random deactivation layer suppresses overfitting during training, optimizing the network's training and inference efficiency. PFFN not only improves model recovery performance but also optimizes network training and inference efficiency.

[0091] The proposed frequency-spatial domain-based TransformerNet (FSTN) network model and skip connection diagram are shown below. Figure 4As shown, ordinary skip connections directly pass the shallow feature information of the encoder to the decoder, causing the network to learn too many redundant features. This invention inserts the FSTBlock module into the skip connections to filter the information flow.

[0092] like Figure 4 As shown, FSTBlock includes a frequency domain and spatial domain feature interaction attention module and a pyramid feedforward network, serving as the core modules of the encoder and decoder. The refine module, composed of multiple FSTBlock modules, is used as a specific implementation example. Figure 4 This embodiment uses a U-shaped network consisting of a 3-level symmetric encoder and decoder. The image deblurring process specifically includes the following steps:

[0093] First, a 1×1 convolutional layer is used to obtain the low-level feature embedding F0∈R of the image to be restored. H×W×C ;

[0094] Next, these shallow features F0 are progressively transformed into deep features F by a 3-level symmetric encoder-decoder. d ∈R H ×W×C Each layer's encoder-decoder contains multiple FSTBlock blocks, with the number of blocks gradually increasing from top to bottom to maintain the network's computational efficiency. In this embodiment, FSTBlock* ni Indicates that n i The encoding or decoding layer consists of n FSTBlock blocks. i ∈{2, 4, 12};

[0095] Starting with high-resolution input, the encoder reduces spatial size and expands channel capacity in layers, while the decoder uses low-resolution latent features as input to gradually recover high-resolution representation. In feature downsampling and upsampling, this invention applies pixel-free shuffling and pixel shuffling operations respectively.

[0096] Next, the deep features F d The image is then refined at high spatial resolution to further enrich the information. Finally, a convolutional layer is applied to the refined features to generate a residual image, and the degraded image is added to the residual image to obtain the final restored image.

[0097] To further enhance the restoration process, encoder features are combined with decoder features through skip connections. This invention introduces the FSTBlock module into the skip connections, which filters and fuses features from the frequency and spatial domains to remove redundant information, thereby improving information transmission efficiency and achieving more accurate image restoration. Specific skip connection strategies include:

[0098] First, the shallow feature information D output by the encoder of layer i is... i The low-resolution latent features E input to the i-th layer decoder i splicing together to form composite feature F i Then, a 1×1 convolution is applied to perform channel fusion on the concatenated features to supplement the details that the decoder may have lost during the upsampling process.

[0099] Subsequently, the FSTBlock module was introduced to quantize and filter frequency domain features and spatial domain features, remove redundant information, and improve the efficiency of information flow.

[0100] Next, 1×1 convolutions are applied to further refine the features, and these effective features are then processed. In terms of channel dimension, it is divided into two feature vectors e i d i Finally, e i With d i The elements are added one by one to obtain the fused features, which are then used as the output of the i-th layer decoder. The input to the next stage of the decoder improves image detail recovery while maintaining computational efficiency;

[0101] This process can be represented by the following formula:

[0102] F i =(Cat(D) i E i ), i∈{1,2}

[0103]

[0104]

[0105] Here, FSTBlock represents the core module of the encoder and decoder, D i E represents the feature map output by the encoder at layer i. i C represents the feature map input to the i-th layer decoder; i H represents the number of channels in the output feature map of the i-th layer encoder. i W represents the image height of the output feature map of the i-th encoder layer. i This represents the image width of the output feature map of the i-th layer encoder; + indicates element-wise addition.

[0106] In the U-shaped network and skip connection structure proposed in this invention, the introduction of the FSTBlock module effectively filters useful features and avoids the accumulation of redundant information.

[0107] This invention utilizes the GoPro dataset for image deblurring training. The GoPro dataset contains 3214 blurred images of size 1280×720, of which 2103 were used for training and 1111 for testing. All experiments were trained on a PC equipped with two NVIDIA GeForce RTX 4090 GPUs, using PyTorch version 1.11.0.

[0108] This invention employs the same loss function as MIMO-UNet (Cho SJ, Ji SW, Hong JP, et al. Rethinking coarse-to-fine approach in single image deblurring[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 4641-4650.) and trains it using the Adam optimizer with default parameters (Kingma DP. Adam: A method for stochastic optimization[J]. arXiv preprint arXiv: 1412.6980, 2014.). The initial learning rate is 10. -3 The minimum value is 10 -7 The images were updated using a cosine annealing strategy. Training images were all set to 256×256 pixels, with a batch size of 8 and a total of 800,000 iterations. This invention employs the same data augmentation method as Restormer (SWZamir, A. Arora, S. Khan, M. Hayat, FSKhan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, Pp. 14821-14831.) during training. To effectively prevent excessive computational complexity and difficulty in capturing local details, the window size was set to 8×8 pixels, ensuring the network can capture sufficient details while maintaining efficient computation.

[0109] To demonstrate the superiority of the proposed method in terms of quantitative metrics, this invention was compared with representative image deblurring methods on the GoPro dataset, and PSNR and SSIM were used to evaluate the quality of the restored images. The comparison results are shown in Table 1.

[0110] Table 1 Comparison of the present invention with other existing methods

[0111]

[0112] The existing deblurring methods listed in the table are as follows: DeepDeblur model from "Nah S, Kim TH, Lee KM. Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring: IEEE Computer Society, 10.1109 / CVPR. 2017.35[P]. 2016."; SVDN model from "Wang S, Zhang J, Yu X, et al. SVDN: A spatially variant degradation network for blindimage super-resolution[J]. Pattern Recognition Letters, 2022, 153: 214-22."; SRN model from "Tao X, Gao H, Shen X, et al. Scale-recurrent network for deep image deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8174-8182."; IR-SDE model from "Luo Z, Gustafsson FK, Zhao Z, et al. Image Restoration with mean-reverting stochastic differential equations[J].arXiv preprint arXiv:2301.11699,2023.";ESTRNN model from "Zhong Z,Gao Y,Zheng Y,et al.Real-world video deblurring:A benchmark dataset and anefficient recurrent neural network[J].International Journal of ComputerVision, 2023,131(1):284-301."; DMPHN model comes from "Zhang H, Dai Y, Li H, et al.用于图像去模糊的深度堆叠分层多补丁网络[C] / / IEEE / CVF计算机视觉与模式识别会议论文集.2019:5978 - 5986.”;SPAIR模型出自“Purohit K,Suin M,Rajagopalan AN,et al.Spatially - adaptive image restoration using distortion - guided networks[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2021:2309 - 2319.”;MB2D模型出自“Park D,Kang D U,Chun S Y.Blur more to deblur better:Multi - blur2deblur for efficient video deblurring[J].arXiv preprint arXiv:2012.12507,2020.”;MPRNet模型出自“Zamir S W,Arora A,Khan S,et al.Multi - stage progressive image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:14821 - 14831.”;HINe模型出自“Chen L,Lu X,Zhang J,et al.Hinet:Half instance normalization network for image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:182 - 192.”;Uformer - B模型出自“Wang Z,Cun X,Bao J,etal.Uformer:A general u-shaped transformer for image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and patternrecognition.2022:17683-17693.".

[0113] Table 1 shows the average PSNR and average SSIM of all methods on the GoPro test set, as well as the number of parameters and computational complexity of these methods. According to the results in Table 1, the method proposed in this invention outperforms most existing methods on the GoPro dataset, and its performance is most outstanding in terms of parameters and computational complexity. Compared with the best-performing Uformer-B model in the prior art, the SSIM of this invention decreases by only 0.011, while the number of parameters is reduced by 44.98M, a reduction of approximately 88%, and the computational complexity is also significantly reduced by 51%. Therefore, compared with current representative deblurring methods, this invention can significantly reduce the number of parameters and computational complexity with almost no impact on performance, demonstrating a significant advantage in practical application deployment.

[0114] To demonstrate the superiority of the method proposed in this invention in terms of subjective visual effect Figures 5-8 The paper compares the effectiveness of this invention with five existing image deblurring methods. The first row of images for each image, from left to right, shows the blurred image, the image processed by the MIMO-Unet model, the image processed by the MPRNet model, and the image processed by the HINet model. The second row of images, from left to right, shows the image processed by the Restormer model, the image processed by the Uformer-B model, the image processed by the method of this invention, and the clear image. One or two important details in the blurred image are selected for magnification, and the magnified image is then restored.

[0115] The method of this invention achieves image restoration quality comparable to current leading deblurring methods, with a significant advantage in having the fewest parameters and computational complexity, demonstrating excellent lightweight performance. This allows the method of this invention to perform computation more efficiently while maintaining similar image quality, possessing greater potential for practical applications, and is particularly suitable for resource-constrained application scenarios.

[0116] To verify the effectiveness of FSA and PFFN in this invention, ablation experiments were conducted on the GoPro dataset. In each ablation experiment, the image deblurring model was trained on 256×256 image patches, and each experiment underwent 800,000 iterations. To comprehensively evaluate the performance of the proposed method, PSNR and SSIM were used as quantitative metrics to measure the quality of the restored image, while Multiply-Accumulate Operations per Second (MACs) and Parameters (Params) were used as metrics to measure the network's lightweight nature and computational complexity.

[0117] Table 2 Ablation experiments of each part proposed in this invention

[0118] FSA PFFN PSNR (dB) SSIM(dB) Param(M) MACs(G) √ 31.38 0.947 4.4 34.02 √ 28.82 0.914 2.1 17.01 √ √ 32.42 0.956 5.9 43.81

[0119] The following three points can be observed from Table 2:

[0120] 1) When the FSTN network contains only the PFFN module, the PSNR reaches 31.38dB and the number of parameters is only 4.4M, demonstrating the module's excellent deblurring performance and extremely low model complexity.

[0121] 2) By introducing the FSA module on the basis of the PFFN module, the PSNR was significantly improved by 1.04dB, reaching 32.42dB, while the number of parameters increased by only 1.5M and the computational complexity increased by only 9.79G. This verifies that the FSA module can improve the deblurring performance while maintaining the efficiency and lightweight design of the model.

[0122] 3) The FSTN deblurring network proposed in this invention achieves a PSNR of 32.41dB with only 5.9M parameters and a computational complexity of only 43.81G. This performance is the lowest among existing methods in terms of the number of parameters and computational complexity, achieving lightweighting of the network while maintaining good performance.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight deblurring method based on the interaction of frequency domain and spatial domain features, characterized in that, Specifically, the following steps are included: A 1×1 convolutional layer is used to obtain the low-level feature embedding of the image to be restored; A U-shaped network consisting of multi-level symmetric encoders and decoders transforms low-level feature embeddings into deep features. Each level of the multi-level symmetric encoder-decoder consists of multiple FSTBlock blocks. The number of FSTBlock blocks in the current level encoder is greater than the number in the previous level. Skip connections are used in the U-shaped network, meaning that the shallow feature information D from the encoder in the i-th layer... i The low-resolution latent features E of the decoder output of layer i i splicing to form composite feature F i The concatenated features are fused using 1×1 convolution, and then the composite features are processed using an FSTBlock block. Next, the features are refined using 1×1 convolution, and the refined features are divided into two feature vectors. The two feature vectors are added element by element, and the feature obtained after addition is used as the input of the next level decoder. The data processing procedure of an FSTBlock block includes: After layer normalization of the input feature map, frequency domain feature maps are extracted through frequency domain attention, and spatial feature maps are extracted through multi-head pyramid channel attention. The process of extracting frequency domain feature maps through frequency domain attention includes: After performing layer normalization on the input feature map, a channel average pooling layer is used to perform pooling operations in the channel dimension, and the channel features at each spatial location are averaged and aggregated to obtain a tensor of shape H×W×1. Divide the H×W×1 tensor into multiple 8×8 pixel windows, and apply a Fast Fourier Transform to each window to transform it from the spatial domain to the frequency domain. Introduce a learnable quantization matrix W f The matrix is ​​then learned using the inverse method of Joint Image Experts Group compression to selectively retain important frequency information in the frequency domain, thus obtaining the filtered frequency domain features. The selected frequency domain features are subjected to inverse Fourier transform and the windows are merged to convert them into spatial domain features. Then, the spatial domain features are nonlinearly mapped using an activation function to obtain the mapped feature weights. The input feature map is weighted after layer normalization using feature weights. The weighted feature map is then fused with the input feature map by addition, and the fused feature map is used as the frequency domain feature map. The frequency domain features and spatial features are interacted, that is, the pixel value with the larger pixel value in the frequency domain features and spatial features is selected as the pixel value of the output feature map after interaction. The output feature map after interaction is skipped to the input feature map and then fed into the pyramid feedforward network. The output of the pyramid feedforward network is used as the output of the FSTBlock block. Deep features are refined at high spatial resolution, and convolutional layers are applied to the refined features to generate residual images. The residual image is added to the image to be restored to obtain the final restored image.

2. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 1, characterized in that, The process of extracting spatial feature maps through multi-head pyramid channel attention includes: After performing layer normalization on the input feature map, the feature map is split into four vectors in the channel dimension. Each vector is simulated by a convolutional layer with different convolutional kernels to capture local information at different scales. The local information at different scales is then concatenated into a vector X. Each feature map with different convolutions and processing is processed through a global average pooling layer and a fully connected layer, and the processed feature maps are concatenated into a vector Y; The vector Y is fed into the classifier and mapped to the probability space. The probability distribution output by the classifier is then refined through a 1×1 convolutional layer. The vector X is weighted using the processed probabilities, and the weighted feature map is fused with the input feature map by addition. The fused feature map is then used as the spatial feature map.

3. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 1, characterized in that, The processing steps of the input feature map by the pyramid feedforward network include: After layer normalization of the input feature map, linear transformation is performed through the first linear transformation layer and the second linear transformation layer respectively. The output of the first linear transformation layer is processed by the pyramid convolutional layer and activation function, and then fused with the output of the second linear transformation layer by multiplication. The fused feature map is processed sequentially through a random deactivation layer and a third linear layer, and then fused with the input feature map by addition. The fused feature map is used as the output feature map of the pyramid feedforward network.

Images