Lightweight deblurring method based on frequency domain and spatial domain feature interaction

Abstract

The invention relates to the technical field of image deblurring and the like, in particular to a lightweight deblurring method based on frequency domain and spatial domain feature interaction, which comprises the following steps of: acquiring low-level feature embedding of an image to be restored by adopting a 1 * 1 convolutional layer; low-level features are embedded and converted into deep-level features through a U-shaped network formed by a multi-level symmetric encoder-decoder, and each level of the multi-level symmetric encoder-decoder is formed by a plurality of FSTBlocks; carrying out refining processing on the deep features under high spatial resolution, and generating a residual image for the refined features by applying a convolutional layer; and adding the residual image and the to-be-restored image to obtain a final restored image. While efficient image recovery is realized, the lightweight of the network is successfully realized, the calculation efficiency and deployment feasibility are greatly improved, and the method has the advantages in practical application.

Description

Technical Field

[0001] The present invention relates to technical fields such as deep learning, computer vision, and image deblurring, and in particular to a lightweight deblurring method based on the interaction of frequency domain and spatial domain features. Background Art

[0002] Since the convolution operation is a spatially invariant local operation, it does not model the spatial variation properties of the image content, resulting in a limited local perception range. Most existing CNN-based methods use the method of increasing network depth to compensate for the limitations of convolution, but it is still difficult to capture long-range dependencies and contains a large number of parameters and computational complexity. Most existing Transformer-based methods use self-attention of non-overlapping windows to perform single image deblurring. Although the patch strategy reduces some of the computational effort, the scaled dot product attention in these methods usually requires complex matrix multiplications, and its spatial and temporal complexity grows quadratically with the increase in image pixels.

[0003] In the image restoration task, detail information such as edges and textures is crucial to the quality of the restored image. The edges and textures in the image usually contain key information such as the shape, contour, and structure of the object. Accurately restoring these details can significantly improve the authenticity and clarity of the restored image. However, most of the existing deep learning-based methods focus on the extraction of local features in the spatial domain and ignore the capture of global information, which leads to the loss of detail information and affects the local blurring phenomenon in the image restoration process. In addition, simply relying on spatial domain convolution operations to model long-range dependencies in the image usually requires increasing the network depth and expanding the convolution kernel size to improve the receptive field, which greatly increases the number of model parameters and computational complexity, thereby having a negative impact on computational efficiency and restoration accuracy. Summary of the invention

[0004] In order to solve the problem that it is difficult to capture global and detail information and the number of model parameters is too large in the image restoration task, the present invention proposes a lightweight deblurring method based on the interaction of frequency domain and spatial domain features, which specifically includes the following steps:

[0005] A 1×1 convolutional layer is used to obtain low-level feature embedding of the image to be restored;

[0006] The low-level feature embedding is converted into deep features through a U-shaped network consisting of a 3-level symmetric encoder-decoder. Each level of the 3-level symmetric encoder-decoder consists of multiple FSTBlocks.

[0007] Refine the deep features at high spatial resolution and apply convolutional layers to the refined features to generate residual images;

[0008] The residual image is added to the image to be restored to obtain the final restored image.

[0009] Furthermore, the U-shaped network composed of multi-level symmetric encoder-decoder adopts skip connection, that is, the shallow feature information D transmitted by the encoder of the i-th layer i The low-resolution latent feature E output by the decoder of the i-th layer i Splicing to form a composite feature F i , apply 1×1 convolution to perform channel fusion on the concatenated features, then use a FSTBlock block to process the composite features, and then use 1×1 convolution to refine the features, divide the refined features into two feature vectors, add the two feature vectors element by element, and use the added features as the input of the next level decoder.

[0010] Furthermore, the data processing process of an FSTBlock block includes:

[0011] After layer normalization of the input feature map, the frequency domain feature map is extracted through frequency domain attention, and the spatial feature map is extracted through multi-head pyramid channel attention;

[0012] The frequency domain features and the spatial features are allowed to interact, that is, the pixel value with the larger pixel value in the frequency domain features and the spatial features is selected as the pixel value of the output feature map after the interaction;

[0013] The output feature map after interaction is skip-connected with the input feature map and then input into the pyramid feed-forward network. The output of the pyramid feed-forward network is used as the output of the FSTBlock block.

[0014] Furthermore, the process of extracting frequency domain feature maps through frequency domain attention includes:

[0015] After the input feature map is layer normalized, the channel average pooling layer is used to perform pooling operations on the channel dimension, and the channel features of each spatial position are averaged and aggregated to obtain a tensor with a shape of H×W×1;

[0016] Divide the H×W×1 tensor into multiple 8×8 pixel windows and apply the fast Fourier transform to each window to convert it from the spatial domain to the frequency domain.

[0017] Introducing a learnable quantization matrix W f , and the inverse method of joint picture expert group compression is used to learn the matrix to selectively retain the important frequency information in the frequency domain and obtain the filtered frequency domain features;

[0018] Apply inverse Fourier transform to the filtered frequency domain features and merge the windows to convert them into spatial domain features. Then use activation function to perform nonlinear mapping on the spatial domain features to obtain the mapped feature weights.

[0019] The feature map after layer normalization of the input feature map is weighted using the feature weights, the weighted feature map is fused with the input feature map by addition, and the fused feature map is used as the frequency domain feature map.

[0020] Furthermore, the process of extracting spatial feature maps through multi-head pyramid channel attention includes:

[0021] After layer normalization of the input feature map, the feature map is split into four vectors in the channel dimension. Each vector simulates a pyramid structure through a convolution layer with different convolution kernels to capture local information of different scales, and the local information of different scales is spliced ​​into a vector X.

[0022] Each different convolution and processed feature map is processed by the global average pooling layer and the fully connected layer respectively, and the processed feature maps are concatenated into a vector Y;

[0023] The vector Y is passed into the classifier and mapped to the probability space. The probability distribution of the classifier output is refined through a 1×1 convolution layer.

[0024] The processed probabilities are used to weight the vector X respectively, and the weighted feature map is fused with the input feature map by addition, and the fused feature map is used as the spatial feature map.

[0025] Furthermore, the pyramid feedforward network processes the input feature map as follows:

[0026] After the input feature map is subjected to layer normalization, it is linearly transformed through the first linear transformation layer and the second linear transformation layer respectively;

[0027] The output of the first linear transformation layer is processed by the pyramid convolution layer and the activation function, and then fused with the output of the second linear transformation layer by multiplication;

[0028] The fused feature map is processed by the random dropout layer and the third linear layer in sequence, and then fused with the input feature map by addition, and the fused feature map is used as the feature map output by the pyramid feedforward network.

[0029] Compared with the prior art, the method proposed in the present invention can significantly reduce the number of parameters and computational complexity while being able to restore high-quality clear images. The present invention was compared with Restormer, a representative deblurring method based on Transformer. The results show that with little impact on performance, the Peak Signal to Noise Ratio (PSNR) only decreased by 0.50dB, and the Structural Similarity Index (SSIM) only decreased by 0.005. In terms of network lightweight, the method of the present invention is particularly outstanding. The number of parameters of the present invention is only 5.9M, which is 77% less than that of Restormer, about 1 / 5 of it; the computational complexity is also reduced by 97.43MACs, a reduction of 69%. In summary, the present invention has successfully achieved network lightweighting while achieving efficient image restoration, greatly improved computational efficiency and deployment feasibility, and has advantages in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a lightweight deblurring method based on the interaction of frequency domain and spatial domain features of the present invention;

[0031] Figure 2 Schematic diagram of the frequency domain attention structure of the present invention;

[0032] Figure 3 Schematic diagram of the multi-head pyramid channel attention structure in the present invention;

[0033] Figure 4 Schematic diagram of the pyramid feedforward network structure in the present invention;

[0034] Figure 5 Schematic diagram of a defuzzification network composed of FSTBlock blocks of the present invention;

[0035] Figure 6 The comparison between the present invention and the prior art is shown in FIG. Figure 1 ;

[0036] Figure 7 The comparison between the present invention and the prior art is shown in FIG. Figure 2 ;

[0037] Figure 8 The comparison between the present invention and the prior art is shown in FIG. Figure 3 ;

[0038] Fig. 9 The comparison between the present invention and the prior art is shown in FIG. Figure 4 ;

[0039] Fig.10 The comparison between the present invention and the prior art is shown in FIG. Figure 5 . DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The present invention proposes a lightweight deblurring method based on the interaction of frequency domain and spatial domain features, such as Figure 1 , specifically including the following steps:

[0042] A 1×1 convolutional layer is used to obtain low-level feature embedding of the image to be restored;

[0043] The low-level feature embedding is converted into deep features through a U-shaped network consisting of a multi-level symmetric encoder-decoder. Each level of the multi-level symmetric encoder-decoder consists of multiple FSTBlocks.

[0044] Refine the deep features at high spatial resolution and apply convolutional layers to the refined features to generate residual images;

[0045] The residual image is added to the image to be restored to obtain the final restored image.

[0046] In order to solve the problem of difficulty in capturing global and detail information and excessive number of model parameters, the present invention first proposes a frequency-spatial domain-based feature-interaction attention module (FSA) to compensate for the loss of spatial domain detail information; by using compact frequency domain feature vectors to reduce the amount of calculation, a long-range dependency relationship is established for the blurred image, and the interaction between high-frequency detail information and spatial local features is realized; then, inspired by the multi-layer perceptron, the present invention proposes a pyramid feedforward network (PFFN), which uses pyramid convolution to perform multi-scale convolution processing on the image, and fuses multi-level convolution features, which can achieve stronger feature extraction capabilities with lower parameter quantity and computational complexity; finally, the present invention proposes a novel jump connection, inserts a frequency-spatial domain-based transformer block (FSTBlock) based on the frequency domain and spatial domain, dynamically screens the feature information transmitted by the middle layer of the encoder, and strengthens useful features, guides the network to pay more attention to key detail information, and improves the image restoration quality.

[0047] The frequency domain attention (Frequency Domain-based Attention, FA) proposed in the present invention for extracting frequency domain feature maps is as follows Figure 1 As shown in Figure 2, the process of extracting frequency domain feature maps through frequency domain attention includes:

[0048] First, the input frequency domain attention tensor x1 is first processed by the normalization layer to obtain the normalized tensor x 1L ;

[0049] Then, the channel average pooling layer is used to perform pooling operations on the channel dimension, averaging and aggregating the channel features of each spatial position to obtain a tensor x with a shape of H×W×1. p , the tensor x p Significantly reduces the number of network parameters and computational complexity, and aggregates global information;

[0050] Subsequently, the image is divided into multiple windows of 8 × 8 pixels, and the fast Fourier transform is applied to each window to convert it from the spatial domain to the frequency domain to obtain the frequency domain feature representation;

[0051] Introducing a learnable quantization matrix W f, and the inverse method of Joint Photographic Experts Group (JPEG) compression is used to learn it, so as to selectively retain the important frequency information in the frequency domain, and obtain the filtered frequency domain feature x f ;

[0052] x f Apply inverse Fourier transform and merge the windows to convert to spatial domain features x s , and then use the activation function to transform the spatial domain feature x s Perform nonlinear mapping to obtain the mapped feature weights;

[0053] Then, the normalized feature x is weighted 1L Multiplying the nonlinear transformation result of the spatial domain features strengthens the key information in the spatial domain features, while also retaining the important frequency components selected in the frequency domain;

[0054] Finally, the residual connection is introduced and added to the original input x1 to generate the final output x1 out ;

[0055] The above process can be expressed by the following formula:

[0056] x 1L =LN(x1)

[0057]

[0058]

[0059]

[0060]

[0061] Where LN(·) represents the layer normalization operation; C represents the number of channels of the feature map, B represents the batch size of the feature map, H represents the height of the feature map, and W represents the width of the feature map; Indicates window division, Indicates window merging; represents the fast Fourier transform, Represents inverse fast Fourier transform; * represents element-by-element point multiplication; Represents the GELU activation function.

[0062] The multi-head pyramid channel attention (MPCA) for extracting spatial feature maps proposed in the present invention is as follows Figure 2 As shown in Figure 2, the process of extracting spatial feature maps through multi-head pyramid channel attention includes:

[0063] The input tensor x2 of the multi-head pyramid channel attention is first processed by the normalization layer to obtain the normalized tensor x 2L , and then split into four vectors in the channel dimension: vector 1y1, vector 2y2, vector 3y3, vector 4y4;

[0064] The pyramid structure is simulated by convolution layers with different convolution kernels to capture local information of different scales. In this embodiment, vector 1 is processed by a 1×1 convolution kernel, vector 2 is processed by a 3×3 convolution kernel, vector 3 is processed by a 3×3 convolution kernel, and vector 4 is processed by a 5×5 convolution kernel.

[0065] Next, these vectors processed by different convolution kernels are concatenated into a unified vector X and stored as intermediate parameters;

[0066] In order to further improve the expressiveness of features and reduce redundant information, a simple channel attention mechanism is applied to each vector, and a global average pooling layer is used to compress spatial information onto the channel and a fully connected layer is used to interact with channel information, which improves computational efficiency and reduces model complexity.

[0067] Then, the four vectors are concatenated into a new feature vector Y;

[0068] Next, the vector Y is passed into the classifier and mapped to the probability space, and the probability distribution of the classifier output is refined through a 1×1 convolutional layer;

[0069] Finally, the output weight is multiplied by the concatenated vector X to weight the features and filter out important local information. After residual connection with the original input x2, the final output x2 is obtained. out ;

[0070] The above process can be expressed by the following formula:

[0071]

[0072] X=Cat(Conv ki×ki (y i )), ki∈{1, 3, 3, 5}, i∈{1, 2, 3, 4}

[0073] Y = Cat(w(Avgpool(Conv ki×ki (y i ))), ki∈{1, 3, 3, 5}, i∈{1, 2, 3, 4}

[0074] x2 out =Conv 1×1 (Softmax(Y))*X+x2

[0075] Among them, Cat represents the concatenation operation, w represents the fully connected layer, Avgpool represents the global average pooling, Conv ki*ki It represents the convolution operation with the convolution kernel ki×ki, and Softmax represents the classifier.

[0076] Let the frequency domain feature x1 out Provided global information and spatial domain features x2 out The local information provided by the FSA is interacted to achieve the complementarity of frequency domain and spatial domain information, and the output of FSA is obtained:

[0077] FSA out =MAX(x1 out , x2 out )

[0078] Among them, MAX means taking the maximum value operation, that is, FSA out Each pixel in takes the feature map x1 out , feature map x2 out The larger value of .

[0079] In the FSA proposed in the present invention, by extracting features in the frequency domain and the spatial domain respectively, the global and local information can be effectively integrated, thereby improving the accuracy of image restoration and the ability to restore details. To further optimize the model performance, FSA adopts a splitting and compression channel strategy, which reduces redundant calculations and parameters, and significantly reduces the complexity of the model.

[0080] The pyramid feedforward network proposed in the present invention is as follows Figure 3 As shown in FIG. 1 , in the Transformer, simply applying a general feedforward network (FFN) fails to significantly improve the deblurring effect. Therefore, the present invention proposes a pyramid feedforward network structure, which can more effectively screen and process multi-level features by introducing pyramid convolution and gating mechanisms in the general feedforward network.

[0081] The pyramid feedforward network structure proposed in this invention processes data as follows: Figure 3 As shown, specifically including:

[0082] The input x3 of the pyramid feedforward network structure is first processed by the normalization layer, and then the number of channels is expanded to twice the original through the linear transformation layer to obtain the intermediate vectors z1 and z2;

[0083] Subsequently, z1 is input into the pyramid convolution layer (PyConv), which extracts multi-level features of the image with different receptive fields. At the same time, the idea of ​​deep separable convolution is applied in the pyramid convolution to encode the information of adjacent pixel positions in the same level space, effectively learn the local image structure, and significantly reduce the computational overhead. Then, it is processed by the activation function and converted into a nonlinear feature z1. out Thereby enhancing the expressiveness and flexibility of the network;

[0084] Next, the feature z1 after nonlinear transformation out The linear transformation feature z2 is multiplied element by element to enhance the feature correlation, and then enters the random inactivation layer to effectively suppress overfitting. Finally, the linear transformation layer is applied, and the number of output channels is restored to the original value and added to the original input x3 to obtain the filtered feature information PFFN out ;

[0085] The above process can be expressed by the following formula:

[0086]

[0087] z2=l(LN(x3))

[0088] PFFN out =x3+l(z1 out *z2)

[0089] Among them, l(·) represents linear transformation and PyConv(·) represents pyramid convolution.

[0090] In the pyramid feedforward network PFFN proposed in the present invention, by introducing receptive fields of different scales, the ability to capture local features is significantly enhanced, so that multi-level information in the image can be effectively extracted. At the same time, the pyramid convolution layer combines the idea of ​​deep separable convolution, which significantly reduces the computational complexity of the network and improves the computational efficiency. The gating mechanism enhances the nonlinear expression ability of the model, dynamically and selectively transmits useful information, while the random inactivation layer plays a role in suppressing overfitting during the training process and optimizes the training and reasoning efficiency of the network. PFFN not only improves the model restoration effect, but also optimizes the network training and reasoning efficiency.

[0091] The network model of the frequency domain and spatial domain feature interaction based on Transformer (Frequency-Spatial Domain-based TransformerNet, FSTN) and the skip connection diagram proposed in this invention are as follows Figure 4As shown, ordinary skip connections directly transfer shallow feature information of the encoder to the decoder, causing the network to learn too many redundant features. The present invention inserts a FSTBlock module in the skip connection to filter the information flow.

[0092] like Figure 4 As shown in FIG, FSTBlock includes a frequency domain and spatial domain feature interaction attention module and a pyramid feedforward network as the core modules of the encoder and decoder. Refine represents a refinement module, which is composed of multiple FSTBlock modules. As a specific implementation method, Figure 4 In this embodiment, a U-shaped network consisting of a three-level symmetric encoder-decoder is used, and the deblurring process of the image specifically includes the following process:

[0093] First, a 1×1 convolutional layer is used to obtain the low-level feature embedding F0∈R of the image to be restored H×W×C ;

[0094] Next, these shallow features F0 are gradually converted into deep features F through a three-level symmetric encoder-decoder. d ∈R H ×W×C , each layer of the encoder-decoder contains multiple FSTBlock blocks, where the number of blocks gradually increases from top to bottom to maintain the computational efficiency of the network. In this embodiment, FSTBlock* ni Indicated by n i The encoding layer or decoding layer consists of FSTBlock blocks, n i ∈{2, 4, 12};

[0095] Starting from high-resolution input, the encoder hierarchically reduces the spatial size and expands the channel capacity, while the decoder uses low-resolution latent features as input and gradually recovers the high-resolution representation. During feature downsampling and upsampling, the present invention applies pixel-unshuffle and pixel-shuffle operations, respectively;

[0096] Next, the deep feature F d Refinement is performed at high spatial resolution to further enrich the information. Finally, a convolutional layer is applied to the refined features to generate a residual image, and the degraded image is added to the residual image to obtain the final restored image.

[0097] In order to further enhance the effect of the restoration process, the features of the encoder are combined with the features of the decoder through a jump connection. The present invention introduces the FSTBlock module in the jump connection, which removes redundant information by screening and fusing the features of the frequency domain and the spatial domain, thereby improving the efficiency of information transmission and achieving a more accurate image restoration effect. The specific jump connection strategies include:

[0098] First, the shallow feature information D output by the encoder of the i-th layer i The low-resolution latent feature E of the input i-th layer decoder i Splicing to form a composite feature F i ,Then, 1×1 convolution is applied to perform channel fusion on the concatenated features to supplement the detail information that may be lost in the upsampling process of the decoder.

[0099] Subsequently, the FSTBlock module is introduced to quantify and filter the frequency domain features and spatial domain features, remove redundant information, and improve the efficiency of information flow;

[0100] Next, a 1×1 convolution is applied to further refine the features and convert these valid features into Divided into two feature vectors e in the channel dimension i d i , and finally e i With d i Add element by element to get the fused features as the output of the i-th layer decoder The input is fed into the next stage of the decoder, which improves the ability to restore image details while maintaining computational efficiency;

[0101] The process can be expressed by the following formula:

[0102] F i =(Cat(D i , E i )), i∈{1,2}

[0103]

[0104]

[0105] Among them, FSTBlock represents the core module of the encoder and decoder, D i Represents the feature map output by the encoder of the i-th layer, E i Represents the feature map of the input i-th layer decoder; C i H represents the number of channels of the output feature map of the i-th layer encoder, i represents the image height of the output feature map of the i-th layer encoder, W i Represents the image width of the output feature map of the i-th layer encoder; + represents element-by-element addition.

[0106] In the U-type network and skip connection structure proposed in the present invention, by introducing the FSTBlock module, useful features are effectively screened and the accumulation of redundant information is avoided.

[0107] The present invention performs image deblurring training tasks on the GoPro dataset. The GoPro dataset contains 3214 blurred images of size 1280×720, of which 2103 are used for training and 1111 are used for testing. All experiments are trained on a PC equipped with two NVIDIA Geforce RTX 4090GPUs, and the version of PyTorch used is 1.11.0.

[0108] The present invention adopts the same loss function as MIMO-UNet (Cho SJ, Ji SW, Hong JP, et al. Rethinking coarse-to-fine approach in single image deblurring [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 4641-4650.) and uses the Adam (Kingma D P. Adam: A method for stochastic optimization [J]. arXiv preprint arXiv: 1412.6980, 2014.) optimizer with default parameters to train it. The initial value of the learning rate is 10 -3 , the minimum value is 10 -7 , and updated using the cosine annealing strategy. The training images are all set to 256×256 pixels, the batch size is set to 8, and the total number of iterations is 80w. During the training process, the present invention adopts the same data enhancement method as Restormer (SW Zamir, A. Arora, S. Khan, M. Hayat, FSKhan, M.-H. Yang, and L. Shao, "Multi-stage progressive image restoration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, Pp. 14821-14831.). In order to effectively prevent the computational complexity from being too high and difficult to capture local details, the window size is set to 8×8 pixels to ensure that the network can fully capture details on the basis of efficient calculation.

[0109] In order to demonstrate the superiority of the proposed method in quantitative indicators, the present invention is compared with representative image deblurring methods on the GoPro dataset, and PSNR and SSIM are used to evaluate the quality of the restored image. The comparison results are shown in Table 1.

[0110] Table 1 Comparison of the present invention with other existing methods

[0111]

[0112] The existing deblurring methods in the table include the DeepDeblur model from "Nah S, Kim TH, Lee KM. Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring: IEEE Computer Society, 10.1109 / CVPR.2017.35[P]. 2016."; the SVDN model from "Wang S, Zhang J, Yu X, et al. SVDN: A spatially variant degradation network for blind image super-resolution[J]. Pattern Recognition Letters, 2022, 153: 214-22."; the SRN model from "Tao X, Gao H, Shen X, et al. Scale-recurrent network for deep image deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8174-8182."; the IR-SDE model from "Luo Z, Gustafsson FK, Zhao Z, et al. Image Restoration with mean-reverting stochastic differential equations[J].arXiv preprint arXiv:2301.11699,2023.";ESTRNN model from"Zhong Z,Gao Y,Zheng Y,et al.Real-world video deblurring:A benchmark dataset and anefficient recurrent neural network[J].International Journal of ComputerVision, 2023,131(1):284-301."; DMPHN model comes from "Zhang H, Dai Y, Li H, et al.用于图像去模糊的深度堆叠分层多补丁网络[C] / / IEEE / CVF计算机视觉与模式识别会议论文集.2019:5978 - 5986.”;SPAIR模型出自“Purohit K,Suin M,Rajagopalan AN,et al.Spatially - adaptive image restoration using distortion - guided networks[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2021:2309 - 2319.”;MB2D模型出自“Park D,Kang D U,Chun S Y.Blur more to deblur better:Multi - blur2deblur for efficient video deblurring[J].arXiv preprint arXiv:2012.12507,2020.”;MPRNet模型出自“Zamir S W,Arora A,Khan S,et al.Multi - stage progressive image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:14821 - 14831.”;HINe模型出自“Chen L,Lu X,Zhang J,et al.Hinet:Half instance normalization network for image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:182 - 192.”;Uformer - B模型出自“Wang Z,Cun X,Bao J,etal.Uformer:A general u-shaped transformer for image restoration[C] / / Proceedings of the IEEE / CVF conference on computer vision and patternrecognition.2022:17683-17693.".

[0113] Table 1 shows the average PSNR and average SSIM of all methods on the GoPro test set, as well as the parameter amount and computational complexity of these methods. According to the results in Table 1, the method proposed in the present invention outperforms most existing methods on the GoPro dataset, and performs most prominently in terms of parameters and computational complexity. Compared with the best-performing Uformer-B model in the prior art, the SSIM of the present invention only dropped by 0.011, while the number of parameters was reduced by 44.98M, a reduction of about 88%, and the computational complexity was also greatly reduced by 51%. It can be seen that compared with the current representative deblurring methods, the present invention can significantly reduce the number of parameters and computational complexity without affecting the performance, reflecting its significant advantages in practical application deployment.

[0114] In order to prove the superiority of the method proposed in this invention in terms of subjective visual effect, Figures 5 to 8 The effects of the present invention and five existing image deblurring methods are compared. The first row of pictures of each image are, from left to right, blurred images, images processed by the MIMO-Unet model, images processed by the MPRNet model, and images processed by the HINet model. The second row of pictures are, from left to right, images processed by the Restormer model, images processed by the Uformer-B model, images processed by the method of the present invention, and real clear images. One to two important details of the blurred image are selected for enlargement, and the enlarged image is restored at the same time.

[0115] The image restoration quality of the method of the present invention is comparable to that of the current leading deblurring methods, and its significant advantage is that it has the least number of parameters and computational complexity, and exhibits excellent lightweight performance. This enables the method of the present invention to perform calculations more efficiently while maintaining similar image quality, and has better practical application potential, especially suitable for resource-constrained application scenarios.

[0116] In order to verify the effectiveness of the FSA and PFFN of the present invention, the present invention conducted ablation experiments on the GoPro dataset. In each ablation experiment, the image deblurring model was trained on a 256×256 image block, and each experiment was repeated 800,000 times. In order to comprehensively evaluate the performance of the proposed method, PSNR and SSIM were used as quantitative indicators to measure the quality of the restored image, while the number of multiply-accumulate operations per second (MACs) and the number of parameters (Parameters, Params) were used as indicators to measure the lightweightness of the network and the computational complexity.

[0117] Table 2 Ablation experiments of various parts proposed in the present invention

[0118] FSA PFFN PSNR(dB) SSIM(dB) Param(M) MACs(G) √ 31.38 0.947 4.4 34.02 √ 28.82 0.914 2.1 17.01 √ √ 32.42 0.956 5.9 43.81

[0119] From Table 2, we can see the following three points:

[0120] 1) When the FSTN network only contains the PFFN module, the PSNR reaches 31.38dB and the number of parameters is only 4.4M, demonstrating the excellent deblurring performance and extremely low model complexity of the module.

[0121] 2) Based on the PFFN module, the FSA module is introduced, and the PSNR is significantly improved by 1.04dB to 32.42dB. At the same time, the number of parameters is only increased by 1.5M, and the computational complexity is only increased by 9.79G. This verifies that the FSA module maintains the high efficiency and lightweight design of the model while improving the deblurring performance.

[0122] 3) The FSTN deblurring network proposed in the present invention achieves a PSNR of 32.41dB with only 5.9M parameters and 43.81G computational complexity, which is the least parameter-intensive and computationally complex method among the existing methods, and achieves lightweight network while maintaining good performance.

[0123] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lightweight deblurring method based on the interaction of frequency domain and spatial domain features, characterized in that: The specific steps include: A 1×1 convolutional layer is used to obtain low-level feature embedding of the image to be restored; The low-level feature embedding is converted into deep features through a U-shaped network consisting of a multi-level symmetric encoder-decoder. Each level of the multi-level symmetric encoder-decoder consists of multiple FSTBlock blocks. Refine the deep features at high spatial resolution and apply convolutional layers to the refined features to generate residual images; The residual image is added to the image to be restored to obtain the final restored image.

2. A lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 1, characterized in that: The number of FSTBlock blocks of the current encoder of the multi-stage symmetric encoder-decoder is greater than the number of FSTBlock blocks of the previous encoder.

3. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 1, characterized in that: The U-shaped network composed of multi-level symmetric encoder-decoder uses skip connections, that is, the shallow feature information D transmitted by the encoder of the i-th layer i The low-resolution latent feature E output by the decoder of the i-th layer i Splice to form a composite feature F i , apply 1×1 convolution to perform channel fusion on the concatenated features, then use a FSTBlock block to process the composite features, and then use 1×1 convolution to refine the features, divide the refined features into two feature vectors, add the two feature vectors element by element, and use the added features as the input of the next level decoder.

4. A lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 1 or 3, characterized in that: The data processing process of an FSTBlock block includes: After layer normalization of the input feature map, the frequency domain feature map is extracted through frequency domain attention, and the spatial feature map is extracted through multi-head pyramid channel attention; The frequency domain features and the spatial features are allowed to interact, that is, the pixel value with the larger pixel value in the frequency domain features and the spatial features is selected as the pixel value of the output feature map after the interaction; The output feature map after interaction is skip-connected with the input feature map and then input into the pyramid feed-forward network. The output of the pyramid feed-forward network is used as the output of the FSTBlock block.

5. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 4, characterized in that: The process of extracting frequency domain feature maps through frequency domain attention includes: After the input feature map is layer normalized, the channel average pooling layer is used to perform pooling operations on the channel dimension, and the channel features of each spatial position are averaged and aggregated to obtain a tensor with a shape of H×W×1; Divide the H×W×1 tensor into multiple 8×8 pixel windows and apply the fast Fourier transform to each window to convert it from the spatial domain to the frequency domain. Introducing a learnable quantization matrix W f , and the inverse method of joint picture expert group compression is used to learn the matrix to selectively retain the important frequency information in the frequency domain and obtain the filtered frequency domain features; Apply inverse Fourier transform to the filtered frequency domain features and merge the windows to convert them into spatial domain features. Then use activation function to perform nonlinear mapping on the spatial domain features to obtain the mapped feature weights. The feature map after layer normalization of the input feature map is weighted using the feature weights, the weighted feature map is fused with the input feature map by addition, and the fused feature map is used as the frequency domain feature map.

6. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 4, characterized in that: The process of extracting spatial feature maps through multi-head pyramid channel attention includes: After layer normalization of the input feature map, the feature map is split into four vectors in the channel dimension. Each vector simulates a pyramid structure through a convolution layer with different convolution kernels to capture local information of different scales, and the local information of different scales is spliced ​​into a vector X. Each different convolution and processed feature map is processed by the global average pooling layer and the fully connected layer respectively, and the processed feature maps are concatenated into a vector Y; The vector Y is passed into the classifier and mapped to the probability space. The probability distribution of the classifier output is refined through a 1×1 convolution layer. The processed probabilities are used to weight the vector X respectively, and the weighted feature maps are fused with the input feature maps by addition, and the fused feature maps are used as the spatial feature maps.

7. The lightweight deblurring method based on the interaction of frequency domain and spatial domain features according to claim 4, characterized in that: The pyramid feedforward network processes the input feature map as follows: After the input feature map is subjected to layer normalization, it is linearly transformed through the first linear transformation layer and the second linear transformation layer respectively; The output of the first linear transformation layer is processed by the pyramid convolution layer and the activation function, and then fused with the output of the second linear transformation layer by multiplication; The fused feature map is processed by the random dropout layer and the third linear layer in sequence, and then fused with the input feature map by addition, and the fused feature map is used as the feature map output by the pyramid feedforward network.

Images