Image blind deblurring method based on local double-domain mutual attention and global channel self-attention

Abstract

The invention relates to a local double-domain mutual attention and global channel self-attention image blind deblurring method, which belongs to the technical field of computer vision, and comprises the following steps: S1, constructing a window-based frequency domain-space domain mutual attention network WFMAN; s2, constructing a global airspace channel self-attention network (GSCAN); s3, constructing a frequency domain filtering gating feature fusion module FFGF; s4, the WFMAN and the GSCAN are connected in series to form a basic module; constructing a novel U-shaped network by using the plurality of basic modules, the FFGF modules and the refining layer; s5, training and testing the novel U-shaped network by using the image data set before and after deblurring to obtain an image blind deblurring model; and S6, deblurring the collected image by using the image blind deblurring model.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to an image blind deblurring method based on local dual-domain mutual attention and global channel self-attention. Background Art

[0002] Existing dynamic scene blind deblurring methods can be mainly divided into two categories: deep learning methods based on convolutional neural networks and deep learning methods based on visual Transformer.

[0003] In recent years, convolutional neural networks have been widely used in the field of image deblurring due to their powerful representation capabilities. Xu et al. first introduced deep learning into the non-blind deblurring of blurred images and proposed a 5-layer convolutional neural network (CNN) to achieve non-blind deblurring. Tao et al. started from the coarse scale of the blurred image and gradually restored the potential image with higher resolution, proposing a scale-recurrent convolutional neural network (SRN). Cho et al. used a single U-shaped network to simulate a multi-cascade U-shaped network and introduced asymmetric feature fusion to propose a multi-input multi-output U-net (MIMO-Unet). Chen et al. proposed a simple nonlinear activation free network (NAFNet) by decomposing existing methods with good results and extracting their basic components while removing or replacing nonlinear activation functions. Mao et al. combined Fourier transform, ReLU activation function and inverse Fourier transform into the traditional convolution residual block and proposed a new residual FFT-ReLU block to achieve the fusion of spatial domain and frequency domain.

[0004] The Transformer architecture has been widely used in the field of image deblurring in recent years due to its huge receptive field, and has achieved good results. Chen et al. first applied the visual Transformer architecture to image restoration and proposed (Image Processing Transformer: IPT), which achieved good results. Inspired by U-Net, Wang et al. proposed a U-shaped image transformer architecture suitable for various image restoration tasks. Zamir et al. applied the self-attention mechanism across feature dimensions rather than spatial dimensions and introduced a gating mechanism in the feedforward neural network to screen favorable features. Li et al. proposed a new network architecture for global, regional and local modeling by fusing three attention mechanisms such as the anchor-based stripe self-attention mechanism. Based on the convolution theorem, Kong et al. proposed a Fourier Transformer that transforms the self-attention matrix multiplication into the matrix frequency domain point product, and introduced a learnable weight in the frequency domain to screen the frequency domain features of blurred images.

[0005] However, although deblurring methods based on convolutional neural networks have achieved certain success in image restoration, they still face challenges such as limited receptive field, easy overfitting, gradient vanishing, and limited ability to handle spatially varying blur. Although methods based on visual Transformers can capture long-range dependencies of images through self-attention mechanisms and improve deblurring effects, they usually require more training data, have high computational complexity and number of parameters, and may cause over-smoothing problems, limiting their applicability.

[0006] In summary, in the deblurring methods based on deep neural networks, convolution operation is the basic operation of these methods. However, the convolution operation is a spatially invariant local operation, and its receptive field is very limited. Even though many methods use larger and deeper models to compensate for the limitations of convolution, simply increasing the capacity of deep models does not always lead to better performance. On the other hand, although the Transformer architecture can compensate for the limited receptive field of convolution operations, its calculation of scaled dot product attention leads to quadratic space and time complexity, which greatly limits its scope of use. Although there are some optimization methods for the Transformer architecture, this may affect the deblurring performance. Summary of the invention

[0007] In view of this, the object of the present invention is to provide an image blind deblurring method with local dual-domain mutual attention and global channel self-attention.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for blind image deblurring based on local dual-domain mutual attention and global channel self-attention includes the following steps:

[0010] S1: Construct a window-based frequency-spatial mutual attention network (WFMAN), which has two branches: spatial domain and frequency domain. The features from the frequency domain are activated by Gaussian error linear units and multiplied as weights to the features in the spatial domain to achieve feature fusion of the mutual attention mechanism.

[0011] S2: Construct a global spatial channel self-attention network (GSCAN), which has only one channel in the spatial domain. After the global channel attention mechanism, it multiplies its own features by its own weights to perform feature fusion of the self-attention mechanism.

[0012] S3: Construct a frequency-domain filter gating feature fusion module (FFGF), which introduces a learnable quantization matrix W to adaptively determine the shallow information from the encoder to be retained, and fully fuse the information between the codec through a simple gating operation;

[0013] S4: WFMAN and GSCAN are connected in series to form a basic module; a new U-shaped network is constructed using multiple basic modules, FFGF modules and refinement layers;

[0014] S5: training and testing the novel U-type network using the image data set before and after deblurring to obtain an image blind deblurring model;

[0015] S6: Deblurring the captured image using the image blind deblurring model.

[0016] Further, the window-based frequency-spatial mutual attention network WFMAN in step S1 specifically includes:

[0017] For the input of the i-th encoder First, the layer normalization operation is performed on it, and then it is divided into two branches, each of which includes a feature extraction stage and a feature interaction stage;

[0018] The upper branch is used to operate in the spatial domain, and its feature extraction stage is described as follows: first, a 1×1 convolution and a 3×3 depth-wise convolution are performed on the features in sequence; then a simple gated SimpleGate operation is performed to split the input features into two equal parts along the channel dimension, and the two parts are directly arithmetic multiplied; then a simplified channel attention mechanism is performed on the features, the features are first globally averaged pooled, then a 1×1 convolution operation is performed, and finally the point multiplication is performed to the original features; then the features are aggregated through a 1×1 convolution operation, and finally a residual connection operation is performed to obtain the output Feature of the feature extraction part of the upper branch top :

[0019]

[0020] in Represents the input of the i-th encoder; LN() represents the layer normalization operation; Convlxl() represents the convolution operation with a convolution kernel size of 1×1; DConv3x3() represents the depth-wise convolution operation with a convolution kernel size of 3×3; SG() represents the simple gating mechanism operation; SCA() represents the simple channel attention mechanism operation;

[0021] The lower branch is used to operate in the frequency domain. Its feature extraction stage is described as follows: first, the features are divided into windows; then the window features are subjected to Fourier transform, linear layer operation, GELU nonlinear activation and inverse Fourier transform to obtain effective information about the fuzzy pattern; finally, the Fourier transformed window features are converted into features of the original size to obtain the output feature of the feature extraction part of the lower branch of WFMAN bot :

[0022]

[0023] Where FFT() represents a two-dimensional fast Fourier transform operation; Linear() represents a linear transform operation in the complex domain; GELU() represents a Gaussian error linear unit activation function; IFFT() represents a two-dimensional fast inverse Fourier transform operation;

[0024] In the feature interaction stage of the upper branch, the upper branch feature Feature top Perform layer normalization and linear mapping operations; in the feature interaction stage of the lower half branch, the lower half branch feature Feature botPerform layer normalization; then perform a 1×1 convolution and a 3×3 depth-wise convolution operation; then perform GELU nonlinear activation, use the features of the lower branch as weights, and multiply them by the features of the upper branch to implement mutual attention operation, so as to learn the frequency-space dual-domain representation to use kernel-level and pixel-level features for deblurring; finally perform a 1×1 convolution operation and a residual connection operation to obtain the output of WFMAN out :

[0025] WFMAN out =Conv1x1{Linear(Norm(Feature top ))

[0026] ⊙GELU(Dconv3x3(Conv1x1(Norm(Feature bot ))))+Feature top

[0027] Feature top Represents the output of the feature extraction part of the upper branch of WFMAN; Feature bot Represents the output of the feature extraction part of the lower branch of WFMAN; Linear represents the linear transformation operation in the real number domain.

[0028] Further, the global spatial channel self-attention network GSCAN described in step S2 specifically includes:

[0029] Feature extraction: WFMAN output WFMAN out As input to GSCAN in , first perform layer normalization operation; then perform a 1×1 convolution and a 3×3 depth-wise convolution in sequence; then perform a simple gated SimpleGate operation to split the input feature into two equal parts along the channel dimension, and directly perform arithmetic dot multiplication on the two parts; then perform a simplified channel attention mechanism on the feature, first perform a global average pooling operation on the feature, then perform a 1×1 convolution operation, and finally perform a dot multiplication on the original feature; then perform a 1×1 convolution operation to aggregate the features, and finally perform a residual connection operation to obtain the output Feature of the GSCAN feature extraction part GSCAN :

[0030] Feature GSCAN =Conv1x1(SCA(SG(DConv3x3(Conv1x1(Norm(GSCAN in )))))))+GSCAN in

[0031] Feature fusion: Feature GSCAN Input to the two input ends of the feature fusion part respectively. GSCAN , layer normalization and linear mapping operations are performed in the upper branch; layer normalization, 1×1 convolution, 3×3 depth-wise convolution, and GELU activation operations are performed in the lower branch; then the weights from the lower branch are multiplied to the upper branch to complete the fusion of the feature self and the self to achieve the effect of self-attention; finally, 1×1 convolution and residual connection are performed to obtain the output of GSCAN 0ut :

[0032] GSCAN out =Conv1x1{Linear(Norm(feature GSCAN ))

[0033] ⊙GELU(Dconv3x3(Conv1x1(Norm(Feature GSCAN ))))}+Feature GSCAN .

[0034] Furthermore, the frequency domain filtering gated feature fusion module FFGF in step S3 specifically includes:

[0035] For the output of the i-th layer encoder First, perform a 1×1 convolution on it, then perform PATCH expansion on the feature, then perform Fourier transform to transform the feature to the frequency domain, then multiply it by a quantization matrix W to adaptively select useful encoder information, and finally perform inverse Fourier transform and PATCH folding to convert the feature to the size of the input;

[0036] Then the corresponding decoder layer features with deep information are The input is concatenated with the filtered encoder layer features on the channel, and then a 1×1 convolution is used to integrate the concatenated information. Finally, SimpleGate is used to achieve effective fusion of features to obtain the FFGF output FFGF out :

[0037]

[0038] in represents the output of the i-th layer encoder, represents the output of the i+1th layer decoder; W represents the quantization matrix; Cat represents the concatenation operation along the channel dimension.

[0039] Further, in step S4, WFMAN and GSCAN are connected in series to form a basic module, which is defined as follows:

[0040]

[0041] in represents the input of the i-th layer encoder or decoder, represents the output of the i-th layer encoder or decoder; WFMAN represents the window-based frequency-spatial mutual attention network; GSCAN represents the global spatial channel self-attention network.

[0042] Further, the novel U-type network in step S4 includes a 3×3 convolution for mapping the channel to 64, three encoder layers, two downsampling operations, two upsampling operations, two FFGFs, three decoder layers, a refinement layer, and a 3×3 convolution for remapping the channel to 3; wherein:

[0043] First encoder layer EB 1 , the second encoder layer EB 2 , the first decoder layer DB 1 and the second decoder layer DB 2 They are all composed of a basic module;

[0044] The third encoder layer EB 3 and the third decoder layer DB 3 They are all composed of 4 identical basic modules stacked together;

[0045] The refinement layer consists of 2 identical basic modules stacked together;

[0046] The input of the second decoder layer

[0047] The input of the first decoder layer

[0048] The downsampling operation includes a bilinear interpolation with a scale of 0.5 and a 3×3 convolution that doubles the channel;

[0049] The upsampling operation consists of a bilinear interpolation with a scale of 2 and a 3×3 convolution that reduces the channels by a factor of two.

[0050] The beneficial effect of the present invention is that compared with the methods that have achieved remarkable results in the field of blind deblurring of dynamic scene images in recent years, the method proposed by the present invention can reconstruct clearer image quality while maintaining a simple architecture. Both objective evaluation indicators and subjective visual effects have been significantly improved. Among them, the Peak Signal to Noise Ratio (PSNR) has increased by an average of 0.20dB-2.19dB, and the Structural Similarity (SSIM) has increased by an average of 0.004-0.02.

[0051] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 Schematic diagram of the structure of the window-based frequency-spatial mutual attention network WFMAN;

[0054] Figure 2 This is a schematic diagram of the global spatial channel self-attention network GSCAN structure;

[0055] Figure 3 This is a schematic diagram of the structure of the frequency domain filtering gated feature fusion module FFGF;

[0056] Figure 4 It is a schematic diagram of the new U-shaped network structure;

[0057] Figure 5-Figure 9 All of them are comparison diagrams of the results of the present invention and the existing method. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0060] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0061] The present invention provides an image blind deblurring method of local dual-domain mutual attention and global channel self-attention. The present invention first proposes a novel U-shaped network, which is composed of a plurality of basic blocks, and each basic block is composed of two functionally specific sub-networks connected in series. The network achieves a strong image deblurring performance while maintaining a small number of entries and a low computational complexity. In order to make each basic module in the model extract image features as much as possible to achieve a good restoration effect, a window-based frequency-spatial mutual attention network (Window-basedFrequency-Spatial Mutual Attention Network: WFMAN) and a global spatial channel self-attention network (Global Spatial Channel Self-Attention Network: GSCAN) are proposed respectively. By connecting these two sub-networks in series to form the basic module of the entire model, a better feature extraction effect is achieved. Among them, the window-based frequency-spatial mutual attention network (WFMAN) has two branches in the spatial domain and the frequency domain: the features from the frequency domain will be activated by Gaussian Error Linear Unit (GELU), and then multiplied by the features in the spatial domain as weights to achieve feature fusion of the mutual attention mechanism. The global spatial channel self-attention network (GSCAN) has only one path in the spatial domain. After the global channel attention mechanism, the network will perform feature fusion of the self-attention mechanism in the form of multiplying its own weights by its own features. Finally, in order to better fuse the shallow features from the encoder and the deep features from the decoder, a simple and efficient feature fusion mechanism, Frequency-Domain Filter Gating Feature Fusion (FFGF), is designed. It adaptively controls the encoder features after Fourier transformation through the matrix W, and splices this feature with the output of the decoder in the channel dimension, and finally performs a simplified gating mechanism operation to achieve a good fusion effect.

[0062] The window-based frequency-spatial mutual attention network (WFMAN) proposed in this invention is as follows: Figure 1As shown. At present, in the field of image deblurring, existing methods often focus on the extraction of spatial domain features, while ignoring the potential connection between blurred and clear image pairs in the frequency domain. However, frequency domain information is crucial to understanding the degree and pattern of image blur, because it can reveal the frequency distribution of the image and the characteristics of the blur kernel. In addition, spatial domain information is equally important for high-quality image restoration, but existing spatial domain-based feature extraction methods, such as traditional convolutional neural networks (CNNs) or Transformer networks, usually have problems with limited receptive fields or high computational complexity. In order to solve these problems, the present invention first designs a channel attention mechanism, which reduces the computational complexity while maintaining the global receptive field. Then, a window-based frequency domain feature extraction method is designed, which divides the input features into different small windows and performs frequency domain feature extraction on the windows. Finally, a feature fusion strategy similar to mutual attention is designed, which interacts the frequency domain information and the spatial domain information in a completely new way.

[0063] like Figure 1 As shown in FIG. 1 , the window-based frequency-space mutual attention network (WFMAN) proposed in the present invention can be divided into two branches, upper and lower, which respectively represent the features from the spatial domain and the features from the frequency domain. Figure 1 The pink background part in the figure is the feature extraction part, while the green background part is the feature interaction part. First, a layer normalization operation (LN) is required to eliminate the scale differences between different channels and promote the stability of training. After the layer normalization operation, the network is divided into two branches, the upper and lower branches, which will be described in detail below.

[0064] For the upper branch, that is, the branch operating in the spatial domain, it is necessary to perform a 1×1 convolution and a 3×3 depth-wise convolution on the features in turn to enhance the expressiveness of the features. Then the SimpleGate (SG) operation is performed. By splitting the input features into two equal parts along the channel dimension and directly performing arithmetic dot multiplication on the two parts, not only the self-fusion of the features is achieved, but also nonlinear operations are introduced. Next, the channel attention mechanism is required for the features. Because the premise is to maintain low complexity and low parameter count, a simple channel attention mechanism (Simple Channel Attention: SCA) is selected. Traditional channel attention compresses spatial information into channels, and then applies multi-layer perception to it to calculate channel attention, which is used for weighted feature maps. It is observed that channel attention can be regarded as a special gating mechanism, thereby simplifying channel attention and retaining only the two most important functions: aggregating global information and channel information interaction. It is chosen to perform a global average pooling operation on the input features first, followed by a 1×1 convolution operation, and finally dot multiplying it to the original features. By introducing a simplified channel attention operation, not only global information is captured, but also the computational efficiency is high. After completing the channel attention operation, a 1×1 convolution operation will be performed to aggregate features, and finally a residual connection operation will be performed. At this point, the feature extraction part of the upper branch has been introduced, and finally the output feature of the feature extraction part of the upper branch of WFMAN can be obtained. top :

[0065]

[0066] in Represents the input of the i-th encoder; LN() represents a layer normalization operation; Convlxl() represents a convolution operation with a convolution kernel size of 1×1; DConv3x3() represents a depth-wise convolution operation with a convolution kernel size of 3×3; SG() represents a simple gating mechanism operation; SCA() represents a simple channel attention mechanism operation.

[0067] For the lower branch, that is, the branch that operates in the frequency domain, it is first necessary to divide the features into windows, because dividing the windows can not only reduce the complexity, but also focus on local features and interact with the global features of the upper branch. Then the window features are subjected to Fourier transform, linear layer operation, GELU nonlinear activation and inverse Fourier transform to obtain effective information about the blur pattern, so as to facilitate the use of kernel-level information for image deblurring operations. Finally, the Fourier transformed window features need to be converted into features of the original size. At this point, the feature extraction part of the lower branch has been introduced, and finally the output Feature of the feature extraction part of the lower branch of WFMAN can be obtained. bot :

[0068]

[0069] Among them, FFT() represents a two-dimensional fast Fourier transform operation; Linear() represents a linear transform operation in the complex domain; GELU() represents a Gaussian error linear unit activation function; IFFT() represents a two-dimensional fast inverse Fourier transform operation.

[0070] Finally, we need to use the feedforward neural network to transform the features of the upper branch (Feature top ) and the features of the lower branch (Feature bot ) for fusion. In order to better fuse the features of the upper and lower branches, the present invention designs a dual-input single-output network. For the upper branch features from the spatial domain, layer normalization and linear mapping operations are performed on them; and for the lower branch features from the frequency domain, they also need to be layer normalized first, and then a 1×1 convolution and a 3×3 depth-by-depth convolution operation need to be performed, and then the features need to be GELU nonlinearly activated. The purpose is to use the features of the lower branch as weights, so that they can be multiplied by the features of the upper branch at the end, thus completing an operation similar to mutual attention. Through such an operation, the frequency-space dual-domain representation can be learned to use kernel-level and pixel-level features for deblurring. Finally, a 1×1 convolution operation and a residual connection operation are required. Finally, the output of WFMAN can be obtained. out :

[0071]

[0072] Feature top Represents the output of the feature extraction part of the upper branch of WFMAN; Feature bot Represents the output of the feature extraction part of the lower branch of WFMAN; Linear represents the linear transformation operation in the real number domain.

[0073] In the window-based frequency-space mutual attention network (WFMAN) proposed in the present invention, the features of the spatial domain and the frequency domain are extracted respectively, and then the features are fused using an operation similar to the mutual attention mechanism to obtain the final output. Through the WFMAN of the present invention, not only the kernel-level and pixel-level features are used for image deblurring through the frequency-space dual-domain representation, but also the main modules in the model are simplified, ensuring the simplicity and efficiency of the model.

[0074] Global features are very important for understanding the overall structure and content of an image, and help maintain the consistency and naturalness of the image during the deblurring process. If only the window-based frequency-spatial mutual attention network (WFMAN) is used as the basic module of the model, the model will focus too much on local features and ignore global features. Therefore, in order to make up for the shortcomings of WFMAN in global information, a global-based spatial channel self-attention network (GSCAN) is designed in series with WFMAN, so that the image features can be extracted both locally and globally, ensuring that the model fully mines the blur information.

[0075] like Figure 2 As shown in the figure, the present invention proposes a global spatial channel self-attention network (GSCAN), which contains only one branch from the spatial domain. First, the input features are processed by feature extraction of the brown background in the figure, and then the features are input to the two input ends of the feature interaction part of the green background respectively, and finally the output of GSCAN is obtained. This design ensures that the network can extract enough global information and complements the local information that WFMAN focuses on.

[0076] For the output of WFMAN WFMAN out , which is also the input of GSCAN in First, we need to perform layer normalization (Norm) operation, followed by a 1×1 convolution and a 3×3 depth-wise convolution to enhance the feature expression capability. Then we perform SimpleGate (SG) operation and simplified channel attention operation (SCA), as well as 1×1 convolution and residual connection operation. Finally, we can get the output Feature of the GSCAN feature extraction part. GSCAN :

[0077] Feature GSCAN

[0078] =Conv1x1(SCA(SG(DConv3x3(Conv1x1(Norm(GSCAN in )))))))+GSCAN in (4)

[0079] After feature extraction, features need to be fused and enhanced. Unlike WFMAN, GSCAN only has one path in the spatial domain, so the feature GSCAN They are input into the two input ends of the feature fusion part respectively, so that the features are fused with each other, achieving the effect of self-attention. GSCAN, layer normalization and linear mapping operations are performed in the upper branch; while layer normalization, 1×1 convolution, 3×3 depth-wise convolution, and GELU activation operations are performed in the lower branch. Then the weights from the lower branch are multiplied to the upper branch to complete the fusion of the feature self and the self. Finally, 1×1 convolution and residual connection are performed to obtain the output of GSCAN GSCAN out :

[0080]

[0081] In the global spatial channel self-attention network (GSCAN) proposed in this invention, the extraction and fusion of global information are emphasized, and features are extracted only in the spatial domain, and the fusion of self and self is performed. GSCAN and the WFMAN introduced above complement each other in local and global information. By connecting these two sub-networks in series, a basic module (BasicBlock) of the whole model is formed, which can well complete the deblurring operation.

[0082] In a deblurring network using a U-shaped structure, feature fusion between the encoder layer and the corresponding decoder layer is an operation that cannot be ignored. However, some existing methods often only use simple addition or splicing operations, and do not achieve good fusion effects. Because the features of the decoder layer often have more fine information, while the features of the encoder layer are often more primitive and rough, not all encoder information and decoder information are helpful for the restoration of the potential clear image. If they are simply added or spliced, part of the effective information cannot be used, resulting in a poor deblurring effect. In order to solve the above problems, the present invention proposes a frequency domain filter gated feature fusion strategy, which introduces a learnable quantization matrix W, so as to adaptively determine which shallow information from the encoder should be retained, and also adds a SimpleGate (SG) operation, which can fully fuse the information between the codec.

[0083] like Figure 3 As shown, for the output of the i-th layer encoder First, perform a 1×1 convolution on it, and then perform PATCH expansion on the features. Here, the PATCH size is set to 8. Next, perform a Fourier transform on the features to transform the features to the frequency domain, and then multiply them by a quantization matrix W to adaptively select useful encoder information. Finally, perform an inverse Fourier transform and PATCH folding operation to convert the features to the size of the input. At this point, the effective information of the encoder has been screened. The next step is to convert the corresponding decoder layer features with deep information into The input is concatenated with the filtered encoder layer features on the channel (Concatenate: Cat), and then a 1×1 convolution is used to integrate the concatenated information. Finally, SimpleGate is used to achieve effective feature fusion. Finally, the output of FFGF can be obtained. out :

[0084]

[0085] in represents the output of the i-th layer encoder, represents the output of the i+1th layer decoder; W represents the quantization matrix; Cat represents the concatenation operation along the channel dimension.

[0086] In the frequency domain filtering gating feature fusion strategy (FFGF) proposed in this invention, a learnable quantization matrix W is used to adaptively screen useful encoder layer features, and it is spliced ​​with the decoder layer features on the channel and then a simple gating operation is performed, which fully integrates the effective information between the codec. FFGF not only achieves high-performance feature fusion, but also does not add particularly complex operations, only increases a small part of the computational complexity, and ensures the lightweight of the entire model.

[0087] In previous deep learning-based methods, the coarse-to-fine strategy has been proven to be effective for deblurring. However, in pursuit of excellent performance, these methods often achieve this by stacking the number of network layers, depth, and number of modules, which not only leads to a huge number of parameters but is also not necessarily effective. With the development of smart devices, more and more mobile devices, such as smartphones, cameras, and drones, are in urgent need of image deblurring algorithms to improve image quality. However, due to the limited computing power of mobile devices, some large models with excellent performance are not suitable for deployment on mobile devices. Therefore, it is particularly important to design a deblurring model that is both lightweight and efficient.

[0088] The overall model architecture proposed by the present invention is as follows Figure 4 As shown. In the present invention, a U-type network structure with sub-network series connection is proposed. A single-layer U-net architecture is selected, and only two downsampling and upsampling operations are performed to ensure the lightweight of the network. In addition, the proposed WFMAN and GSCAN are connected in series as the basic modules of the model, so that local and global information can be fully utilized and the network can achieve excellent performance. Finally, a refinement layer is added to the network, which can further restore the details of the image. Such a design not only meets the demand for computing efficiency of mobile devices, but also ensures the efficiency and accuracy of the deblurring task.

[0089] like Figure 4 As shown, EBi represents the i-th encoder, represents the input of the i-th encoder, Represents the output of the i-th encoder. DB i represents the i-th decoder, represents the input of the i-th decoder, Represents the output of the i-th decoder, where i∈{1, 2, 3}. BasicBlock represents the basic module of the network, which is obtained by connecting WFMAN and GSCAN in series. The definition of BasicBlock is as follows:

[0090]

[0091] in represents the input of the i-th layer encoder or decoder, represents the output of the i-th layer encoder or decoder; WFMAN represents the window-based frequency-spatial mutual attention network; GSCAN represents the global spatial channel self-attention network.

[0092] The entire U-network consists of a 3×3 convolution that maps the channels to 64, three encoder layers, two downsampling operations, two upsampling operations, two FFGFs, three decoder layers, a refinement layer, and a 3×3 convolution that remaps the channels to 3. EB 1 , EB 2 , DB 1 , DB 2 They are composed of a BasicBlock, and EB 3 , DB 3 Each layer consists of four consecutive BasicBlock stacks, and the refinement layer consists of two BasicBlock stacks. The downsampling operation includes a bilinear interpolation with a scale of 0.5 and a 3×3 convolution that expands the channel by two times; the upsampling operation includes a bilinear interpolation with a scale of 2 and a 3×3 convolution that reduces the channel by two times.

[0093] Finally, the WFMAN, GSCAN, FFGF and sub-network series U-shaped network architecture proposed in the present invention are combined to obtain the final network model. This model not only meets the requirements of lightweight and can be deployed on the mobile terminal, but also achieves excellent deblurring effect and has strong application value.

[0094] Experimental verification:

[0095] Ablation experiment: The present invention uses peak signal-to-noise ratio and structural similarity as quantitative indicators, and proves the effectiveness of the proposed WFMAN, GSCAN and FFGF through the GOPRO dataset. The unit of the number of parameters is one million.

[0096] The method proposed by the present invention includes WFMAN, GSCAN and FFGF. It should be noted that when only WFMAN is used without GSCAN, or when only GSCAN is used without WFMAN, the number of subnetworks used is doubled to achieve the effect of connecting WFMAN and GSCAN in series. When FFGF is not used, the direct addition method is used for feature fusion.

[0097] Table 1 shows the ablation experiments of various parts proposed in the present invention.

[0098] Table 1

[0099] WFMAN GSCAN FFGF PSNR(dB) MACs(G) Params.(M) √ 33.20 59.5 18.12 √ 33.08 59.25 8.94 √ √ 33.22 59.37 13.53 √ √ 33.38 62.18 18.23 √ √ 33.28 61.93 9.05 √ √ √ 33.39 62.06 13.64

[0100] From Table 1, we can see the following three points:

[0101] 1) When only the WFMAN and GSCAN proposed by the present invention are used, the PSNRs achieved are 33.20 and 33.08 respectively, while the computational complexity is kept at around 59, which is competitive among the existing SOTA methods. When WFMAN and GSCAN are connected in series and FFGF is added to the network, the PSNR of this method reaches 33.39, while the computational complexity is only 62.06 and the number of parameters is only 13.64, which has great advantages over the existing methods in terms of both model performance and complexity.

[0102] 2) When only GSCAN is used, the PSNR of the network does not have an advantage, but the number of parameters can be kept very low; when only WFMAN is used, the PSNR of the network performs well, but it brings a large number of parameters, reaching 18.12. However, if the sub-network series connection strategy proposed in the present invention is adopted, WFMAN and GSCAN are connected in series, not only does it not increase the additional computing cost, but it also achieves a slight improvement in PSNR. Most importantly, this series connection method greatly reduces the number of network parameters to 13.53, which is about 75% of the original, thus achieving an excellent balance between performance and model size.

[0103] 3) When the proposed FFGF module is added to the network, the PSNR of the network is significantly improved by 0.17 to 0.2, achieving a significant improvement. It is worth mentioning that adding FFGF only increases the number of parameters of the network by 0.11 and the computational complexity by 2.69. It can be seen that FFGF is very important to the entire network. It not only greatly improves the performance, but also requires only a low cost.

[0104] In order to demonstrate the superiority of the method proposed in the present invention in terms of quantitative indicators, the present invention is compared with fifteen existing methods on the GoPro and HIDE datasets. Specifically, all methods are trained on the GoPro training set and then tested on the GoPro test set and the HIDE test set, respectively. Table 2 shows the comparison results between the present invention and other existing methods.

[0105] Table 2

[0106]

[0107] Table 2 shows the average PSNR and average SSIM of all methods on the GoPro test set and the HIDE test set, as well as the computational complexity and number of parameters of these methods. As shown in Table 2, the method proposed in the present invention not only surpasses most of the existing methods on the GoPro dataset, but also has the lowest computational complexity and number of parameters among all methods. In order to prove the superiority of the method proposed in the present invention in subjective visual effects, the deblurring effect is compared with five existing methods. Figure 5-9 As shown in the figure, from left to right are blurred image, MIMO-UNet+, MPRNet, HINet, NAFNet, Uformer-B, clear image inferred by this method, and real clear image. It can be seen from the figure that the image deblurred by the existing method has different degrees of distortion, blur and artifacts, while the method of the present invention can not only obtain richer details and sharp edges, but also achieve the highest PSNR. Table 2 and Figure 5-9 It shows that the method proposed in the present invention surpasses the existing methods in both qualitative and quantitative indicators.

[0108] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0109] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0110] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0111] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0112] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0113] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0114] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0115] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0116] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0117] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0118] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for blind image deblurring based on local dual-domain mutual attention and global channel self-attention, characterized by: The following steps are involved: S1: Construct a window-based frequency-spatial mutual attention network WFMAN, which has two branches: spatial domain and frequency domain. The features from the frequency domain are activated by Gaussian error linear units and multiplied as weights to the features in the spatial domain to achieve feature fusion of the mutual attention mechanism. S2: Construct a global spatial channel self-attention network GSCAN, which has only one channel in the spatial domain. After the global channel attention mechanism, it multiplies its own features by its own weights to perform feature fusion of the self-attention mechanism. S3: Construct a frequency domain filter gated feature fusion module FFGF, which introduces a learnable quantization matrix W to adaptively determine the shallow information from the encoder to be retained, and fully fuse the information between the codec through a simple gating operation; S4: WFMAN and GSCAN are connected in series to form a basic module; a new U-shaped network is constructed using multiple basic modules, FFGF modules and refinement layers; S5: training and testing the novel U-type network using the image data set before and after deblurring to obtain an image blind deblurring model; S6: Deblurring the captured image using the image blind deblurring model.

2. The image blind deblurring method based on local dual-domain mutual attention and global channel self-attention according to claim 1, characterized in that: The window-based frequency-spatial mutual attention network WFMAN in step S1 specifically includes: For the input of the i-th encoder First, the layer normalization operation is performed on it, and then it is divided into two branches, each of which includes a feature extraction stage and a feature interaction stage; The upper branch is used to operate in the spatial domain, and its feature extraction stage is described as follows: first, a 1×1 convolution and a 3×3 depth-wise convolution are performed on the features in sequence; then a simple gated SimpleGate operation is performed to split the input features into two equal parts along the channel dimension, and the two parts are directly arithmetic multiplied; then a simplified channel attention mechanism is performed on the features, the features are first globally averaged pooled, then a 1×1 convolution operation is performed, and finally the point multiplication is performed to the original features; then the features are aggregated through a 1×1 convolution operation, and finally a residual connection operation is performed to obtain the output Feature of the feature extraction part of the upper branch top : in Represents the input of the i-th encoder; LN() represents the layer normalization operation; Conv1x1() represents the convolution operation with a convolution kernel size of 1×1; DConv3x3() represents the depth-wise convolution operation with a convolution kernel size of 3×3; SG() represents the simple gating mechanism operation; SCA() represents the simple channel attention mechanism operation; The lower branch is used to operate in the frequency domain. Its feature extraction stage is described as follows: first, the features are divided into windows; then the window features are subjected to Fourier transform, linear layer operation, GELU nonlinear activation and inverse Fourier transform to obtain effective information about the fuzzy pattern; finally, the Fourier transformed window features are converted into features of the original size to obtain the output feature of the feature extraction part of the lower branch of WFMAN bot : Where FFT() represents a two-dimensional fast Fourier transform operation; Linear() represents a linear transform operation in the complex domain; GELU() represents a Gaussian error linear unit activation function; IFFT() represents a two-dimensional fast inverse Fourier transform operation; In the feature interaction stage of the upper branch, the upper branch feature Feature top Perform layer normalization and linear mapping operations; in the feature interaction stage of the lower half branch, the lower half branch feature Feature bot Perform layer normalization; then perform a 1×1 convolution and a 3×3 depth-wise convolution operation; then perform GELU nonlinear activation, use the features of the lower branch as weights, and multiply them by the features of the upper branch to implement mutual attention operation, so as to learn the frequency-space dual-domain representation to use kernel-level and pixel-level features for deblurring; finally perform a 1×1 convolution operation and a residual connection operation to obtain the output of WFMAN out : WFMAN out =Conv1x1{Linear(Norm(Feature top )) ⊙GELU(Dconv3x3(Conv1x1(Norm(Feature bot ))))+Feature top Feature top Represents the output of the feature extraction part of the upper branch of WFMAN; Feature bot Represents the output of the feature extraction part of the lower branch of WFMAN; Linear represents the linear transformation operation in the real number domain.

3. The image blind deblurring method based on local dual-domain mutual attention and global channel self-attention according to claim 1, characterized in that: The global spatial channel self-attention network GSCAN described in step S2 specifically includes: Feature extraction: WFMAN output WFMAN out As input to GSCAN in , first perform layer normalization operation; then perform a 1×1 convolution and a 3×3 depth-wise convolution in sequence; then perform a simple gated SimpleGate operation to split the input feature into two equal parts along the channel dimension, and directly perform arithmetic dot multiplication on the two parts; then perform a simplified channel attention mechanism on the feature, first perform a global average pooling operation on the feature, then perform a 1×1 convolution operation, and finally perform a dot multiplication on the original feature; then perform a 1×1 convolution operation to aggregate the features, and finally perform a residual connection operation to obtain the output Feature of the GSCAN feature extraction part GSCAN : Feature GSCAN =Conv1x1(SCA(SG(DConv3x3(Conv1x1(Norm(GSCAN in ))))))+GSCAN in Feature fusion: Feature GSCAN Input to the two input ends of the feature fusion part respectively. GSCAN , layer normalization and linear mapping operations are performed in the upper branch; layer normalization, 1×1 convolution, 3×3 depth-wise convolution, and GELU activation operations are performed in the lower branch; then the weights from the lower branch are multiplied to the upper branch to complete the fusion of the feature self and the self to achieve the effect of self-attention; finally, 1×1 convolution and residual connection are performed to obtain the output of GSCAN out : GSCAN out =Conv1x1{Linear(Norm(Feature GSCAN )) ⊙GELU(Dconv3x3(Conv1x1(Norm(Feature GSCAN ))))}+Feature GSCAN 。 4. The image blind deblurring method based on local dual-domain mutual attention and global channel self-attention according to claim 1, characterized in that: The frequency domain filtering gated feature fusion module FFGF in step S3 specifically includes: For the output of the i-th layer encoder First, perform a 1×1 convolution on it, then perform PATCH expansion on the feature, then perform Fourier transform to transform the feature to the frequency domain, then multiply it by a quantization matrix W to adaptively select useful encoder information, and finally perform inverse Fourier transform and PATCH folding to convert the feature to the size of the input; Then the corresponding decoder layer features with deep information are The input is concatenated with the filtered encoder layer features on the channel, and then a 1×1 convolution is used to integrate the concatenated information. Finally, SimpleGate is used to achieve effective fusion of features to obtain the FFGF output FFGF out : in represents the output of the i-th layer encoder, represents the output of the i+1th layer decoder; W represents the quantization matrix; Cat represents the concatenation operation along the channel dimension.

5. The image blind deblurring method based on local dual-domain mutual attention and global channel self-attention according to claim 1, characterized in that: Step S4 connects WFMAN and GSCAN in series to form a basic module, which is defined as follows: in represents the input of the i-th layer encoder or decoder, represents the output of the i-th layer encoder or decoder; WFMAN represents the window-based frequency-spatial mutual attention network; GSCAN represents the global spatial channel self-attention network.

6. The image blind deblurring method based on local dual-domain mutual attention and global channel self-attention according to claim 1, characterized in that: The novel U-type network in step S4 includes a 3×3 convolution that maps the channel to 64, three encoder layers, two downsampling operations, two upsampling operations, two FFGFs, three decoder layers, a refinement layer, and a 3×3 convolution that remaps the channel to 3; wherein: The first encoder layer EB1, the second encoder layer EB2, the first decoder layer DB1 and the second decoder layer DB2 are each composed of a basic module; The third encoder layer EB3 and the third decoder layer DB3 are both composed of 4 identical basic modules stacked; The refinement layer consists of 2 identical basic modules stacked together; The input of the second decoder layer The input of the first decoder layer The downsampling operation includes a bilinear interpolation with a scale of 0.5 and a 3×3 convolution that expands the channel by two times; the upsampling operation includes a bilinear interpolation with a scale of 2 and a 3×3 convolution that reduces the channel by two times.

Images