Image deblurring method based on combination of CNN (Convolutional Neural Network) and Transform

Abstract

The invention provides an image deblurring method based on combination of a CNN (Convolutional Neural Network) and a Transform. According to the method, a training set, a verification set and a test set are prepared by using a public data set; in an encoder stage, inverse residual fast Fourier transform is adopted to extract features of an input blurred image, and a pixel attention feature fusion module is used. A channel modulation self-attention mechanism and a mixed scale feedforward network are introduced into the decoder part, global context modeling is achieved, and more information is obtained. In addition, a same-scale feature fusion module is also applied in the decoder, so that the hierarchical information of the image is learned and the feature representation is enriched. In the method, a small U-shaped network is used for multi-scale learning in the first layer of the decoder and in the feature enhancement network. And finally, combining the enhanced features with the original image to generate a clear deblurred image. According to the method, by optimizing model parameters and introducing multiple innovative mechanisms, the image deblurring effect and the generalization and robustness of the model are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image restoration, and in particular to an image deblurring method based on the combination of CNN and Transformer. Background Art

[0002] Image deblurring is a classic challenge in the field of computer vision, which aims to remove blur effects to restore clear visual data. Image blurring problems are caused by multiple factors, such as device vibration, inaccurate focus, or fast target movement, which will lead to a decrease in image clarity, have a significant negative impact on the perception, and pose a series of obstacles to subsequent visual processing tasks such as object recognition, target tracking, and image analysis. For this problem, exploring efficient methods to correct image blur and regain high-resolution images has become a key scientific research issue that needs to be solved in this field.

[0003] With the development of deep learning, convolutional neural networks (CNNs) play an important role in computer vision tasks. Current CNN-based methods adopt an end-to-end approach and design different frameworks to learn the relationship between blurred and clear images without estimating the blur kernel, and can directly restore clear images from blurred images. However, due to the limited receptive field of the convolution operator, CNN cannot model long-distance pixel dependencies. Unlike the convolution operation that models local features, Transformers can scale dot products to promote long-range feature learning and model global context. They have been proven to be effective methods in many advanced visual tasks, and Transformer-based methods have achieved better performance than CNN-based methods. The self-attention mechanism of Transformer cannot capture local blur features well, and local blur images often exist in real blurry images, resulting in missing details in deblurred images. In addition, the current deblurring Transformer network has high computational complexity and large number of parameters. It is a challenging task to achieve the best balance between power and efficiency. Summary of the invention

[0004] The purpose of the present invention is to propose an image deblurring model that can fully learn the hierarchical information in the image, propose multi-scale learning, and restore the details and edge information of the image. In order to achieve the above-mentioned invention purpose, the present invention provides the following technical solutions:

[0005] An image deblurring method based on the combination of CNN and Transformer, comprising:

[0006] (1) Prepare deblurred image training set, validation set and test set based on the public dataset;

[0007] (2) The encoder uses the inverse residual fast Fourier transform (IRFFT) to extract the features of the input blurred image. After two downsamplings and three layers of encoder, three different scale features are extracted.

[0008] (3) Use the pixel-based attention feature fusion module PAFFM to fuse the encoder features at three different scales;

[0009] (4) Introducing channel modulated self-attention (CMSA) and mixed-scale feedforward network (MSFN) in the Transformer module of the decoder to model the global context of features and obtain global information;

[0010] (5) In the decoder, the pixel attention feature fusion module PAFFM is used to fuse the features fused by the three-layer encoder with the features after upsampling to learn the hierarchical information of the image and enrich the features;

[0011] (6) In the first layer of the decoder and the feature enhancement network, a small U-shaped network is used to reduce the resolution of the reconstructed features for multi-scale learning to further enhance the detail information;

[0012] (7) Merge the enhanced features with the original image to obtain a deblurred image;

[0013] Furthermore, in step (1), a public blurred image dataset GOPRO is obtained, including 2000 pairs of training datasets, 103 pairs of verification datasets and 1111 pairs of test datasets.

[0014] Furthermore, in step (2), given the blurred image Iblur, it first undergoes a 3×3 convolution to change its number of channels as the input of the first layer encoder Where C is the number of channels, H and W are the height and width of the image respectively; the output features are obtained by IRFFT After two downsamplings, the feature resolution is reduced and the number of channels is increased. The fuzzy features are extracted by IRFFT to obtain three different scale features.

[0015] Furthermore, in the inverse residual fast Fourier transform (IRFFT), the features go through three streams. The process of the left stream is as follows: the features are first applied with fast Fourier transform to obtain frequency domain features, and then the number of channels is increased by 1×1 convolution and then passed through the nonlinear activation function GELU. Then, the number of channels is changed back to the original number by 1×1 convolution and the inverse fast Fourier transform is applied to restore to the pixel-level features. The formula of the left stream of IRFFT is as follows:

[0016]

[0017] in is the fast Fourier transform, is the inverse fast Fourier transform, GELU(·) is the nonlinear activation function;

[0018] The intermediate flow process is as follows: the feature passes through a normalization layer after increasing the number of channels through a 1×1 convolution, and then passes through a 3×3 depth convolution and then returns to the original number of channels through a GELU activation function and a 1×1 convolution; the IRFFT intermediate flow formula is as follows:

[0019]

[0020] Where LN(·) is layer normalization, It is a 3×3 depth convolution

[0021] The right stream process is as follows: the features are convolved with Do-Conv3×3 deep hyperparameters, more parameters are used to accelerate training, and better performance is achieved by using deep convolution to enhance the convolution layer; the output features are obtained through GELU nonlinear activation function and Do-Conv3×3, and the IRFFT right stream formula is as follows:

[0022]

[0023] The overall IRFFT formula is as follows:

[0024]

[0025] Furthermore, in step (3), the three features are resized and feature fusion is performed using a pixel-attention feature fusion module PAFFM, where pixel attention is similar to channel attention, but without the pooling layer, expressed as:

[0026] PA(X)=X*σ(W p2 max(0,W p1 (X)))

[0027] Where σ is the nonlinear activation function Sigmoid, W*1 and W*2 are fully connected layers, ReLU is used between the two fully connected layers, and * is the channel-level product operation;

[0028] In addition, PAFFM also adopts the separation and recombination strategy, assuming that the input feature is F∈R 2C×H×W , divide the feature map equally along the channel dimension to obtain two feature maps F1 and F2 of the same shape, then divide one of the feature maps F1 equally along the channel dimension using the same operation to obtain F11 and F12, and then perform element-by-element multiplication to restore them to the original shape, and obtain a new feature map and multiply it element-by-element with F2 to obtain the output feature.

[0029] Furthermore, in step (4), the decoder Transformer module consists of a channel modulated self-attention (CMSA) and a mixed-scale feedforward network (MSFN), where CMSA uses dilated neighborhood attention (DiNA) and a cross-channel learner (CCL); DiNA can be used for local and global learning by adjusting the dilation factor without introducing additional complexity; given a feature map X∈R n×d , where n is the number of tokens, d is the dimension; Q and K are the query and key; and the relative position deviation B(i,j) between two tokens i and j. Given the expansion factor δ and the neighborhood size k, the attention weight of the i-th token is defined as:

[0030]

[0031] The corresponding matrix V i (k,δ) Its elements are the k adjacent values ​​of the i-th mark, expressed as follows:

[0032]

[0033] The DiNA output feature map of the i-th token is implemented in the following way:

[0034]

[0035] CCL is essentially a lightweight channel attention module that adaptively adjusts the output feature map of the self-attention layer according to the normalized input data; given a feature map tensor X∈R after the normalization layer C×H×W , CCL is calculated as follows:

[0036]

[0037] in represents 2D adaptive average pooling, and the output size is a 1×1 tensor; f is the compression of the C×1×1 matrix to produce a 1×C matrix; the overall CSMA formula is as follows:

[0038] CMSA(X)=DiNA(X)⊙CCL(X),

[0039] where ⊙ denotes element-wise multiplication.

[0040] Furthermore, the mixed-scale feedforward network MSFN inserts two multi-scale deep convolution paths during the transmission process. Given an input tensor X l-1 ∈R C×H×W, after layer normalization, 1×1 convolution is used to expand the channel dimension in a ratio of r, and then it is input into two parallel branches; in the feature transformation process, 3×3 and 5×5 depth convolutions are used to enhance multi-scale local information extraction; in this way, the entire feature fusion process of the developed MSFN can be expressed as:

[0041]

[0042] Where σ represents the ReLU activation function, and denotes 3×3 and 5×5 depthwise convolutions; [·] is channel concatenation.

[0043] Furthermore, in step (5), the features output by the third layer encoder are first After the third layer decoder performs feature reconstruction, local and global feature enhancement, and then upsampling is performed to obtain the output The fused encoder three-layer feature feature2 and Fusion, fully learning the image layer information, and then after upsampling once more to obtain Fuse again with the fused encoder three-layer feature feature1;

[0044] Furthermore, in step (6), a small U-shaped network is used in the first layer of the decoder and the feature enhancement network. The small U-shaped network adopts a two-layer structure, the encoder is a Transformer block, and the decoder is an IRFFT. The reconstructed features are reduced in resolution for multi-scale learning, the details and edge contours of the image are restored, and the visual effect of the image is enhanced.

[0045] Furthermore, in step (7), the enhanced features are subjected to 3×3 convolution to reduce the number of channels and obtain the reference image I R ∈R 3×H×W , combined with the original blurred image Iblur, to obtain a clear image after deblurring, as shown below:

[0046] I deblur =I blur +I R ,

[0047] Where Ideblur is the deblurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is the overall structure diagram of the image deblurring model based on the combination of CNN and Transformer of the present invention;

[0049] Figure 2It is a structural diagram of IRFFT (inverted residual fast Fourier transform) of the present invention;

[0050] Figure 3 It is a structural diagram of PAFFM (pixel attention feature fusion module) of the present invention;

[0051] Figure 4 The internal structure of the Transformer block of the present invention includes CMSA (channel modulation self-attention) and MSFN (mixed-scale feed-forward network) structure diagrams;

[0052] Figure 5 It is a small U-shaped network structure diagram of the present invention;

[0053] Figure 6 This is a visual comparison diagram of the present invention on the GoPro dataset;

[0054] Figure 7 This is a visual comparison diagram of the present invention on the HIDE dataset;

[0055] Figure 8 This is a visual comparison chart of the present invention on the RealBlur dataset. DETAILED DESCRIPTION

[0056] Here, exemplary embodiments will be described in detail, and unless otherwise indicated, the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples consistent with some aspects of the present disclosure as detailed in the appended claims.

[0057] like Figure 1 As shown in the figure, the present invention provides an image deblurring model based on the combination of CNN and Transformer. Given a blurred image Iblur, it is first subjected to a 3×3 convolution to change its channel number as the input of the first layer encoder. Where C is the number of channels, H and W are the height and width of the image respectively. The output features are obtained by IRFFT After two downsamplings, the feature resolution is reduced and the number of channels is increased. The fuzzy features are extracted by IRFFT to obtain three different scale features. After the third layer decoder and upsampling, we get The three-layer features of the fused encoder and Fusion, fully learning the image layer information, and then after upsampling once more to obtain And the fused encoder three-layer feature feature1∈R C×H×W The first layer of the decoder and the feature enhancement network use a small U-shaped network. The small U-shaped network adopts a two-layer structure. The encoder is a Transformer block and the decoder is an IRFFT. The reconstructed features are reduced in resolution for multi-scale learning to restore the details and edge contours of the image and enhance the visual effect of the image. The enhanced features are subjected to a 3×3 convolution to reduce the number of channels to obtain the reference image I R ∈R 3×H×W , and merged with the original blurred image Iblur to obtain a clear image after deblurring.

[0058] like Figure 2 As shown in the figure, the inverse residual fast Fourier transform IRFFT feature goes through three branches. The process of the left stream is: the feature first applies fast Fourier transform to obtain frequency domain features, then passes through the nonlinear activation function GELU after 1×1 convolution to increase the number of channels, and then passes through 1×1 convolution to return to the original number of channels and apply inverse fast Fourier transform to restore to pixel-level features. The formula of the left stream of IRFFT is as follows:

[0059]

[0060] in is the fast Fourier transform, is the inverse fast Fourier transform, and GELU(·) is the nonlinear activation function.

[0061] The intermediate flow process is as follows: the feature passes through a 1×1 convolution to increase the number of channels, then passes through a normalization layer, passes through a 3×3 depth convolution, and then passes through a GELU activation function and a 1×1 convolution to return to the original number of channels. The IRFFT intermediate flow formula is as follows:

[0062]

[0063] Where LN(·) is layer normalization, It is a 3×3 depth convolution.

[0064] The right stream process is as follows: the features are convolved with Do-Conv3×3 deep hyperparameters, more parameters are used to accelerate training, and better performance is achieved by using deep convolution to enhance the convolution layer. The output features are obtained through the GELU nonlinear activation function and Do-Conv3×3, and the IRFFT right stream formula is as follows:

[0065]

[0066] The overall IRFFT formula is as follows:

[0067]

[0068] like Figure 3 As shown in Figure 1, features of different scales are fused based on the pixel attention feature fusion module PAFFM, where pixel attention is similar to channel attention, but without the pooling layer, expressed as:

[0069] PA(X)=X*σ(W p2 max(0,W p1 (X))),

[0070] Among them, σ is the nonlinear activation function Sigmoid, W*1 and W*2 are fully connected layers, ReLU is used between the two fully connected layers, and * is the channel-level product operation.

[0071] The shapes of feature1 and feature2 obtained by fusion of the three features are different. Specifically, feature2 needs to be fused with the features after upsampling in the second layer, and feature1 needs to be fused with the features after upsampling in the first layer. They are expressed as follows:

[0072]

[0073] in feature1∈R C×H×W .

[0074] like Figure 4 As shown in Figure 1, the Transformer module consists of a channel modulated self-attention CMSA and a mixed scale feedforward network MSFN. CMSA uses dilated neighborhood attention DiNA and a cross-channel learner CCL. DiNA can be used for local and global learning by adjusting the dilation factor without introducing additional complexity. Given a feature map X∈R n×d , where n is the number of tokens and d is the dimension. Q and K are the query and key; and the relative position deviation B(i,j) between two tokens i and j, given the expansion factor δ and the neighborhood size k, the attention weight of the i-th token is defined as:

[0075]

[0076] The corresponding matrix V i (k,δ) Its elements are the k adjacent values ​​of the i-th mark, expressed as follows:

[0077]

[0078] The DiNA output feature map of the i-th token is implemented in the following way:

[0079]

[0080] CCL is essentially a lightweight channel attention module that adaptively adjusts the output feature map of the self-attention layer according to the normalized input data. Given a feature map tensor X∈R after the normalization layer C×H×W , CCL is calculated as follows:

[0081]

[0082] in represents 2D adaptive average pooling, and the output size is 1×1 tensor. f is the compression of C×1×1 matrix to produce 1×C matrix. The overall CSMA formula is as follows:

[0083] CMSA(X)=DiNA(X)⊙CCL(X),

[0084] where ⊙ denotes element-wise multiplication.

[0085] The mixed-scale feedforward network MSFN inserts two multi-scale deep convolution paths during the transmission process. Given an input tensor X l-1 ∈R C×H×W , after layer normalization, 1×1 convolution is used to expand the channel dimension with a ratio of r, and then it is input into two parallel branches. In the feature transformation process, 3×3 and 5×5 depth convolutions are used to enhance multi-scale local information extraction. In this way, the entire feature fusion process of the developed MSFN can be expressed as:

[0086]

[0087] Where σ represents the ReLU activation function, and denotes 3×3 and 5×5 depthwise convolutions. [·] is channel concatenation.

[0088] like Figure 5 As shown, the small U-shaped network adopts a two-layer encoder-decoder structure, the encoder is a Transformer block, and the decoder is IRFFT. Given the feature F1∈R C×H×W , after the first layer encoder captures local and global features, and then downsamples to obtain Through PAFFM, the two features are fused to obtain FEATURE1∈R C×H×W , After upsampling in the second layer of the decoder, Fusion with FEATURE1. Enhanced features are obtained after IRFFT of the first layer.

[0089] like Figure 6 As shown in the figure, the visual comparison between the present invention and other models on the GoPro dataset shows that the deblurred images generated by other methods still have blurred parts, lose image information, and obvious artifacts can still be seen in the license plate number, affecting the overall perception of the image. The license plate number image of the method in this paper is clearer, without artifacts, and the visual effect is closer to a clear image.

[0090] like Figure 7 As shown in the figure, the visual comparison between the present invention and other models on the HIDE dataset shows that compared with other deblurring methods, the method proposed in this article performs well in generating deblurred facial images. The eyebrows, eyes and other parts of the generated facial images are clearer, and the facial details can be better preserved, making the overall appearance more realistic.

[0091] like Figure 8 As shown in FIG. 1 , a visual comparison between the present invention and other models on the RealBlur_R and RealBlur_J datasets shows that the images generated by the proposed method under low-light conditions significantly enhance the clarity of the characters and have finer structural details.

[0092] The above is only one of the specific implementations of the present invention, but the technical features of the present invention are not limited thereto. Any modification, equivalent replacement or improvement made within the technical concept of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image deblurring method based on the combination of CNN and Transformer, characterized in that: include: (1) Prepare deblurred image training set, validation set and test set based on the public dataset; (2) The encoder uses the inverse residual fast Fourier transform (IRFFT) to extract the features of the input blurred image. After two downsamplings and three layers of encoder, three different scale features are extracted. (3) Use the pixel-based attention feature fusion module PAFFM to fuse the encoder features at three different scales; (4) Introducing channel modulated self-attention (CMSA) and mixed-scale feedforward network (MSFN) in the Transformer module of the decoder to model the global context of features and obtain global information; (5) In the decoder, the pixel attention feature fusion module PAFFM is used to fuse the features fused by the three-layer encoder with the features after upsampling to learn the hierarchical information of the image and enrich the features; (6) In the first layer of the decoder and the feature enhancement network, a small U-shaped network is used to reduce the resolution of the reconstructed features for multi-scale learning to further enhance the detail information; (7) The enhanced features are combined with the original image to obtain the deblurred image.

2. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (1), a public blurred image dataset GOPRO is obtained, including 2000 pairs of training datasets, 103 pairs of verification datasets, and 1111 pairs of test datasets.

3. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (2), given the blurred image Iblur, it first undergoes a 3×3 convolution to change its number of channels as the input of the first layer encoder Where C is the number of channels, H and W are the height and width of the image respectively; the output features are obtained by IRFFT After two downsamplings, the feature resolution is reduced and the number of channels is increased. The fuzzy features are extracted by IRFFT to obtain three different scale features.

4. The image deblurring method based on the combination of CNN and Transformer according to claim 3, characterized in that: In the inverse residual fast Fourier transform (IRFFT), the features go through three streams. The process of the left stream is as follows: the features are first applied with fast Fourier transform to obtain frequency domain features, and then the number of channels is increased by 1×1 convolution and then passed through the nonlinear activation function GELU. Then, the number of channels is changed back to the original number by 1×1 convolution and the inverse fast Fourier transform is applied to restore to the pixel-level features. The formula of the left stream of IRFFT is as follows: in is the fast Fourier transform, is the inverse fast Fourier transform, GELU(·) is the nonlinear activation function; The intermediate flow process is as follows: the feature passes through a 1×1 convolution to increase the number of channels, then passes through a normalization layer, passes through a 3×3 depth convolution, and then passes through a GELU activation function and a 1×1 convolution to return to the original number of channels; The IRFFT intermediate flow formula is as follows: Where LN(·) is layer normalization, It is a 3×3 depth convolution; The right stream process is as follows: the features are convolved with Do-Conv3×3 deep hyperparameters, more parameters are used to accelerate training, and better performance is achieved by using deep convolution to enhance the convolution layer; the output features are obtained through GELU nonlinear activation function and Do-Conv3×3, and the IRFFT right stream formula is as follows: The overall IRFFT formula is as follows:

5. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (3), the three features are resized and feature fusion is performed using the pixel attention feature fusion module PAFFM, where pixel attention is similar to channel attention, but without the pooling layer, expressed as: PA(X)=X*σ(W p2 max(0,W p1 (X))), Where σ is the nonlinear activation function Sigmoid, W*1 and W*2 are fully connected layers, ReLU is used between the two fully connected layers, and * is the channel-level product operation; In addition, PAFFM also adopts the separation and recombination strategy, assuming that the input feature is F∈R 2C×H×W , divide the feature map equally along the channel dimension to obtain two feature maps F1 and F2 of the same shape, then divide one of the feature maps F1 equally along the channel dimension using the same operation to obtain F11 and F12, and then perform element-by-element multiplication to restore them to the original shape, and obtain a new feature map and multiply it element-by-element with F2 to obtain the output feature.

6. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (4), the decoder Transformer module consists of a channel modulated self-attention CMSA and a mixed-scale feedforward network MSFN, where CMSA adopts dilated neighborhood attention DiNA and a cross-channel learner CCL; DiNA can be used for local and global learning by adjusting the dilation factor without introducing additional complexity; given a feature map X∈R n×d , where n is the number of tokens, d is the dimension; Q and K are query and key; As well as the relative position deviation B(i,j) between two tags i and j, given the expansion factor δ and the neighborhood size k, the attention weight of the i-th tag is defined as: The corresponding matrix Its elements are the k adjacent values ​​of the i-th mark, expressed as follows: The DiNA output feature map of the i-th token is implemented in the following way: CCL is essentially a lightweight channel attention module that adaptively adjusts the output feature map of the self-attention layer according to the normalized input data; given a feature map tensor X∈R after the normalization layer C×H×W , CCL is calculated as follows: in represents 2D adaptive average pooling, and the output size is a 1×1 tensor; f is the compression of the C×1×1 matrix to produce a 1×C matrix; the overall CSMA formula is as follows: CMSA(X)=DiNA(X)⊙CCL(X), where ⊙ denotes element-wise multiplication.

7. The image deblurring method based on the combination of CNN and Transformer according to claim 6, characterized in that: The mixed-scale feedforward network MSFN inserts two multi-scale deep convolution paths during the transmission process. Given an input tensor X l-1 ∈R C×H×W , after layer normalization, the channel dimension is expanded by a 1×1 convolution with a ratio of r, and then input into two parallel branches; In the feature transformation process, 3×3 and 5×5 deep convolutions are used to enhance multi-scale local information extraction; thus, the entire feature fusion process of the developed MSFN can be expressed as: Where σ represents the ReLU activation function, and denotes 3×3 and 5×5 depthwise convolutions; [·] is channel concatenation.

8. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (5), firstly, the features output by the third layer encoder are After the third layer decoder performs feature reconstruction, local and global feature enhancement, and then upsampling is performed to obtain the output The fused encoder three-layer feature feature2 and Fusion, fully learning the image layer information, and then after upsampling once more to obtain It is fused again with the fused encoder three-layer feature feature1.

9. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (6), a small U-shaped network is used in the first layer of the decoder and the feature enhancement network. The small U-shaped network adopts a two-layer structure, the encoder is a Transformer block, and the decoder is IRFFT. The reconstructed features are reduced in resolution for multi-scale learning to restore the details and edge contours of the image and enhance the visual effect of the image.

10. The image deblurring method based on the combination of CNN and Transformer according to claim 1, characterized in that: In step (7), the enhanced features are subjected to 3×3 convolution to reduce the number of channels and obtain the reference image I R ∈R 3×H×W , combined with the original blurred image Iblur, to obtain a clear image after deblurring, as shown below: I deblur =I blur +I R , Where Ideblur is the deblurred image.

Images