Image deblurring method of Transform model based on multi-scale loss supervision

Abstract

The invention discloses an image deblurring method of a Transform model based on multi-scale loss supervision, and the method comprises the following steps: firstly, designing a brand new AttBlock module for extracting key features of an image and realizing feature recovery of the image, and the module is still suitable for a high-resolution image; then, based on an atmospheric scattering model, calculating multi-scale loss from the multi-scale output image of the decoder; and finally, a multi-scale loss supervision structure is designed, the training process of back propagation and parameter updating of the model is supervised, and the peak signal-to-noise ratio of the deblurred image is remarkably improved. According to the method, the characteristics of a dual-supervision network and a Transform model are combined, the attention module and supervision network structure of the model is improved, and a high-performance image deblurring network is designed. Through verification of a large number of experiments, compared with other image deblurring networks, the method provided by the invention has better performance indexes, and meanwhile, the calculation efficiency is more efficient compared with a network of the same scale.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and specifically designs an image deblurring method based on a Transformer model with multi-scale loss supervision. Background Art

[0002] In recent years, the Transformer paradigm has shown significant performance improvements in natural language processing and has been widely used in fields such as computer vision. However, there are few Transformer models that are suitable for image deblurring.

[0003] The core mechanism of the Transformer model is its self-attention mechanism, which integrates the global context information of the image through a multi-head attention module network to extract the key features of the image. This process usually consists of three parts: image block generator, multi-head self-attention and feedforward network. The principle can be expressed by formula (1):

[0004]

[0005] X∈R in formula (1) N×C , N is the number of image blocks to segment the input image, W Q , W K , W V are the weights of query Q, key K and value V respectively, and FFN is a feedforward network composed of fully connected layers. However, the self-attention mechanism of this structure has two defects: first, the computational complexity increases quadratically with the increase of spatial resolution; second, it is difficult for the fully connected layer to retain spatial resolution information. This makes it difficult for Transformer to be applied to high-resolution image deblurring tasks. To address this problem, most previous methods adopted the idea of ​​image segmentation, dividing the input image into non-overlapping image blocks of size 48×48, and applying the attention mechanism to each image block to extract image features. The defect of this method is that it destroys the global information relationship of the image and affects the effect of image restoration. Therefore, under the condition of limited computing resources, improving the performance indicators of deblurred images is an important issue in the field of image deblurring.

[0006] Image deblurring is essentially an ill-posed problem, that is, a blurred image will output multiple reasonable deblurred images. Solving this problem usually requires strong image prior information. Since convolutional neural networks are not only lightweight, but can also learn prior knowledge well from large-scale data, they are widely used in the field of image restoration. However, convolutional neural networks are limited by two problems: one is that the limited receptive field hinders the integration of global information, and the other is that static weights are difficult to adapt to dynamic input content. Therefore, combining convolutional neural networks with self-attention mechanisms can effectively give play to the advantages of both and avoid their disadvantages.

[0007] The network of image deblurring methods based on deep learning is usually an encoder-decoder structure. The stacked encoders extract the blur characteristics of the image layer by layer, and then the decoders restore the detailed information of the image layer by layer. However, this traditional structure not only causes the stacking of computational complexity, but also the deblurring effect of the output image is generally poor. The main reason is that the parameter update of the traditional structure only depends on the final output image, while ignoring the information of other scales of the original image. Therefore, combining the multi-scale information of the image to establish a multi-loss supervision structure can effectively improve this defect.

[0008] In summary, the present invention proposes an image deblurring method based on a Transformer model with multi-scale loss supervision. Specifically, one is to design an AttBlock module containing a dual attention mechanism based on an encoder-decoder structure; the other is to design a multi-scale loss supervision structure. The AttBlock module contains channel attention and spatial resolution attention composed of two-dimensional convolutions. In the encoding stage, the blurred air streak features of the image are extracted from the two dimensions of channel and spatial resolution, and the blurred air streak features are weakened and the image is restored in the decoder stage. Finally, the loss function of the output image at two scales is calculated to supervise the parameter update during the model training process until the deblurred image reaches the optimal performance index. Summary of the invention

[0009] The present invention proposes an image deblurring method based on a Transformer model supervised by multi-scale loss. The method comprises: 1) obtaining global context information by using a designed AttBlock module; 2) realizing feature fusion and feature recovery by using a designed AttBlock module; 3) updating parameters of a structure optimization model by using a designed multi-scale loss supervision.

[0010] An image deblurring method based on a Transformer model with multi-scale loss supervision, characterized by comprising:

[0011] 1. Design a multi-convolutional head attention to expand the receptive field and obtain image information from different channels.

[0012] 2. Design a gated convolutional feedforward network to preserve spatial resolution information.

[0013] 3. Design a multi-scale loss supervision structure to optimize the parameter update of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The framework of the proposed image deblurring method;

[0015] Figure 2 This is a schematic diagram of the structure of the AttBlock module;

[0016] Figure 3 Schematic diagram of the structure of multi-scale loss supervision. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0018] The purpose of the present invention is to provide an image deblurring method based on a Transformer model supervised by multi-scale loss, and the implementation idea is as follows: first, a new AttBlock module is designed to obtain global context information, that is, multi-convolutional head attention and gated convolutional feedforward network are used to extract multi-scale features of the image, and the encoder of the Transformer is composed by stacking AttBlock modules; then, the decoder of the Transformer is composed by stacking AttBlock modules to realize feature fusion and restoration of the image; finally, the loss value of the output image is calculated from two different scales, and a multi-scale loss supervision structure is designed to supervise and optimize the parameter update of the model. The image deblurring method based on the Transformer model supervised by multi-scale loss of the present invention is suitable for high-resolution image deblurring tasks while improving the accuracy of image deblurring, and specifically includes the following steps:

[0019] 1. AttBlock module coding

[0020] The encoder contains 4 layers of AttBlock modules. Each layer of modules is downsampled to obtain the next layer of modules to extract image features of different dimensions. The AttBlock module combines the strengths of the Transformer model and convolutional neural network, uses multi-convolutional head attention to extract cross-channel information of the image, and uses the gated convolutional feedforward network to extract spatial resolution information, realizing the interaction and feature encoding of multi-scale global context information.

[0021] The AttBlock module consists of two parts: multi-convolutional head attention (MCSA) and gated convolutional feedforward network (Gate). The operation process can be expressed by formula (1):

[0022]

[0023] 1.1. Multi-convolutional head attention

[0024] The main computational complexity in the Transformer model comes from the self-attention layer. For an image with W×H pixels, the computational complexity of the traditional self-attention layer grows quadratically, O(W 2 H 2 ). To address this problem, the present invention designs a multi-convolutional head attention based on a convolutional neural network, and its computational complexity is linear. First, based on the normalization layer, a 1×1 convolution is used to aggregate cross-channel context information in the pixel dimension; secondly, a 3×3 deep convolution is used to generate the query Q, key K, and value V to achieve channel dimension context information encoding; then, a conversion attention map is generated by performing a dot product between Q and K, and a dot product is performed with V to obtain the attention result. The above operation process can be expressed by formula (2):

[0025]

[0026] In formula (2) and X are the output feature map and input map; α is a model parameter used to control the size of the dot product of K and Q in the Softmax function. The multi-convolutional head attention structure is as follows Figure 2 shown.

[0027] 1.2. Gated Convolutional Feedforward Network

[0028] The feedforward network of the fully connected layer brings high accuracy to self-attention, but it brings a huge computational cost. If the number of fully connected layers is reduced in order to reduce the amount of computation, the accuracy of self-attention will drop significantly. In addition, since the blur is usually low in saturation, the fully connected layer has a poor effect in retaining spatial resolution in the image deblurring task. To address this problem, the present invention designs a gated convolution feedforward network based on a convolutional neural network. First, based on the normalization layer, a 1×1 two-dimensional convolution is used to expand the feature channel and reduce the dimension of the channel respectively; secondly, a gating mechanism is used to aggregate the contextual information of the spatial resolution; finally, deep convolution is used to complete the encoding. The gating mechanism is to obtain a weight by passing one of the outputs of the two-dimensional convolution through the ReLU activation function, and then perform a dot multiplication with the other output. The operation process of the gating mechanism can be expressed by formula (3):

[0029]

[0030] In formula (3), ⊙ represents the channel dot product, and φ represents the image weight activated by GELU nonlinearity. The gated convolution feedforward network structure is as follows: Figure 2 shown.

[0031] 2. AttBlock module decoding

[0032] like Figure 1 The decoder shown in the figure contains four layers of AttBlock modules. Each layer of modules is upsampled to obtain the next layer of modules, and then the encoder's same-scale features are input to the decoder's AttBlock module through skip connections to achieve multi-scale feature fusion. Finally, the output features of image restoration are obtained through 1×1 convolution.

[0033] 3. Multi-scale loss supervision

[0034] Consider the deblurring effect of the image from multiple scales, such as Figure 3 As shown in the figure, a multi-scale loss supervision structure is designed. First, the first loss function is calculated based on the output image and the atmospheric scattering model. The calculation formula is shown in formula (4).

[0035]

[0036] In formula (4), I(x) is the image to be processed, A is the global atmospheric light component, t(x) is the atmospheric transmittance, K is the transparency channel parameter, B is the RGB channel parameter, J(x) is the theoretical prediction value of the unblurred image to be restored, J o is the actual input value of the graph, is the actual predicted value of the graph; the MSE function is the mean square error function. The specific calculation steps of the process are as follows:

[0037] (1) Extract the point with the highest brightness in the original image as the value of the atmospheric light component A. That is, J = (IA) / t+A;

[0038] (2) Set the threshold t0. If the t value is less than the t0 value, then t=t0 to avoid the entire image being white due to the t value being too small.

[0039] (3) The final J(x) is expressed as

[0040] (4) Divide the output image into channels to obtain the K value of the transparency channel and the B value of the RGB channel respectively.

[0041] (5) Substituting K and B into formula (4) we get The loss value is obtained using the mean square error function MSE.

[0042] Then, the output image is refined and fused with the original image to obtain a new output image. The Refine module is composed of the AttBlock module, which aims to retain the texture information of the original image so that the restored image is closer to the original image. The second loss function is calculated according to formula (4). Finally, based on these two loss functions of different scales, the update of parameters such as reverse gradient and learning rate is comprehensively processed to optimize the training process of the model. The above operation process can be expressed by formula (5):

[0043]

[0044] In formula (4), X is the parameter of model training, lr represents the learning rate, grad represents the total reverse gradient, and grad1 and grad2 represent the reverse gradients of the two loss functions respectively.

Claims

1. An image deblurring method based on a Transformer model with multi-scale loss supervision, characterized in that: The following steps are involved: Step A: The original image is obtained by stacking the encoder composed of AttBlock modules to obtain the multi-scale features of the image; Step B: A decoder composed of stacked AttBlock modules is used to realize feature fusion and feature recovery of the image; Step C, dividing the output of the decoder into channels and calculating the loss of image prediction using an atmospheric scattering model; Step D: The parallel output image of the decoder is processed by Refine and fused with the original image, and the atmospheric scattering model is used again to calculate the loss of image prediction. Step E: Back propagation and parameter updating based on the multi-loss supervision model significantly improves the performance of image deblurring.

2. The method according to claim 1, characterized in that Each layer of encoder reduces the image resolution by downsampling and extracts image features, and iterates the 4-layer encoder to obtain the final key features.

3. The method according to claim 1, characterized in that The AttBlock module structure is composed of two parts in series: multi-convolutional head attention and gated convolutional feedforward network. The operation process can be expressed by formula (1):

4. The method according to claim 1, characterized in that The image prediction loss is derived from the atmospheric scattering model, and the calculation process can be expressed by formula (2): Where I(x) is the image to be processed, J(x) is the unblurred image to be restored, A is the global atmospheric light component, and t(x) is the atmospheric transmittance. The above parameters can be obtained by channel segmentation of the final output of the AttBlock module.

5. The method according to claim 1, characterized in that The Refine is composed of AttBlock, the purpose of which is to fine-tune the image and further refine the image features. At the same time, in order to retain the texture details of the original image, it is added and fused with the fine-tuned image, so that multi-scale losses can be calculated.

6. The method according to claim 1, characterized in that The multi-loss supervision refers to the prediction loss values ​​obtained in step C and step D, taking into account the back propagation and parameter update of the two: starting from the output layer, the gradient of each layer is calculated in reverse, and passed back to the previous layer to adjust the model parameters to seek the minimum prediction loss value.

7. The method according to claim 2, characterized in that The key design of the multi-convolutional head attention is to apply the self-attention mechanism across channels to generate an implicitly encoded attention map. The network normalizes the tensor from the layer Generate the three attention elements Q, K, V and apply deep convolution to encode the spatial information across channels. The process can be described by formula (2):

8. The method according to claim 2, characterized in that The key design of the gated convolution feedforward network is to use the gated mechanism to perform point multiplication on two 1×1 convolution channels, one of which is activated by GELU nonlinearity to obtain the weight of the image blur feature. At the same time, deep convolution is used to encode cross-channel spatial information, which helps to improve the effect of image feature extraction and restoration. The process can be described by formula (3): Where ⊙ represents the channel dot product and φ represents the image weight activated by GELU nonlinearity.

Images