A motion blur removal method based on a two-stage Transformer encoder / decoder with fused bilateral recombined attention.

Abstract

This invention provides a motion blur removal method based on a two-stage Transformer encoder-decoder with fused bilateral reconstructive attention. To improve the deblurring effect, a Transformer bilateral dynamic local window is proposed, employing dynamic growth of the local window self-attention in both horizontal and vertical directions. This allows the network to learn from local to global blur recovery in different directions. A supervised residual attention mechanism suitable for the two-stage encoder-decoder and a bilateral reconstructive attention mechanism optimizing residual feature propagation are combined. This enables the model to focus on residual edge features and content recovery features at different stages, and the bilateral reconstructive attention is used to reconstruct and propagate residual features, further improving model performance. By combining the bilateral reconstructive attention mechanism, supervised residual attention, and the bilateral dynamic local window Transformer two-stage encoder-decoder, the expressive power of the network is improved, and the deblurring effect is enhanced.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for de-motion blurring based on a two-stage Transformer codec that fuses bilateral recombined attention. Background Technology

[0002] In the process of digital image imaging and transmission, due to factors such as ambient lighting conditions, atmospheric motion, relative motion between the imaging object and the camera, and human techniques, images may experience information loss and quality degradation, resulting in image blurring and preventing people from accurately acquiring the rich information contained in the image. Images, as an important medium for transmitting information, play an indispensable role in people's lives and work. However, image blurring is a common problem: camera shake, rapid object movement, or defocusing can all reduce image quality and make images blurry. Furthermore, various factors, such as depth variations and occlusion of motion boundaries, further complicate blurring. Many shooting scenes change constantly over time and are not repeatable; if blurring occurs due to camera shake or rapid object movement, the image may become unusable. The degradation of blurred image quality greatly restricts the functionality of visual systems such as visual surveillance and object detection. Therefore, research on image deblurring methods to effectively utilize these blurred images is of great significance.

[0003] Image deblurring has always been an important task in computer vision and image processing. Motion blurring is a common phenomenon in the imaging process. The application of motion image deblurring is increasingly found in various fields. For example, it's used to recover relatively clear images from blurred images, such as those reflecting hand tremors under normal lighting or those reflecting the relative speed of an aircraft to the ground during aerial reconnaissance. The goal of deblurring is to recover a clear underlying image with the necessary edge structure and details.

[0004] Motion blur removal methods are mainly divided into two areas: non-blind motion blur removal algorithms and blind motion blur removal methods. The non-blind motion blur removal problem refers to finding the potentially sharp image given a known blur kernel. Early methods minimized the model using regularization functions and introduced measures to prevent overfitting and improve model generalization performance, suppressing the ringing effect to a certain extent. However, this easily leads to information loss, resulting in the loss of elemental information in the recovered image. The blind motion blur removal problem, on the other hand, uses location kernels to find the potentially sharp image, which is more realistic. Early blind motion blur removal methods could also utilize solutions from non-blind methods, first constructing a model to further estimate the blur kernel, and then restoring the image using prior image knowledge and various regularization optimizations based on non-blind methods. However, the restoration results were unsatisfactory. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention.

[0006] The technical solution of this invention is: a de-motion blurring method based on a two-stage Transformer codec with fused bilateral recombined attention, comprising the following steps:

[0007] S1) Construct a two-stage Transformer codec network, which contains two codecs;

[0008] S2) Obtain the blurred image X and the corresponding clear image Y, and format the blurred image X and the clear image Y as the input dataset;

[0009] S3) Input the blurred image X processed in step S2) and the corresponding clear image Y into the two-stage Transformer codec network constructed in step S1) for training, so as to extract the features of the image and learn how to recover the content and details of the blurred image.

[0010] S4) For the fuzzy test image X blur The input is fed into the trained two-stage Transformer encoder-decoder network, and the output features of the one-stage encoder-decoder are backpropagated with supervised residual attention to obtain the output features of the one-stage encoder-decoder.

[0011] S5) The first-stage output features are concatenated with the second-stage feature channels, and content restoration is further performed. A bilateral reassembly attention mechanism is then used to receive the residual feature input from the first-stage encoder / decoder, and the second-stage encoder / decoder restoration process is performed to finally obtain the output potentially sharp image Y. latent sharp image .

[0012] Preferably, in step S1), each encoder comprises 27 Transformer Blocks, 2 downsampling convolutional layers with a stride of 2, 2 upsampling deconvolutional layers, and multiple feature extraction convolutional layers.

[0013] Preferably, in step S4), the fuzzy test image X is... blur Perform the following processing:

[0014] S401), for the test image X blur After passing through the first-stage encoder-decoder network, a supervised residual attention mechanism is used to perform residual image feature recovery learning on the feature map, preparing it for passing to the second-stage encoder-decoder to fuse the input map features. The supervised residual feature F S1 The calculation formula is as follows:

[0015] FS1 =Sigmoid(F SRA )*TED1(X blur );

[0016] Among them, F SRA The residual features are obtained by subtracting the real image from the input image and then learning them through a convolution. The Sigmoid operation uses the supervised residual features as a probability distribution. TED1 represents a one-stage encoder-decoder network.

[0017] Preferably, step S5) specifically involves:

[0018] S501), a one-stage codec, obtains F S1 After supervising the residual features, the process moves to the two-stage encoder-decoder stage, where the two-stage feature map obtained by convolving the blurred image is combined with the supervising residual features F. S1 Channel concatenation completes the feature information, which is then input into a two-stage encoder-decoder to finally output a potentially sharp image. The calculation formula is as follows:

[0019] Y latent sharp image =TED2(Concat(wX) blur +b,F S1 ));

[0020] Where TED2 represents a one-stage encoder-decoder network, Concat(·) represents the channel concatenation operation, and w and b represent the weights and biases of the convolution.

[0021] Preferably, step S401 specifically includes the following steps:

[0022] S4011) A one-stage encoder-decoder with one layer of convolution is used to extract the input blurred image X. blur The basic texture structure, and the feature map calculation formula for the first-stage encoder-decoder layer convolution are as follows:

[0023] F1 = LeakyReLU(w1X) blur +b1);

[0024] Where LeakyReLU represents the activation function, w1 and b1 represent the weights and biases of a single convolutional layer in a one-stage encoder-decoder, respectively, and F1 represents the output feature map;

[0025] (S4012) The computational feature map F1 of the first-stage encoder-decoder's first-layer convolution is embedded as a patch feature into the second-layer convolution of the first-stage encoder-decoder to further extract non-linear features and perform dynamic local region recovery; the second-layer convolution of the first-stage encoder-decoder uses a Transformer group consisting of a Transformer block for shallow feature learning, and the specific calculation formula is as follows:

[0026] F t1 = 1 * Transformer(F1);

[0027] In the formula, F t1 This represents the output features of the second-layer convolution of a one-stage encoder-decoder;

[0028] (S4013) The one-stage encoder-decoder uses a three-layer convolutional downsampling operation with a stride of 2 to reduce the feature map size and extract abstract features. The specific calculation formula is as follows:

[0029] F2=w2F t1 +b2;

[0030] In the formula, w2 and b2 represent the weights and biases of the three convolutional layers in the first-stage encoder-decoder, respectively.

[0031] S4014), a one-stage encoder-decoder with four convolutional layers, uses a Transformer group consisting of two Transformer blocks to further learn abstract features. The specific calculation formula is as follows:

[0032] F t2 = 2 * Transformer(F2);

[0033] (S4015) The one-stage encoder-decoder uses five convolutional layers with a stride of 2 for downsampling to further reduce the feature map size and extract more abstract and high-level features. The specific calculation formula is as follows:

[0034] F3 = w3F t2 +b3;

[0035] In the formula, w3 and b3 represent the weights and biases of the five convolutional layers in the first-stage encoder-decoder, respectively.

[0036] S4016), a one-stage encoder-decoder with six convolutional layers, uses a Transformer group consisting of 21 Transformer blocks to further learn and repair high-level abstract features. The specific calculation formula is as follows:

[0037] F t3 =21*Transformer(F3);

[0038] S4017), the one-stage encoder-decoder uses seven convolutional layers with upsampling deconvolution operations to further increase the feature map size, gradually restoring the image size and recovering high-level abstract features. The calculation formula is as follows:

[0039] F4 = w4F t3 +b4;

[0040] In the formula, w4 and b4 represent the weights and biases of the seven convolutional layers in the one-stage encoder-decoder, respectively.

[0041] (S4018) The one-stage encoder-decoder uses eight layers of convolutions to learn abstract features using groups of Transformers consisting of two Transformer blocks. The calculation formula is as follows:

[0042] F t4 = 2 * Transformer(F4);

[0043] S4019) The one-stage encoder-decoder uses nine layers of convolution to restore the feature map to the size of the input image through upsampling deconvolution operations, thus restoring the image feature information. The calculation formula is as follows:

[0044] F5 = w5F t4 +b5;

[0045] In the formula, w5 and b5 represent the weights and biases of the nine convolutional layers in the one-stage encoder-decoder, respectively.

[0046] (S4020) A one-stage encoder-decoder with ten layers of convolution uses a group of Transformer blocks (each Transformer block) to learn image features. The calculation formula is as follows:

[0047] F t5 = 1 * Transformer(F5);

[0048] S4021) The eleven-layer convolution of the first-stage encoder-decoder uses a supervised residual attention mechanism to perform residual image feature recovery learning on the feature map, preparing it for fusion of input map features in the second-stage encoder-decoder. The calculation formula is as follows:

[0049] F6 = Sigmoid(F SRA )*F t5 ;

[0050] Among them, F SRA The residual features are obtained by subtracting the real image from the input image and then performing a convolutional learning process. The Sigmoid operation treats these supervised residual features as a probability distribution and applies them to the F-axis. t5 Multiplying them yields the residual characteristics.

[0051] Preferably, in S501), the following steps are included: After obtaining the F6 supervised residual features, the first-stage encoder-decoder enters the second-stage encoder-decoder stage. Each layer of the second-stage encoder-decoder includes two convolutions and one feature channel concatenation. First, in the same layer of the same stage, the blurred image uses convolution to obtain the basic texture structure. The obtained feature map is then concatenated with the supervised residual feature F6 channel to complete the feature information. Finally, a convolution is used to restore the number of channels and extract the feature information. The calculation formula is as follows:

[0052] F7 = LeakyReLU(w7(Concat(w6X)) blur +b6,F6))+b7)

[0053] Where LeakyReLU represents the activation function, Concat(·) represents the channel concatenation operation, F7 represents the feature map of the second-stage layer output, and w6, w7, b6, and b7 represent respectively;

[0054] The operations from layer 2 to layer 10 of the two-stage codec are the same as those from layer 2 to layer 10 of the same-stage codec. High-level abstract features are obtained and feature information is restored through one codec process to obtain feature map F8.

[0055] Finally, the eleventh layer of the two-stage encoder-decoder is the output of the convolutional operation, which is the final potentially sharp image. The calculation formula is as follows:

[0056] Y latent sharp image =w8F8+b8;

[0057] Where w8 and b8 represent the weights and biases of the eleventh layer of the two-stage encoder-decoder, respectively, and F8 represents the feature map.

[0058] Preferably, the Transformer group is composed of one or more cascaded Transformer blocks, and the Transformer blocks consist of a multi-head attention mechanism, a multi-layer perceptron, and a Layer Norm normalization operation.

[0059] Preferably, the calculation formula for the multilayer perceptron in the Transformer block is as follows:

[0060] mlp(x)=GeLU(w m1 x+b m1 )w m2 +b m2 ;

[0061] Where GeLU represents the activation function, w m1 and w m2 It is the weight of the perceptron, b m1 and b m2 It is the bias value of the perceptron.

[0062] Preferably, the Layer Norm normalization operation in the Transformer block is a lateral normalization operation of the layer. It considers the hierarchical variation of a single training sample, calculates the mean and variance of the layer input, and uses layer normalization to transform the input of different dimensions. The Layer Norm calculation formula is as follows:

[0063]

[0064] Where x is the input, y is the output, E[x] represents the mean of the input x, Var[x] represents the variance of the input x, ∈ is a positive value greater than zero, and γ and β are learnable and adjustable parameters.

[0065] The beneficial effects of this invention are as follows:

[0066] 1. This invention utilizes a two-stage codec for blurred image restoration. The first-stage codec extracts residual edge features, while the second-stage codec restores content features, thus achieving better restoration of image edges and content details in stages.

[0067] 2. This invention proposes a dynamic bilateral local expansion window, which divides the feature map into horizontal and vertical dynamic local expansion windows, performs self-attention between the two dynamic windows, and constructs global semantic information from local features, which is more in line with the characteristics of restoring blurred images.

[0068] 3. This invention proposes a residual supervision attention mechanism suitable for two-stage codecs. By using the residual map for feature backpropagation, feature attention is directly applied to the output of the first-stage codec, making the first-stage codec pay more attention to residual edge information and improving the content detail recovery efficiency of the second-stage codec.

[0069] 4. This invention proposes a bilateral reconstructed attention mechanism to reconstruct residual feature information on both sides, thereby optimizing and reconstructing pixel sites in local regions, and further feeding them into the second-stage content feature recovery stage to improve model performance.

[0070] 5. This invention can obtain high-quality potentially sharp images on the public GoPro dataset. In quantitative comparison of deblurring effects, both peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) surpass previous deblurring methods. Attached Figure Description

[0071] Figure 1 This is a flowchart of the method of the present invention;

[0072] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0073] Figure 3 This is a schematic diagram of the bilateral reorganization attention mechanism of the present invention;

[0074] Figure 4 This is a schematic diagram of the multi-head attention mechanism in the Transformer of this invention;

[0075] Figure 5This is a schematic diagram of the residual supervision attention mechanism of the present invention;

[0076] Figure 6 This is a comparison of the deblurring effects of the present invention and existing algorithms on the GoPro dataset; Detailed Implementation

[0077] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0078] Example 1

[0079] like Figure 1 As shown, this embodiment provides a motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention, including the following steps:

[0080] S1) Construct a two-stage Transformer encoder-decoder network, which contains two encoders and decoders; each encoder contains 27 Transformer Blocks, two downsampling convolutional layers with a stride of 2, two upsampling deconvolutional layers, and multiple convolutional layers for feature extraction. Its structure is as follows: Figure 2 As shown.

[0081] S2) Obtain the blurred image X and the corresponding clear image Y, and format the blurred image X and the clear image Y as the input dataset;

[0082] S3) Input the blurred image X processed in step S2) and the corresponding clear image Y into the two-stage Transformer codec network constructed in step S1) for training, so as to extract the features of the image and learn how to recover the content and details of the blurred image.

[0083] S4) For the fuzzy test image X blur The input is then fed into a pre-trained two-stage Transformer encoder-decoder network. The output features of the one-stage encoder-decoder are then subjected to supervised residual attention backpropagation to obtain the output features of the one-stage encoder-decoder; specifically:

[0084] S401), for the test image X blur After passing through the first-stage encoder-decoder network, a supervised residual attention mechanism is used to perform residual image feature recovery learning on the feature map, preparing it for passing to the second-stage encoder-decoder to fuse the input map features. The supervised residual feature F S1 The calculation formula is as follows:

[0085] F S1 =Sigmoid(F SRA )*TED1(X blur );

[0086] Among them, F SRA The residual features are obtained by subtracting the real image from the input image and then learning them through a convolution. The Sigmoid operation uses the supervised residual features as a probability distribution. TED1 represents a one-stage encoder-decoder network.

[0087] S5) The first-stage output features are concatenated with the second-stage feature channels, and content restoration is further performed. A bilateral reassembly attention mechanism is then used to receive the residual feature input from the first-stage encoder / decoder, and the second-stage encoder / decoder restoration process is performed to finally obtain the output potentially sharp image Y. latent sharp image Specifically:

[0088] S501), a one-stage codec, obtains F S1 After supervising the residual features, the process moves to the two-stage encoder-decoder stage, where the two-stage feature map obtained by convolving the blurred image is combined with the supervising residual features F. S1 Channel concatenation completes the feature information, which is then input into a two-stage encoder-decoder to finally output a potentially sharp image. The calculation formula is as follows:

[0089] Y latent sharp image =TED2(Concat(wX) blur +b,F S1 ));

[0090] Where TED2 represents a one-stage encoder-decoder network, Concat(·) represents the channel concatenation operation, and w and b represent the weights and biases of the convolution.

[0091] In a preferred embodiment, step S401 specifically includes the following steps:

[0092] S4011) A one-stage encoder-decoder with one layer of convolution is used to extract the input blurred image X. blur The basic texture structure, and the feature map calculation formula for the first-stage encoder-decoder layer convolution are as follows:

[0093] F1 = LeakyReLU(w1X) blur +b1);

[0094] Where LeakyReLU represents the activation function, w1 and b1 represent the weights and biases of a single convolutional layer in a one-stage encoder-decoder, respectively, and F1 represents the output feature map;

[0095] (S4012) The computational feature map F1 of the first-stage encoder-decoder's first-layer convolution is embedded as a patch feature into the second-layer convolution of the first-stage encoder-decoder to further extract non-linear features and perform dynamic local region recovery; the second-layer convolution of the first-stage encoder-decoder uses a Transformer group consisting of a Transformer block for shallow feature learning, and the specific calculation formula is as follows:

[0096] F t1 = 1 * Transformer(F1);

[0097] In the formula, F t1 This represents the output features of the second-layer convolution of a one-stage encoder-decoder;

[0098] (S4013) The one-stage encoder-decoder uses a three-layer convolutional downsampling operation with a stride of 2 to reduce the feature map size and extract abstract features. The specific calculation formula is as follows:

[0099] F2=w2F t1 +b2;

[0100] In the formula, w2 and b2 represent the weights and biases of the three convolutional layers in the first-stage encoder-decoder, respectively.

[0101] S4014), a one-stage encoder-decoder with four convolutional layers, uses a Transformer group consisting of two Transformer blocks to further learn abstract features. The specific calculation formula is as follows:

[0102] F t2 = 2 * Transformer(F2);

[0103] (S4015) The one-stage encoder-decoder uses five convolutional layers with a stride of 2 for downsampling to further reduce the feature map size and extract more abstract and high-level features. The specific calculation formula is as follows:

[0104] F3 = w3F t2 +b3;

[0105] In the formula, w3 and b3 represent the weights and biases of the five convolutional layers in the first-stage encoder-decoder, respectively.

[0106] S4016), a one-stage encoder-decoder with six convolutional layers, uses a Transformer group consisting of 21 Transformer blocks to further learn and repair high-level abstract features. The specific calculation formula is as follows:

[0107] F t3 =21*Transformer(F3);

[0108] S4017), the one-stage encoder-decoder uses seven convolutional layers with upsampling deconvolution operations to further increase the feature map size, gradually restoring the image size and recovering high-level abstract features. The calculation formula is as follows:

[0109] F4 = w4F t3 +b4;

[0110] In the formula, w4 and b4 represent the weights and biases of the seven convolutional layers in the one-stage encoder-decoder, respectively.

[0111] (S4018) The one-stage encoder-decoder uses eight layers of convolutions to learn abstract features using groups of Transformers consisting of two Transformer blocks. The calculation formula is as follows:

[0112] F t4 = 2 * Transformer(F4);

[0113] S4019) The one-stage encoder-decoder uses nine layers of convolution to restore the feature map to the size of the input image through upsampling deconvolution operations, thus restoring the image feature information. The calculation formula is as follows:

[0114] F5 = w5F t4 +b5;

[0115] In the formula, w5 and b5 represent the weights and biases of the nine convolutional layers in the one-stage encoder-decoder, respectively.

[0116] (S4020) A one-stage encoder-decoder with ten layers of convolution uses a group of Transformer blocks (each Transformer block) to learn image features. The calculation formula is as follows:

[0117] F t5 = 1 * Transformer(F5);

[0118] S4021) The eleven-layer convolution of the first-stage encoder-decoder uses a supervised residual attention mechanism to perform residual image feature recovery learning on the feature map, preparing it for fusion of input map features in the second-stage encoder-decoder. The calculation formula is as follows:

[0119] F6 = Sigmoid(F SRA )*F t5 ;

[0120] Among them, F SRA The residual features are obtained by subtracting the real image from the input image and then performing a convolutional learning process. The Sigmoid operation treats these supervised residual features as a probability distribution and applies them to the F-axis. t5 Multiplying them yields the residual characteristics.

[0121] In a preferred embodiment, S501 specifically includes the following steps: After obtaining the F6 supervised residual features, the first-stage encoder / decoder enters the second-stage encoder / decoder stage. Each layer of the second-stage encoder / decoder includes two convolutions and one feature channel concatenation. First, in the same layer of the same stage, the blurred image uses convolution to obtain the basic texture structure. The obtained feature map is then concatenated with the supervised residual feature F6 channel to complete the feature information. Finally, one convolution is used to restore the number of channels and extract the feature information. The calculation formula is as follows:

[0122] F7 = LeakyReLU(w7(Concat(w6X)) blur +b6,F6))+b7)

[0123] Where LeakyReLU represents the activation function, Concat(·) represents the channel concatenation operation, F7 represents the feature map of the second-stage layer output, and w6, w7, b6, and b7 represent respectively;

[0124] The operations from layer 2 to layer 10 of the two-stage codec are the same as those from layer 2 to layer 10 of the same-stage codec. High-level abstract features are obtained and feature information is restored through one codec process to obtain feature map F8.

[0125] Finally, the eleventh layer of the two-stage encoder-decoder is the output of the convolutional operation, which is the final potentially sharp image. The calculation formula is as follows:

[0126] Y latent sharp image =w8F8+b8;

[0127] Where w8 and b8 represent the weights and biases of the eleventh layer of the two-stage encoder-decoder, respectively, and F8 represents the feature map.

[0128] In a preferred embodiment, the Transformer group is composed of one or more cascaded Transformer blocks, and the Transformer blocks consist of a multi-head attention mechanism, a multi-layer perceptron, and a LayerNorm normalization operation.

[0129] In this preferred embodiment, the calculation formula for the multilayer perceptron in the Transformer block is as follows:

[0130] mlp(x)=GeLU(w m1 x+b m1 )w m2 +b m2 ;

[0131] Where GeLU represents the activation function, w m1 and w m2 It is the weight of the perceptron, b m1 and bm2 It is the bias value of the perceptron.

[0132] In this preferred embodiment, the Layer Norm normalization operation in the Transformer block is a lateral normalization operation of the layer. It considers the hierarchical changes of a single training sample, calculates the mean and variance of the layer input, and uses layer normalization to transform the inputs of different dimensions. The Layer Norm calculation formula is as follows:

[0133]

[0134] Where x is the input, y is the output, E[x] represents the mean of the input x, Var[x] represents the variance of the input x, ∈ is a positive value greater than zero, and γ and β are learnable and adjustable parameters.

[0135] Example 2

[0136] In this embodiment, the acquired blurred image X is randomly cropped to 256×256 and normalized before being fed into the network for training. The network parameters are trained based on the PyTorch framework. The training parameters of the network in this embodiment are set as follows: batch size is set to 2, the number of training epochs is 600; the number of channels in different stages is set to [48, 96, 192]; the number of Transformer Blocks N1, N2, and N3 in different stages is set to [1, 2, 21], and the number of heads H1, H2, and H3 is set to [2, 4, 8]; the expansion ratio of the MLP multilayer perceptron is set to 4; two Dropout operations are performed in each Transformer Block with the parameter set to drop=0.1; and feature abstraction with reduction=16 scaling is performed in the bilateral recombinant attention mechanism module.

[0137] For the fuzzy test image X blur After normalization, the image is input into the two-stage Transformer encoder-decoder de-motion blurring neural network. The first-stage network obtains residual features, which are then input into the second-stage network to obtain the final deblurred image. Figure 3 , Figure 4 and Figure 5 The paper provides detailed schematic diagrams of the bilateral recombination attention mechanism, the multi-head attention mechanism, and the residual supervised attention mechanism in the two-stage Transformer encoder-decoder neural network of the present invention.

[0138] like Figure 3As shown, the bilateral reorganization attention mechanism proposed in this invention first performs average pooling and max pooling in the X and Y directions (i.e., the H and W dimensions of the input features) respectively on the input features F. When the features have been refined into elements of local regions, average pooling can retain the basic content information of the input features to the greatest extent, while max pooling can further separate the differences between elements, i.e., separate edge information. After the two pooling operations are completed, the information in the same direction is concatenated along the Y direction to obtain a bilateral pooled feature map that integrates content and edge information. Then, the bilateral pooled feature map is compressed through a multilayer perceptron (MLP) with a reduced channel, and then convolution and separation operations are performed to obtain unilateral feature maps in the X and Y directions. Finally, the unilateral feature maps in the X and Y directions are subjected to a sigmoid operation to obtain a probability map and matrix multiplication is performed to obtain the final bilateral reorganization attention map. The calculation formula of the bilateral reorganization attention mechanism is as follows:

[0139] F XY =(Concat Y (X AVG (F), X MAX (F), Y AVG (F), Y MAX (F)));

[0140] F X F Y =Split Y (MLP(F XY ));

[0141] F OUT =F*(Sigmoid(F X )*Sigmoid(F Y ));

[0142] Among them, X AVG and X MAX These represent average pooling and max pooling along the X direction, respectively. The resulting feature map along the W direction has a dimension of C*H*2. Similarly, Y... AVG and Y MAX These represent average pooling and max pooling along the H direction, respectively, resulting in a feature map with dimensions C*2*W along the H direction; Concat Y This indicates a splicing operation along the Y direction. MLP stands for Multilayer Perceptron. Split Y It is the feature map F separated along the Y direction. X ∈C*H*1 and F Y ∈C*1*W, and finally, the sum is multiplied by a matrix, and the resulting attention map is multiplied by the initial input feature F to obtain the final bilateral reconstructed feature F. OUT .

[0143] The multi-head attention mechanism of the Transformer module of this invention is as follows: Figure 4 As shown, the input features R∈H*W*C are first linearly transformed into h Heads. These Heads are then divided into two groups: one group undergoes a vertical window FW*W self-attention mechanism, and the other group undergoes a horizontal window FW*W self-attention mechanism. These mechanisms learn the local vertical and horizontal transfer features of the blurred image, respectively. Finally, the two channels are combined to form the restored features with combined local regions. By setting different FW values ​​at different stages, different information expressed by the feature maps at different stages can be addressed.

[0144] like Figure 5 As shown, in the residual supervised attention mechanism of the present invention, the input feature map with F0∈H*W*2C is first processed through two paths. The first path is converted into feature information F1∈H*W*C, and the second path is converted into a residual latent sharp image (i.e., backpropagation residual image) R∈H*W*3. Then, the residual latent sharp image is converted into residual features F2∈H*W*C, and after a sigmoid function operation, an attention map ∈H*W*C is obtained. The attention map is multiplied by F1 to obtain the feature F∈H*W*C, which is then concatenated with the features of the two-stage encoder and decoder.

[0145] To illustrate the advantages of the network used in this invention, Table 1 presents a quantitative scoring table for PSNR and SSIM deblurring on the GoPro dataset, comparing this invention with some existing algorithms. The GoPro dataset is widely used in the field of deblurring. (The table can be found on...) Figure 6 As can be seen, this invention achieved the best PSNR and SSIM values, indicating that the deblurring results obtained by the method of this invention are of the highest quality. Figure 6 The figure shows a comparison of the deblurring performance of the present invention with existing algorithms (DeblurGAN-v2, DMPHN, ERCNN, SRN) on the GoPro dataset. As can be seen from the figure, the method proposed in this invention can better recover high-frequency details and remove artifacts, achieving the best deblurring effect compared to existing methods.

[0146] Table 1. Quantitative analysis of different methods on the GoPro dataset.

[0147]

[0148] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention, characterized in that, Includes the following steps: S1) Construct a two-stage Transformer codec network, which contains two codecs; S2) Obtain the blurred image X and the corresponding clear image Y, and format the blurred image X and the clear image Y as the input dataset; S3) Input the blurred image X processed in step S2) and the corresponding clear image Y into the two-stage Transformer codec network constructed in step S1) for training, so as to extract the features of the image and learn how to recover the content and details of the blurred image. S4) For the fuzzy test image X blur The input is fed into the trained two-stage Transformer encoder-decoder network, and the output features of the one-stage encoder-decoder are backpropagated with supervised residual attention to obtain the output features of the one-stage encoder-decoder. Specifically: S401), for the test image X blur After passing through the first-stage encoder-decoder network, a supervised residual attention mechanism is used to perform residual image feature recovery learning on the feature map, preparing it for passing to the second-stage encoder-decoder to fuse the input map features. The supervised residual feature F S1 The calculation formula is as follows: F S1 =Sigmoid(F SRA )*TED1(X blur ); Among them, F SRA It is the residual feature obtained by subtracting the real image from the input image and then learning it through one convolution. The Sigmoid operation uses the supervised residual feature as a probability distribution. TED1 represents a one-stage encoder-decoder network. S5) The first-stage output features are concatenated with the second-stage feature channels, and content restoration is further performed. A bilateral reassembly attention mechanism is then used to receive the residual feature input from the first-stage encoder / decoder, and the second-stage encoder / decoder restoration process is performed to finally obtain the output potentially sharp image Y. latentsharpimage Specifically: S501), a one-stage codec, obtains F S1 After supervising the residual features, the process moves to the two-stage encoder-decoder stage, where the two-stage feature map obtained by convolving the blurred image is combined with the supervising residual features F. S1 Channel concatenation completes the feature information, which is then input into a two-stage encoder-decoder to finally output a potentially sharp image. The calculation formula is as follows: Y latentsharpimage =TED2(Concat(wX blur +b,F S1 )); Where TED2 represents a one-stage encoder-decoder network, Concat(·) represents the channel concatenation operation, and w and b represent the weights and biases of the convolution.

2. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 1, characterized in that: Each encoder contains 27 Transformer Blocks, 2 downsampling convolutional layers with a stride of 2, 2 upsampling deconvolutional layers, and multiple convolutional layers for feature extraction.

3. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 1, characterized in that, Step S401) is as follows: S4011) A one-stage encoder-decoder with one layer of convolution is used to extract the input blurred image X. blur The basic texture structure, and the feature map calculation formula for the first-stage encoder-decoder layer convolution are as follows: F1=LeakyReLU(w1X blur +b1); Where LeakyReLU represents the activation function, w1 and b1 represent the weights and biases of a single convolutional layer in a one-stage encoder-decoder, respectively, and F1 represents the output feature map; (S4012) The computational feature map F1 of the first-stage encoder-decoder's first-layer convolution is embedded as a patch feature into the second-layer convolution of the first-stage encoder-decoder to further extract non-linear features and perform dynamic local region recovery; the second-layer convolution of the first-stage encoder-decoder uses a Transformer group consisting of a Transformer block for shallow feature learning, and the specific calculation formula is as follows: F t1 =1*Transformer(F1); In the formula, F t1 This represents the output features of the second-layer convolution of a one-stage encoder-decoder; (S4013) The one-stage encoder-decoder uses a three-layer convolutional downsampling operation with a stride of 2 to reduce the feature map size and extract abstract features. The specific calculation formula is as follows: F2=w2F t1 +b2; In the formula, w2 and b2 represent the weights and biases of the three convolutional layers in the first-stage encoder-decoder, respectively. S4014), a one-stage encoder-decoder with four convolutional layers, uses a Transformer group consisting of two Transformer blocks to further learn abstract features. The specific calculation formula is as follows: F t2 =2*Transformer(F2); (S4015) The one-stage encoder-decoder uses five convolutional layers with a stride of 2 for downsampling to further reduce the feature map size and extract more abstract and high-level features. The specific calculation formula is as follows: F3=w3F t2 +b3; In the formula, w3 and b3 represent the weights and biases of the five convolutional layers in the first-stage encoder-decoder, respectively. S4016), a one-stage encoder-decoder with six convolutional layers, uses a Transformer group consisting of 21 Transformer blocks to further learn and repair high-level abstract features. The specific calculation formula is as follows: F t3 =21*Transformer(F3); S4017), the one-stage encoder-decoder uses seven convolutional layers with upsampling deconvolution operations to further increase the feature map size, gradually restoring the image size and recovering high-level abstract features. The calculation formula is as follows: F4=w4F t3 +b4; In the formula, w4 and b4 represent the weights and biases of the seven convolutional layers in the one-stage encoder-decoder, respectively. (S4018) The one-stage encoder-decoder uses eight layers of convolutions to learn abstract features using groups of Transformers consisting of two Transformer blocks. The calculation formula is as follows: F t4 =2*Transformer(F4); S4019) The one-stage encoder-decoder uses nine layers of convolution to restore the feature map to the size of the input image through upsampling deconvolution operations, thus restoring the image feature information. The calculation formula is as follows: F5=w5F t4 +b5; In the formula, w5 and b5 represent the weights and biases of the nine convolutional layers in the one-stage encoder-decoder, respectively. (S4020) A one-stage encoder-decoder with ten layers of convolution uses a group of Transformer blocks (each Transformer block) to learn image features. The calculation formula is as follows: F t5 =1*Transformer(F5); S4021) The eleven-layer convolution of the first-stage encoder-decoder uses a supervised residual attention mechanism to perform residual image feature recovery learning on the feature map, preparing it for fusion of input map features in the second-stage encoder-decoder. The calculation formula is as follows: F6=Sigmoid(F SRA )*F t5 ; Among them, F SRA The residual features are obtained by subtracting the real image from the input image and then performing a convolutional learning process. The Sigmoid operation treats these supervised residual features as a probability distribution and applies them to the F-axis. t5 Multiplying them yields the residual characteristics.

4. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 1, characterized in that, S501 specifically includes the following steps: After obtaining the F6 supervised residual features, the first-stage encoder / decoder enters the second-stage encoder / decoder stage. Each layer of the second-stage encoder / decoder consists of two convolutions and one feature channel concatenation. First, in the same layer, the blurred image uses convolution to obtain the basic texture structure. The obtained feature map is then concatenated with the F6 supervised residual features to complete the feature information. Finally, a convolution is used to restore the number of channels and extract feature information. The calculation formula is as follows: F7=LeakyReLU(w7(Concat(w6X blur +b6,F6))+b7) Where LeakyReLU represents the activation function, Concat(·) represents the channel concatenation operation, F7 represents the feature map of the second-stage layer output, and w6, w7, b6, and b7 represent respectively; The operations from layer 2 to layer 10 of the two-stage codec are the same as those from layer 2 to layer 10 of the same-stage codec. High-level abstract features are obtained and feature information is restored through one codec process to obtain feature map F8. Finally, the eleventh layer of the two-stage encoder-decoder is the output of the convolutional operation, which is the final potentially sharp image. The calculation formula is as follows: AND latentsharpimage =w8F8+b8; Where w8 and b8 represent the weights and biases of the eleventh layer of the two-stage encoder-decoder, respectively, and F8 represents the feature map.

5. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 3, characterized in that: The Transformer group consists of one or more cascaded Transformer blocks, which are composed of multi-head attention mechanisms, multilayer perceptrons, and Layer Norm normalization operations.

6. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 5, characterized in that: The calculation formula for the multilayer perceptron in the Transformer block is as follows: mlp(x)=GeLU(w m1 x+b m1 ) m2 +b m2 ; Where GeLU represents the activation function, w m1 and w m2 It is the weight of the perceptron, b m1 and b m2 It is the bias value of the perceptron.

7. The motion blur removal method based on a two-stage Transformer codec with fused bilateral recombined attention as described in claim 5, characterized in that: The Layer Norm normalization operation in the Transformer block is a lateral normalization operation of a layer. It considers the hierarchical variations of a single training sample, calculates the mean and variance of the layer input, and uses layer normalization to transform the inputs of different dimensions. The Layer Norm calculation formula is as follows: Where x is the input, y is the output, E[x] represents the mean of the input x, Var[x] represents the variance of the input x, ∈ is a positive value greater than zero, and γ and β are learnable and adjustable parameters.

Images