Single image deblurring method based on generative adversarial network with transformer fusion

Abstract

The application belongs to the technical field of image restoration in computer vision, and discloses a single image deblurring method based on a generative adversarial network fusing a transformer, which comprises the following steps: constructing a scale feature reconstruction enhancement module XT, wherein the module XT comprises an aggregation module ST, a mapping module DT and a rendering module PT; processing a target image to be deblurred to obtain a multi-scale pyramid feature map, and inputting the multi-scale pyramid feature map into the module XT to reconstruct scale features and enhance the representation ability of the scale features; merging the multi-scale pyramid feature map with the enhanced representation ability after decoding, and restoring the multi-scale pyramid feature map to the original image size to output a clear target image. The method reduces the network complexity, expands the receptive field, and can perform high-resolution recovery on a blurred image.

Description

Technical Field

[0001] This invention belongs to the field of image restoration technology in computer vision, and specifically relates to a method for deblurring a single image using a generative adversarial network that integrates Transformer. Background Technology

[0002] Image deblurring techniques can be applied not only to everyday photography but also to data preprocessing for advanced visual tasks such as object detection and semantic segmentation, making them of significant practical importance and research value. Deep learning methods, due to their effectiveness and versatility, have become the mainstream approach for handling complex real-world blur.

[0003] Traditional image processing determines the motion-blurred image degradation function by estimating the point spread function. However, real-world blurring is far more complex. In extreme cases, it's necessary to calculate the corresponding blur kernel function for each pixel, and calculation errors can easily lead to poor image restoration or even image distortion. Peak signal-to-noise ratio (PSNR), as a crucial indicator of deblurring performance, is increasingly difficult to improve while ensuring reduced computational complexity.

[0004] There are many deep learning-based image deblurring methods with greater versatility. Some of these methods use CNN networks to assist in estimating the point spread function of the blurred image. However, these methods may encounter difficulties when estimating and processing complex spatial blur kernels. Other methods do not directly estimate the blur kernel but instead use multi-scale sub-network inputs combined with CNN convolutional networks to obtain a larger receptive field. Based on this technique, the weights of corresponding modules are shared in different sub-networks, reducing the number of network parameters. However, stacking sub-networks still generates significant computational complexity. Transformers can provide the larger receptive field required for complex blurred scenes, and reusing Transform modules can provide global relevance. However, the complexity increases quadratically with the image input resolution and cannot restore high-resolution images.

[0005] In summary, solving the point spread function for each pixel in traditional image processing is an ill-posed problem. Multi-scale networks based on CNNs can handle complex blur, but stacking sub-networks has the disadvantages of large number of parameters and high complexity. Networks that reuse Transformer modules consume too much energy for high-resolution images, and the computational cost of the network is insufficient to grow quadratically with the size of the input image resolution. They also ignore the local correlation of convolution. Summary of the Invention

[0006] To address the above problems, this invention provides a single-image deblurring method integrating a Transformer-based generative adversarial network (GAN), which reduces network complexity while expanding the receptive field, enabling high-resolution restoration of blurred images. The method includes:

[0007] A scale feature reconstruction and enhancement module XT is constructed, which includes an aggregation module ST, a mapping module DT, and a rendering module PT.

[0008] The aggregation module ST simultaneously acquires local correlations within a window, acquires global connections between different windows, increases the nonlinear representation capability of the output through the FFN layer, and uses convolution to fuse residual connection information, outputting an aggregated feature map.

[0009] The mapping module DT obtains the similarity matrix between the high-scale high-semantic feature map and the current-scale feature map, and multiplies it with the current-scale feature map matrix to obtain a multi-dimensional mapped feature map;

[0010] The rendering module PT extracts the summarization information matrix of the low-scale low-semantic feature map and multiplies it with the high-level feature projection matrix, and then superimposes the downsampled low-level features to obtain a multi-dimensional rendering feature map.

[0011] The target image to be deblurred is processed to obtain a multi-scale pyramid feature map, which is then input into module XT to reconstruct scale features and enhance their representation capabilities.

[0012] The decoded multi-scale pyramid feature maps with enhanced representation capabilities are merged and restored to their original size to output a clear target image.

[0013] The method for reconstructing scale features and enhancing their representational capabilities includes:

[0014] The feature maps of multiple scales in the multi-scale pyramid feature map are input into the aggregation module ST to obtain a multi-scale pyramid feature map with enhanced representation capabilities.

[0015] Alternatively, the higher-level feature maps in the multi-scale pyramid feature map and the current-scale feature map can be combined and input into the mapping module DT, and the current-scale feature map can be input into the aggregation module ST. The outputs of the two modules are cascaded and merged into the original-scale feature map to obtain a multi-scale pyramid feature map with enhanced representation capabilities.

[0016] Alternatively, the higher-level feature maps and the current-scale feature map in the multi-scale pyramid feature map can be combined and input into the mapping module DT, the lower-level feature maps and the current-scale feature map can be combined and input into the rendering module PT, and the current-scale feature map can be input into the aggregation module ST. The outputs of the three modules are cascaded and merged into the original-scale feature map to obtain a multi-scale pyramid feature map with enhanced representation capabilities.

[0017] Furthermore, the number of multi-scale pyramid feature maps input to module XT is the same as the number of multi-scale pyramid feature maps output by module XT that enhance its representational capabilities.

[0018] Furthermore, when two or more modules from the aggregation module ST, mapping module DT, and rendering module PT are used in the method of reconstructing scale features and enhancing their representational capabilities, the sub-modules in module XT run in parallel.

[0019] Furthermore, the aggregation module ST uses a window-based multi-head self-attention module to simultaneously acquire local correlations within a window, and uses a multi-head self-attention module after window displacement to acquire global connections between different windows; the mapping module DT obtains the similarity matrix between the high-scale high-semantic feature map and the current-scale feature map through dot product, and multiplies it with the current-scale feature map matrix to obtain a multi-dimensional mapped feature map; the rendering module PT extracts the generalized information matrix of the low-scale low-semantic feature map through global average pooling, multiplies it with the high-level feature projection matrix, and superimposes the downsampled low-level features to obtain a multi-dimensional rendered feature map.

[0020] Furthermore, the method includes the following steps:

[0021] S1. Obtain the dataset; obtain image pairs of blurred and clear images to create the target dataset for the deblurring task;

[0022] S2. Dataset preprocessing; preprocess the dataset obtained in S1, and adjust the images to an appropriate size;

[0023] S3. Divide the dataset into a training set and a validation set;

[0024] S4. Construct the scale feature reconstruction and enhancement module XT;

[0025] S5. Construct the overall network to obtain the XT-GAN network model that integrates Transformers;

[0026] S6. Train the XT-GAN network model;

[0027] S7. Deblur a single image using the trained XT-GAN network model.

[0028] Furthermore, in step S1, image pairs of blurred and clear images under different scenes are collected; in step S2, the images are adjusted to a suitable size by cropping or padding; and the dataset is labeled, with the blurred image corresponding to the intermediate frame of the continuous clear image, and labeled as blurred image and clear image respectively.

[0029] Further, in said step S5, the generator adopts the first eight layers of the GhostNet convolutional network as the encoder backbone network. One 3×3 convolutional layer, one data normalization layer and one ReLU activation layer are added before the backbone network to preprocess input features, and the module XT is inserted after the decoder backbone network to enhance the output pyramid features. The decoder network module is responsible for restoring the image size, two VGG19 are used as discriminators, and the XT-GAN network model is obtained.

[0030] Further, in said step S6, the training strategy comprises: setting the batch size to 1, using Adam as the training rate optimizer, training for 2000 epochs, adopting 3 epochs of freezing training to warm up the model after the training starts, then continuing training with a learning rate of 10-4 until 50 epochs, and continuing to reduce the learning rate using a linear decay strategy, which decays to 10-7 at the 2000th epoch; random cropping, random dynamic blur, random median blur, random image compression, random sharpening and random grayscale are used for data augmentation, and the head-num of ST is set to 4.

[0031] Further, in step S7, the single image deblurring method comprises: preprocessing the size of a target image to be deblurred, then inputting the preprocessed image into the XT-GAN network model, and running the XT-GAN network model to output a clear image.

[0032] Further, the size preprocessing method comprises the following steps:

[0033] T1. Acquiring the height H and width W of the input image;

[0034] T2. Calculating the minimum height h and width w of the image required by the network; wherein h=(H / / 32+1)*32, w=(W / / 32+1)*32, and / / represents integer division;

[0035] T3. Judging whether image padding is required, and performing padding if H < h or W < w.

[0036] Compared with the prior art, one or more of the above technical solutions can achieve at least one of the following beneficial effects:

[0037] The invention discloses a single image deblurring method based on generative adversarial network fused with Transformer,

[0038] 1. An end-to-end network deblurring method is adopted, a deblurring result can be directly obtained only by inputting an image to be deblurred, which reduces time cost and has faster detection speed.

[0039] 2. Using FPN pyramid features as a lightweight alternative to multi-scale feature input avoids the stacking of multi-scale sub-networks and reduces network complexity.

[0040] 3. The proposed method replaces the commonly used upsampling pyramid fusion method with the XT module, which strengthens and reconstructs the pyramid features of the output and expands the receptive field.

[0041] 4. Because it uses highly correlated pyramid multi-scale features extracted from each layer of the convolutional network, the Transformer module of the XT structure uses parallel computing to avoid module reuse and stacking. At the same time, it can combine local and global information to associate features of different scales, complete the convolutional pyramid feature processing with less complexity, and improve accuracy.

[0042] 5. The XT-GAN network model retains more scale information of the original features while reducing complexity. It is also more suitable for mini-batch training of deblurring networks, avoiding the problem of excessive variance differences between different batches of features when using BN standardization. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the aggregation module ST structure of the present invention.

[0045] Figure 2 This is a schematic diagram of the mapping module DT structure of the present invention.

[0046] Figure 3 This is a schematic diagram of the rendering module PT structure of the present invention.

[0047] Figure 4 This is a schematic diagram of the scale feature reconstruction enhancement module XT in Example 3.

[0048] Figure 5 This is a schematic diagram of the overall network structure of Example 4.

[0049] Figure 6 This is a schematic diagram of the process of deblurring a single image using the XT-GAN network model in Example 4.

[0050] Figure 7 Flowchart of the method for preparing the XT-GAN network model in Example 5. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment provides a single-image deblurring method that integrates Transformer and Generative Adversarial Network, including the following steps:

[0054] 1. Construct a scale feature fusion module with feature reconstruction and enhancement capabilities.

[0055] A scale feature reconstruction enhancement module XT with feature reconstruction enhancement capabilities is constructed. It includes an aggregation module ST with the function of aggregating local and global information, a mapping module DT that maps high-level features, and a rendering module PT that renders low-level features. Combining the mapping module, rendering module, and aggregation module will yield the module XT with feature reconstruction enhancement capabilities.

[0056] Specifically, such as Figure 1 As shown, the aggregation module ST first uses a window-based multi-head self-attention module (W-MSA) to simultaneously obtain local correlations within the window, and then uses multi-head self-attention after window displacement to obtain global connections between different windows. The FFN layer increases the nonlinear representation capability of the output, and uses 1×1 convolution to fuse residual connection information. The final output information aggregation feature map will simultaneously cover the features extracted from global connections and local correlations.

[0057] Attention W-MSA = resize(Attention) n ).

[0058] Attention W-MSA This indicates the W-MSA operation, resize indicates merging and partitioning the windows, and Attention... n This indicates that W-MSA calculations are performed within the divided n windows. After the shifted window ensures that the features of adjacent windows are under the same window, a second W-MSA calculation is performed, and a mask is used to occlude windows that were not originally adjacent but are now in the computation space.

[0059] like Figure 2As shown, the mapping module DT uses dot product to obtain the similarity matrix between the high-scale high semantic feature map and the current-scale feature map, and multiplies it with the current-scale feature map matrix to obtain the output of DT;

[0060]

[0061] Q dt K dt V dt W dt-q W dt-k W dt-v Let K, Q, V represent DT and their respective weight matrices, M. low M represents low-level features. high Indicates high-level characteristics, Out dt For the output of DT, Q dt T Q represents dt The transpose of the matrix, mat, denotes matrix multiplication.

[0062] like Figure 3 As shown, the rendering module PT uses global average pooling to extract the general information matrix of the low-scale low semantic feature map and multiply it with the high-level feature projection matrix, and then superimposes the downsampled low-level features to obtain the PT output.

[0063]

[0064] Q pt K pt V pt Representing K, Q, V, W of PT pt-q W pt-k Represents the weight matrix, avg represents global average pooling, down represents downsampling operation, and Out represents the weight matrix. pt This indicates the output of PT.

[0065] 2. Process the target image to be deblurred to obtain a multi-scale pyramid feature map, and input it into module XT to reconstruct scale features and enhance its representation ability.

[0066] As one implementation method, the methods for reconstructing scale features and enhancing their representational capabilities include:

[0067] The scale feature maps in the multi-scale pyramid feature map are input into the aggregation module ST to obtain a multi-scale pyramid feature map with enhanced representation capabilities.

[0068] 3. After merging the decoded multi-scale pyramid feature maps, restore them to the original image size and output a clear target image.

[0069] In this embodiment, after processing the target image to be deblurred, a multi-scale pyramid feature map is obtained. This map is used as input to the XT module, and the aggregation module ST processes the scale feature map to obtain a multi-scale pyramid feature map with enhanced representation capabilities, thereby achieving the purpose of image deblurring. In this method, the aggregation module ST only needs to process the current scale feature map once, which can ensure a certain image clarity (see Table 3 in Comparative Example 3) and overcome the shortcomings of existing technologies such as large number of parameters and high complexity.

[0070] Example 2

[0071] The difference between this embodiment and Embodiment 1 lies in the method of reconstructing scale features and enhancing their representational ability: the higher-level feature map and the current scale feature map in the multi-scale pyramid feature map are combined and input into the mapping module DT, and the current scale feature map is input into the aggregation module ST. The outputs of the two are cascaded and merged into the original scale feature map to obtain a multi-scale pyramid feature map with enhanced representational ability.

[0072] Example 3

[0073] The difference between this embodiment and Embodiments 1 and 2 lies in the method of reconstructing scale features and enhancing their representational capabilities.

[0074] like Figure 4 As shown, for the four input pyramid feature maps at different scales, the XT modules with different structures output aggregated feature maps. Specifically, the higher-level feature maps and the current-scale feature map in the multi-scale pyramid feature map are combined and input into the mapping module DT, the lower-level feature maps and the current-scale feature map are combined and input into the rendering module PT, and the current-scale feature map is input into the aggregation module ST. The outputs of the three modules are cascaded and merged with the original scale feature map to obtain a multi-scale pyramid feature map with enhanced representation capabilities.

[0075] The scale feature fusion XT module can represent feature maps at different scales; the representation of feature maps at different scales is jointly completed by modules DT and PT in the XT module. The scale feature fusion XT module can reconstruct scale features and enhance their representation capabilities; in this embodiment, the enhancement of scale feature representation capabilities will be jointly completed by modules ST, DT, and PT.

[0076] In a specific implementation, this embodiment inputs four pyramid feature maps at four different scales to module XT. The four pyramid feature maps are ordered from highest to lowest scale as a, b, c, and d. For feature map a, a itself undergoes ST module operations; b, c, and d are combined with a and input into the PT module for calculation, cascading the original scale a to output XT(a) feature. For feature map b, b itself undergoes ST module operations; a and b are combined with b and input into the DT module for calculation; c and d are combined with b and input into the PT module for calculation, cascading the original scale a to output XT(b) feature. For feature map c, c itself undergoes ST module operations; a and b are combined with c and input into the DT module for calculation; d and c are combined with c and input into the PT module for calculation, cascading the original scale a to output XT(c) feature. For feature map d, d itself undergoes ST module operations; a, b, and c are combined with d and input into the DT module for calculation, cascading the original scale a to output XT(d) feature. Thus, four XT modules with different structures are obtained, each corresponding to a pyramid feature input at a different scale.

[0077] Example 4

[0078] This embodiment provides a single-image deblurring method that integrates Transformer with a generative adversarial network, including the following steps:

[0079] S1. Obtain the dataset.

[0080] First, image pairs of blurred and sharp images are obtained to create the target dataset for the deblurring task. Specifically, image pairs of blurred and sharp images in different scenes are collected. Using the publicly available GoPro incremental dataset as an example, this embodiment includes 6600 image pairs with an image size of 720×1280.

[0081] S2. Dataset preprocessing.

[0082] The dataset obtained in S1 is preprocessed by cropping or padding the images to a suitable size. In a specific implementation, the image size is padded to 736×1312. Next, the dataset is labeled, with blurred images corresponding to intermediate frames of consecutive sharp images, labeled as blurred images and sharp images respectively, and saved to a folder.

[0083] S3. Dataset partitioning.

[0084] The dataset is divided into training and validation sets in a 2:1 ratio. In this embodiment, the training set contains 4400 pairs, and the test set contains 2200 pairs.

[0085] S4. Construct the scale feature reconstruction enhancement module XT.

[0086] Construct the scale feature reconstruction enhancement module XT as shown in Example 3.

[0087] First, considering multi-scale information, the multi-scale features output by the backbone network are used as input, and 1×1 convolutions are used to enhance or compress the information content of the feature maps. Second, considering the fusion of feature information, the ST module is constructed to output the aggregated information of global and local information of the current scale feature map. LayerNorm is used to normalize the ST feature map output. The DT and PT feature enhancement modules are constructed to establish the correlation matrix between features of different scales. The constructed HIBlock module is used to normalize the DT and PT feature map outputs. Finally, the ST, DT and PT modules are combined to form the scale feature reconstruction enhancement module XT with feature reconstruction capability.

[0088] S5. Construct the overall network to obtain the generative adversarial network fused with Transformer: XT-GAN network model.

[0089] like Figure 5 As shown, the generator uses the first eight layers of the GhostNet convolutional network as the encoder backbone. A 3×3 convolutional layer, a data normalization layer, and a ReLU activation layer are added before the backbone for input feature preprocessing. The module XT constructed by S4 is inserted into the decoder backbone to enhance its output pyramid features. The decoder network module is responsible for restoring the image size. Two VGG19s are used as discriminators to obtain the XT-GAN network model.

[0090] Among them, the generator loss function L used by XT-GAN G :

[0091]

[0092] Where L content For content loss, L adv-full L adv-patch As the adversarial loss for the dual-scale discriminator, the output result L G It is λ times the average of their two outputs and L content The sum of , and the value of λ is set to 0.01.

[0093] Content loss L content :

[0094] L content =αL perc +βL mse .

[0095] Where L perc To perceive loss, L mse For mean squared error, the values ​​of α and β are set to 0.006 and 0.5, respectively. The discriminator loss L used in XT-GAN... D :

[0096]

[0097] Where L D-full It is the global discriminator loss value, L D-patch This is the local discriminator loss value; the output result is the multi-scale discriminator loss value L. D It is the average of their outputs.

[0098] Discriminator loss L D With the loss of the opponent L adv

[0099]

[0100]

[0101] Where D represents the discriminator network, I real It is the label image of the input image, I fake It is a fake sample generated by the generator from the input image, which is saved for later use.

[0102] S6. Train the T-GAN network model.

[0103] The preprocessed datasets from steps S2 and S3 are input into the XT-GAN network model constructed in step S5. First, the parameters are initialized using ImageNet pre-trained weights. The input image is downsampled to a resolution of 256×256 before being input into the network. The final output of the network is B×3×256×256, where B is the training batch size.

[0104] XT-GAN is trained at a fixed rate, followed by a learning rate decay strategy to prevent overfitting. In this embodiment, XT-GAN is trained with a batch size of 1, using Adam as the training rate optimizer for 2000 epochs. After the initial training, a 3-epoch freeze is used to warm up the model, followed by training at a learning rate of 10⁻⁴ up to 50 epochs. A linear decay strategy is then used to further reduce the learning rate, decreasing it to 10⁻⁷ at the 2000th epoch. Data augmentation includes random cropping, random motion blur, random median blur, random image compression, random sharpening, and random grayscale. The model weights that perform best on the validation set are used as the final training result.

[0105] S7. Deblur a single image using the trained XT-GAN network model.

[0106] The process of deblurring a single image is as follows: Figure 6As shown. The network reads the finally obtained model weights for the single-image deblurring task. The motion-blurred image is padded to an appropriate size and input into the XT-GAN network model, and finally the deblurring result generated by the network is output. After preprocessing the image size, the result can be obtained by directly inputting it into the network loaded with the S6 training model.

[0107] Example 5

[0108] As Figure 7 shown, the single-image deblurring method based on Transformer-fused generative adversarial network of this example, the processing of image size during data set preprocessing in Example 4 is performed in the following manner:

[0109] T1. Obtain the height H and width W of the input image;

[0110] T2. Calculate the minimum height h and width w of the image required by the network; where h=(H / / 32+1)*32; w=(W / / 32+1)*32; where / / represents integer division, for example 9 / / 2=4;

[0111] T3. Determine whether image padding is required. If H<h or W<w, perform padding, and use the PadIfNeeded method in the albumentations library for processing.

[0112] The remaining steps are the same as those in Example 4.

[0113] Comparative Example 1

[0114] The XT-GAN network is compared on the synthetic GoPro dataset using incremental data.

[0115] Table 1 compares the peak signal-to-noise ratio of the XT-GAN network model and different deblurring methods. The smaller the floating-point computation amount, the faster the running speed, and the deblurring effect is represented by the peak signal-to-noise ratio. The higher the value, the better the image restoration effect. When a high-resolution image of 720×1280 is input, the floating-point computation amount of the network is reduced by at least 70% compared with other networks, and the peak signal-to-noise ratio reaches 29.13.

[0116] Table 1 Synthetic training dataset GoPro

[0117]

[0118] Comparative Example 2

[0119] Table 2 shows the simulation of removing real-world scenes using the model trained on the GoPro dataset on the RealBlur-J test set. Table 2 compares XT-GAN with different deblurring methods, using Peak Signal-to-Noise Ratio (PSNR) and structural similarity to compare model robustness. As can be seen from Table 2, the XT-GAN network model trained on the GoPro dataset still exhibits good robustness on the RealBlur-J dataset, achieving a PSNR of 28.40 dB.

[0120] Table 2. Real-world test dataset RealBlur-J

[0121]

[0122] Comparative Example 3

[0123] like Figure 4 As shown, the XT feature reconstruction enhancement module of this invention comprises three parts: DT, PT, and ST, which are combined in parallel. The network structure diagram of this invention is shown below. Figure 5 As shown, the generator includes an encoder, an XT module, a decoder, and the discriminator includes two VGG19 networks.

[0124] Table 3 demonstrates the effectiveness of each component in the XT module using ablation experiments.

[0125] Table 3. Validity of each component in the XT module

[0126] √ 28.75 √ √ √ 29.13 √ √ 29.06 √ √ 28.83 √ √ 27.79 √ 29.02 √ 28.89 √ 28.61

[0127] Experiments have shown that structures containing ST can provide more effective information, and even using ST alone can achieve good results. Although the results of using DT and PT alone are not satisfactory, they can provide additional scale information for ST, and the combination of ST, DT and PT can achieve the best results.

[0128] Comparative Example 4

[0129] Table 4 shows that the difference between MSA and SA lies in the number of Head-Num heads in the multi-head self-attention module, which is related to the total number of feature maps output by MSA.

[0130] Table 4 shows the number of Head-Num values ​​for ST.

[0131]

[0132] The network achieves the best deblurring effect when the Head-Num is equal to 16. It achieves the highest PSNR and SSIM evaluation values ​​of 29.20dB and 0.925, respectively. The Head-Num=4, which performs best in the RealBlur-J

[31] test set, was chosen as the hyperparameter setting. In practice, the Head-Num that performs best in the target dataset should be selected.

[0133] Obviously, the above embodiments are merely examples to clearly illustrate the technical solutions of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A single image deblurring method based on a fusion Transformer-based generative adversarial network, characterized in that, The method includes: A scale feature reconstruction and enhancement module XT is constructed, which includes an aggregation module ST, a mapping module DT, and a rendering module PT. The aggregation module ST simultaneously acquires local correlations within a window and global connections between different windows, increases the nonlinear representation capability of the output through the FFN layer, and uses convolution to fuse residual connection information to output an aggregated feature map. The mapping module DT obtains the similarity matrix between the high-level feature map and the current-scale feature map, and multiplies it with the current-scale feature map matrix to obtain a multi-dimensional mapped feature map. The rendering module PT extracts the product of the general information matrix of the low-level feature map and the projection matrix of the high-level feature map, and superimposes the downsampled low-level features to obtain a multi-dimensional rendering feature map. The target image to be deblurred is processed to obtain a multi-scale pyramid feature map. Each scale feature map in the multi-scale pyramid feature map is input into module XT. The scale features of each scale feature map are reconstructed and their representation ability is enhanced to obtain a multi-scale pyramid feature map with enhanced representation ability. The multi-scale feature maps in the decoded multi-scale pyramid feature map with enhanced representation capability are merged to restore the multi-scale pyramid feature map with enhanced representation capability to the original image size, and output a clear target image. The method for reconstructing scale features and enhancing the representational ability of each scale feature map includes: Each scale feature map in the multi-scale pyramid feature map is input into the aggregation module ST to obtain the corresponding scale feature map with enhanced representation capabilities. Alternatively, the high-level feature map and the current-scale feature map in the multi-scale pyramid feature map can be combined and input into the mapping module DT to obtain the output of the mapping module DT. The current-scale feature map can be input into the aggregation module ST to obtain the output of the aggregation module ST. The output of the mapping module DT, the output of the aggregation module ST, and the current-scale feature map can be cascaded together and the aggregated feature information can be used to obtain the current-scale feature map with enhanced representation capabilities. Alternatively, the high-level feature map and the current-scale feature map in the multi-scale pyramid feature map can be combined and input into the mapping module DT to obtain the output of the mapping module DT. The low-level feature map and the current-scale feature map can be combined and input into the rendering module PT to obtain the output of the rendering module PT. The current-scale feature map can be input into the aggregation module ST to obtain the output of the aggregation module ST. The output of the mapping module DT, the output of the rendering module PT, the output of the aggregation module ST, and the current-scale feature map are then cascaded together and aggregated to obtain the current-scale feature map with enhanced representation capabilities.

2. The single-image deblurring method based on a generative adversarial network incorporating Transformer as described in claim 1, characterized in that, The number of multi-scale pyramid feature maps input to module XT is the same as the number of multi-scale pyramid feature maps output by module XT with enhanced representation capabilities.

3. The single-image deblurring method based on a generative adversarial network incorporating Transformer as described in claim 1, characterized in that, When two or more modules from the aggregation module ST, mapping module DT, and rendering module PT are used in the method of reconstructing scale features and enhancing their representation capabilities, the sub-modules in module XT run in parallel.

4. The single-image deblurring method using a generative adversarial network incorporating Transformer as described in claim 1, characterized in that, The aggregation module ST uses a window-based multi-head self-attention module to simultaneously acquire local correlations within a window, and uses a multi-head self-attention module after window displacement to acquire global connections between different windows; the mapping module DT obtains the similarity matrix between the high-level feature map and the current-scale feature map through dot product, and multiplies it with the current-scale feature map matrix to obtain a multi-dimensional mapped feature map; the rendering module PT extracts the generalized information matrix of the low-level feature map through global average pooling, multiplies it with the high-level feature projection matrix, and superimposes the downsampled low-level features to obtain a multi-dimensional rendered feature map.

5. The single-image deblurring method based on a generative adversarial network incorporating Transformer according to any one of claims 1 to 4, characterized in that, The method includes the following steps: S1. Obtain the dataset; obtain image pairs of blurred and clear images to create the target dataset for the deblurring task; S2. Dataset preprocessing; preprocess the dataset obtained in S1, adjusting the image size to an appropriate value; S3. Divide the dataset into a training set and a validation set; S4. Construct a scale feature reconstruction and enhancement module XT with feature reconstruction and enhancement capabilities; S5. Construct the overall network to obtain the XT-GAN network model that integrates Transformers; S6. Train the XT-GAN network model; S7. Deblur a single image using the trained XT-GAN network model.

6. The single-image deblurring method based on a generative adversarial network incorporating Transformer as described in claim 5, characterized in that, In step S1, image pairs of blurred and clear images under different scenes are collected; in step S2, the images are adjusted to a suitable size by cropping or padding; and the dataset is labeled, with the blurred image corresponding to the intermediate frame of the continuous clear image, and labeled as blurred image and clear image respectively.

7. The single-image deblurring method using a generative adversarial network incorporating Transformer as described in claim 5, characterized in that, In step S5, the generator uses the first eight layers of the GhostNet convolutional network as the encoder backbone network. A 3×3 convolutional layer, a data normalization layer, and a ReLU activation layer are added before the backbone network for preprocessing the input features. The XT module is inserted into the decoder backbone network to enhance the pyramid features of its output. The decoder network module is responsible for restoring the image size. Two VGG19s are used as discriminators to obtain the XT-GAN network model.

8. The single-image deblurring method based on a generative adversarial network incorporating Transformer as described in claim 5, characterized in that, In step S6, the training strategy includes: setting the batch size to 1, using Adam as the training rate optimizer, training for 2000 epochs, using a 3-epoch freeze training to warm up the model after training begins, then continuing training with a learning rate of 10⁻⁴ up to 50 epochs, and using a linear decay strategy to further reduce the learning rate, decaying to 10⁻⁷ at the 2000th epoch; data augmentation uses random cropping, random motion blur, random median blur, random image compression, random sharpening, and random grayscale, and the head-num of ST is set to 4.

9. The single-image deblurring method based on a generative adversarial network incorporating Transformer as described in claim 5, characterized in that, In step S7, the method for deblurring a single image includes: preprocessing the size of the target image to be deblurred and then inputting it into the XT-GAN network model, and running the XT-GAN network model to output a clear image.

10. The single-image deblurring method using a generative adversarial network incorporating Transformer as described in claim 5, characterized in that, The method for adjusting image size includes the following steps: T1. Obtain the height H and width W of the input image; T2. Calculate the minimum height h and width w of the image required by the network; wherein h=(H / / 32+1)×32, w=(W / / 32+1)×32, and / / represents integer division; T3. Determine whether it is necessary to pad the image, and perform padding if H<h or W<w.

Images