Transform image high-resolution reconstruction method based on branch enhancement

Abstract

A Transform image super-resolution reconstruction method based on branch enhancement comprises the steps that image super-resolution reconstruction is carried out, a BET basic module framework is designed, a parallel attention fusion module is mainly introduced, and a PAFM core is composed of parallel branches including a window multi-head self-attention branch and a convolution branch. And super-resolution reconstruction of the image is carried out by adopting a parallel fusion network structure form. SE and CA modules in a CNN branch are introduced, and while more pixels are activated for reconstruction, similar self-attention can be utilized to enhance interaction between necessary information while efficient global dependency modeling is obtained, so that extraction of detail features and fusion of overall features are enhanced. Through a plurality of attention mechanisms and feature fusion operations, the PAFM can effectively enhance the representation capability of important features and improve the network performance. And the branches are complemented in advantages in a collaborative manner in the image super-division reconstruction process.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, relates to image super-resolution reconstruction, and specifically to a Transformer image high-resolution reconstruction method based on branch enhancement. Background Art

[0002] Super-resolution (SR) is a long-standing and important problem, which aims to recover a high-resolution (HR) image from a given low-resolution (LR) image. This problem is common in real life and is limited by many factors. For example, under poor imaging conditions, the camera device may not be able to capture a clear enough image. In addition, the limitation of storage space often causes the image to be compressed during storage, resulting in the loss of detail information. When the image is transmitted through the network, due to the need for encoding and decoding before and after the channel, noise is often introduced, and the blur of the image is aggravated. All of the above factors will lead to different degrees of image quality loss. In the field of computer vision, image resolution is one of the key factors affecting the processing results of downstream tasks. Low-resolution images often only guarantee the overall structural consistency, but lack high-frequency detail information, which affects the accuracy and effect of subsequent tasks. For example, in applications such as expression recognition, object detection, and image restoration, lower resolution will greatly reduce the performance of the algorithm. Expression recognition needs to capture subtle changes in the face, object detection relies on clear edge information, and image restoration needs to restore as many original details as possible.

[0003] There are three main methods for traditional image super-resolution reconstruction in the early days: interpolation-based super-resolution algorithms, such as bicubic interpolation and nearest neighbor interpolation; degradation model-based super-resolution algorithms, such as iterative back-projection and convex set projection; learning-based super-resolution algorithms, including sparse coding methods. Traditional super-resolution algorithms have achieved great success, but with the increase of the scale factor from x2 to x3, x4 and even x8, more and more information is required for super-resolution reconstruction, and artificially defined prior knowledge can no longer meet the needs, making it difficult to achieve the purpose of reconstructing high-quality images. Super-resolution methods based on convolutional neural networks are popular because of their powerful ability to extract high-frequency details of images. From the earliest SRCNN method to the SRGAN method to various optimization methods today. However, using CNN-based methods to establish global connectivity often requires the design of complex network structures, and the overall process is relatively difficult. As an alternative, the Transformer-based method uses powerful self-attention (SA) to model the global dependency of input data and shows better performance. However, some studies have found that Transformer lacks a local mechanism for information aggregation in local areas. Some studies have proposed the Swin Transformer method, which introduces window self-attention and greatly reduces computational complexity and memory usage while ensuring high performance through a sliding window mechanism and hierarchical structure, making it more efficient when processing high-resolution images. However, this type of network can only use a limited range of input information space, and there is little information interaction between windows, resulting in insufficient cross-window information fusion. This means that the potential of the Transformer in existing networks has not yet been fully utilized. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Transformer image super-resolution reconstruction method based on branch enhancement, which combines the image high-resolution algorithm of a convolutional neural network (CNN) and a visual Transformer to achieve higher quality image super-resolution reconstruction.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A Transformer image super-resolution reconstruction method based on branch enhancement, characterized by comprising the following steps:

[0007] Step 1: Shallow feature extraction:

[0008] Given a low-resolution input image I LR (H×W×C in )H, W and Cin are the height, width and number of input channels of the image respectively, and a 3×3 convolution layer H is used. sf (·) Extract shallow features F 0 , see formula (1);

[0009] F 0 =H sf (I LR ) (1);

[0010] Step 2: Deep feature extraction;

[0011] From the shallow feature F 0 Extract the deep features F df (H×W×C), see formula (2);

[0012] F df =H df (F 0 ) (2);

[0013] Among them, H df (·) is composed of m residual BET modules cascaded, with H BET (·) indicates that the residual connection accelerates the convergence speed by directly passing the input to the subsequent layer, see formula (3);

[0014] H df(i) =H BET (F i-1 )i=1,2…k (3);

[0015] BET is composed of n PAFM modules stacked and connected to a 3x3Conv. PAFM (·) and H Conv (·) indicates, see formula (4);

[0016] H BET (F i-1 )=H PAFM (F i-1 )+H Conv (F i-1 )i=1,2…k (4);

[0017] The PAFM consists of a CNN auxiliary branch H CNN (·)(Including SE (Squeeze and Excitation) module and Channel Attention module, W-MSA branch H W-MSA (·) Two-way branching, α is the weight factor of the CNN branch, therefore;

[0018] H PAFM (F j-1 )=α·HCNN (F j-1 )+H W-MSA (F j-1 )j=1,2…k (5);

[0019] Step 3: high-resolution image reconstruction;

[0020] Reconstruct high-quality images by aggregating shallow features and deep features. SR , where H RE (·) is the function of the reconstruction module. The shallow features mainly contain low-frequency information, and the deep features contain the lost high-frequency information. At the same time, sub-pixel convolution is used to upsample the features to reconstruct a high-quality image I SR See formula (6);

[0021] I SR =H RE (F 0 +F df ) (6).

[0022] Furthermore, the loss function for super-resolution reconstructed image SR is obtained by minimizing L 1 Pixel loss to optimize network parameters and are the i-th reconstructed image and the high-resolution original image respectively;

[0023]

[0024] Among them, L 1 The loss function calculates the sum of the absolute differences between the actual value and the target value. In this type of supervised image super-resolution task, the goal is to make the generated super-resolution reconstructed image (SR) as close as possible to the real high-resolution image (HR). 1 The loss is calculated as the error between the values ​​of SR and HR at the corresponding pixel positions.

[0025] Furthermore, the shallow feature extraction module mainly includes a 3x3 convolution and ReLU activation function. By sliding the convolution kernel on the input image, the dot product between the convolution kernel and the local area of ​​the input image is calculated. The result of the convolution operation is represented as a new feature map. After the convolution operation, a nonlinear transformation is performed through the ReLU activation function to increase the expression ability of the model.

[0026] Furthermore, the main structure of deep feature extraction includes the cascade of several BET modules and the residual connection of each module. Each BET is mainly composed of the cascade of several PAFM modules, and finally connected to a 3x3 convolution module and residual connection.

[0027] Furthermore, the PAFM module is a parallel image processing module, including a convolution branch and a W-MSA branch, which is used to extract and enhance image features and improve the adaptability of the model to different types of inputs.

[0028] Furthermore, in the PAFM module:

[0029] First, through Layer Norm (layer normalization), the mean of the feature is set to 0 and the standard deviation is set to 1, which helps to speed up the training of the network;

[0030] After that, the input image is copied to two parallel paths, each of which performs different feature processing tasks. The convolution branch mainly includes the Squeeze and Excitation (SE) module and the Channel Attention (CA) module.

[0031] The SE module adjusts the channel weights of image features, increases the weights of important channels, and reduces the weights of unimportant channels, so that the network can focus more on important features. It mainly includes two 3x3Conv layers with a GLUE activation function in the middle.

[0032] The CA module further enhances the focus on important channel features. By calculating the attention weights, the network can identify which channel features are more important. It includes a global average pooling layer, followed by a convolutional layer plus a ReLU activation function and a convolutional layer plus a Sigmoid activation function.

[0033] Furthermore, the high-resolution image reconstruction described herein fuses shallow features, deep features, and initial input features through residual connections, thereby ensuring the effective combination of low-frequency information and high-frequency details. The fused features are upsampled by sub-pixel convolution to improve the resolution.

[0034] Furthermore, the sub-pixel convolution increases the number of channels and then uses a pixel shuffle operation to upsample the feature map from a lower resolution to a higher resolution. The pixel shuffle operation is a special transformation that rearranges pixels at the same spatial position in multiple channels into a larger spatial dimension.

[0035] The beneficial effects of the present invention are:

[0036] The present invention designs a BET (Branch-Enhanced Transformer) basic module framework, in which a parallel attention fusion module (PAFM) is mainly introduced. The PAFM core is composed of parallel branches, including a window multi-head self-attention (Window-Multihead Self Attention, W-MSA) branch and a (Convolutional Neural Network, CNN) convolution branch. The super-resolution reconstruction of the image is performed using a parallel fusion network structure. Since the pixel range of the W-MSA module is limited, erroneous textures may be restored. If more input pixels can be used, this situation can be further improved. Therefore, the present invention introduces the SE (Squeeze and Excitation) and CA (Channel Attention) modules of the CNN branch. While activating more pixels for reconstruction, similar self-attention can be used to enhance the interaction between necessary information while obtaining efficient global dependency modeling, thereby enhancing the extraction of detail features and the fusion of overall features. Through a variety of attention mechanisms and feature fusion operations, PAFM can effectively enhance the representation ability of important features and improve network performance. It enables the branches to complement each other in the process of image super-resolution reconstruction.

[0037] In addition, the dataset of the present invention selects the DIV2K dataset, which contains 800 high-resolution images, and uses bicubic downsampling to process the images to generate pairs of high-low resolution images for model training, thereby achieving higher quality image super-resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a principle flow chart of the present invention.

[0039] Figure 2 This is a diagram of the internal structure of the network.

[0040] Figure 3 It is the comparison picture of image_012.png in the embodiment.

[0041] Figure 4 This is a comparison diagram of Barbara.png in the embodiment. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings.

[0043] like Figure 1As shown in FIG. 1 , a Transformer image super-resolution reconstruction method based on branch enhancement is composed of three modules: shallow feature extraction, deep feature extraction and high-resolution (HR) image reconstruction module, including the following steps:

[0044] Step 1: Shallow Features Extraction

[0045] Given a low-resolution input image I LR (H×W×C in )H, W and C in are the height, width and number of input channels of the image respectively. We use a 3×3 convolutional layer H sf (·) Extract shallow features

[0046] F 0 =H sf (I LR ) (1);

[0047] The convolution layer is good at early visual processing, which is conducive to more stable optimization and better results. In addition, this simple method can map the input image space to a higher-dimensional feature space, and then perform deep feature extraction from it.

[0048] Step 2: Deep Features Extraction

[0049] From F 0 Extract the deep features F df (H×W×C)

[0050] F df =H df (F 0 ) (2);

[0051] Among them, H df (·) is composed of m residual BET modules cascaded, with H BET (·) indicates that the residual connection directly passes the input to the subsequent layers, effectively alleviating the gradient vanishing and gradient exploding problems, accelerating the convergence speed, and supporting deeper network structures. This makes deep neural networks easier to train and optimize.

[0052] H df(i) =H BET (F i-1 )i=1,2…k (3);

[0053] BET is composed of n PAFM modules stacked and connected to a 3x3Conv. PAFM (·) and H Conv (·)express.

[0054] H BET (F i-1 )=H PAFM (F i-1 )+H Conv (F i-1 )i=1,2…k (4);

[0055] The PAFM consists of a CNN auxiliary branch H CNN (·)(Including SE (Squeeze and Excitation) module and Channel Attention module). W-MSA branch H W-MSA (·) Two-way branch. α is the weight factor of the CNN branch. Therefore:

[0056] H PAFM (F j-1 )=α·H CNN (F j-1 )+H W-MSA (F j-1 )j=1,2…k (5);

[0057] Step 3: High-resolution image reconstruction (HR Image Reconstruction);

[0058] Reconstruct high-quality images by aggregating shallow and deep features. SR , where H RE (·) is the function of the reconstruction module. The shallow features mainly contain low-frequency information, while the deep features focus on recovering the lost high-frequency information while upsampling the features using sub-pixel convolution.

[0059] I SR =H RE (F 0 +F df ) (6);

[0060] Furthermore, the loss function

[0061] For image SR, by minimizing L 1 Pixel loss to optimize network parameters and are the i-th reconstructed image and the high-resolution original image respectively.

[0062]

[0063] 2. Module introduction;

[0064] 2.1 Shallow feature extraction;

[0065] Shallow feature extraction is a common step in image processing and computer vision to extract preliminary feature information from the input image. These features usually include information such as edges and textures, which provide a basis for subsequent deep feature extraction and tasks. This module mainly includes a 3x3 convolution and ReLU activation function. By sliding the convolution kernel on the input image, the dot product between the convolution kernel and the local area of ​​the input image is calculated. The result of the convolution operation is represented as a new feature map. After the convolution operation, a nonlinear transformation is performed through the ReLU activation function to increase the expressive power of the model.

[0066] 2.2 Deep feature extraction;

[0067] Combination Figure 1 and Figure 2 The main structure of deep feature extraction includes the cascade of several BET modules and the residual connection of each module. Each BET is mainly composed of the cascade of several PAFM modules, and finally connected to a 3x3 convolution module and residual connection.

[0068] PAFM (Parallel Attention Fusion Module)

[0069] As above Figure 2 The PAFM module shown in the figure is a parallel image processing module that processes and optimizes image features simultaneously through dual branches. Each path has different functional modules to extract and enhance image features and improve the adaptability of the model to different types of inputs. There are mainly convolution branches and W-MSA branches. In this module:

[0070] First, Layer Norm is used to make the mean of the features 0 and the standard deviation 1, which helps to speed up the training of the network.

[0071] After that, the input image is copied to two parallel paths, each of which performs different feature processing tasks. The convolution branch mainly includes the Squeeze and Excitation (SE) module and the Channel Attention (CA) module.

[0072] The SE module adjusts the channel weights of image features, increases the weights of important channels, and reduces the weights of unimportant channels, so that the network can focus more on important features. It mainly includes two 3x3Conv layers with a GLUE activation function in the middle.

[0073] The CA module further enhances the focus on important channel features and enables the network to identify which channel features are more important by calculating the attention weights. It mainly includes a global average pooling layer, followed by a convolutional layer with a ReLU activation function and a convolutional layer with a Sigmoid activation function.

[0074] Among them, the nonlinear characteristics of the activation function enable the neural network to process and represent complex nonlinear relationships, thereby improving the expressiveness of the model. In addition, it can also control the output range and increase the convergence speed to enhance the training effect of the model.

[0075] Given an input of size H×W×C, H and W are the height and width of the feature map space, respectively, and C is the channel dimension size. After inputting it into the convolution branch:

[0076] In the SE module, after the first convolutional layer, the specification becomes The GLUE activation function is still After the second convolutional layer, the number of channels changes back to C, so the size is H×W×C.

[0077] When passing through the CA module, the feature map first undergoes global average pooling to become 1×1×C, and then enters the first convolutional layer to become After the ReLU activation function, the specification remains unchanged, and then it enters the second convolutional layer. The number of channels changes back to C, with a specification of 1×1×C. After the Sigmoid function, the specification remains 1×1×C. The result at this time is the weight of the channel attention, which is element-wise multiplied with the input feature map to obtain the enhanced feature map. Therefore, the final output result is the enhanced feature map with a specification of H×W×C.

[0078] At the same time, the W-MSA branch is parallel to the convolution branch. This module is based on the standard multi-head self-attention of the original Transformer layer. It introduces local attention and window displacement mechanism into it to construct a hierarchical feature map. At the same time, the displacement mechanism can enable information interaction between windows and realize window interleaving, which reduces the amount of calculation while increasing the window size of attention in a directional way. The module is mainly composed of a multi-head self-attention module based on shifted windows (Multi-head Self-Attention, MSA

[0079] For MSA, given an input of size H×W×C, the input is first divided into non-overlapping M×M local windows and the input is reshaped into Features, among which is the total number of windows. Then, the standard self-attention is calculated for each window (i.e., local attention). For a local window feature X(M2 ×C), the calculation formulas of Q, K and V matrices are:

[0080] Q = XP Q

[0081] K=XP K

[0082] V=XP V (8);

[0083] Where P Q , P K and P V The projection matrix specification shared by different windows is (M 2 ×d). The attention matrix is ​​calculated by the self-attention mechanism within the local window as:

[0084]

[0085] After the two-branch operation is completed, the convolution branch is added to the W-MSA branch according to the α weight for information fusion. Finally, it passes through two layers of normalization (Layer Norm, LN) and multi-layer perceptron (Multi-layer Perceptron, MLP) and performs residual connection before output.

[0086] The MLP layer uses two fully connected layers with a GELU nonlinear activation function between layers to further transform features. The fully connected layer connects each input data to each neuron inside it. Each connection has a weight, which is adjusted during the training process so that the neural network can better complete the task. The output of the fully connected layer is the weighted sum of the input data and the corresponding weight, which is then transformed nonlinearly through an activation function to increase the expressive power of the model. In this way, the fully connected layer can transform the input data into new and more useful features, providing richer information for subsequent operations.

[0087] In PAFM, the whole process is expressed as:

[0088] X=MSA(LN(X))+α·CA(SE(LN(X)))+X;

[0089] X = MLP (LN (X)) + X (10);

[0090] 2.3 High-resolution image reconstruction;

[0091] The shallow features, deep features, and the original input features are fused through residual connections to ensure the effective combination of low-frequency information and high-frequency details. The fused features are upsampled by sub-pixel convolution to improve the resolution. Sub-pixel convolution is a technique for image upsampling, especially for super-resolution tasks. Unlike traditional upsampling methods (such as nearest neighbor interpolation, bilinear interpolation, or deconvolution), sub-pixel convolution can upsample more efficiently and has better results during training.

[0092] Sub-pixel convolution increases the number of channels and then upsamples the feature map from a lower resolution to a higher resolution using a pixel shuffle operation. The pixel shuffle operation is a special transformation that rearranges pixels at the same spatial location in multiple channels into a larger spatial dimension.

[0093] The size of the input feature map is H×W×C, and it is upsampled to sH×sW×C, where s is the upsampling scale factor. Sub-pixel convolution passes through the convolution layer to convert the input feature map from H×W×C to H×W×(s 2 C), and then perform a PixelShuffle operation to rearrange the channels of the feature map into a larger spatial dimension, namely sH×sW×C.

[0094] Example:

[0095] The training dataset used in this experiment is the DIV2K dataset.

[0096] Using the DIV2K dataset, we used x2, x3, and x4 bicubic downsampling to process high-resolution images (HR) to generate paired low-resolution data for training the BETSR model. The test set was tested using the Benchmark dataset, which includes Set5, B100, Set14, Urabn100, and Manga datasets.

[0097] DIV2K has images with 2K resolution (e.g., 2048×1080). In order to make full use of input image resources and speed up IO speed during training, this experiment crops the 2K resolution images into sub-images as the original input images for model training (cropped into 480x480 sub-images in the experiment).

[0098] The model in this paper is implemented using the PyTorch framework. During the training process, we randomly cropped image patches of size 48×48 from the training sub-atlas as training input, and used horizontal flipping and random rotation for data augmentation. The total number of training iterations was set to 500K and the initial learning rate was set to 2×10 -4, use the learning rate scheduler MultiStepLR, and reduce it by half at 250K, 400K, 450K, and 475K. In this way, the learning rate is multiplied by a coefficient (gamma) of 0.5 at the preset milestones. In this way, the learning rate can be gradually reduced during the training process, so that the model can adjust the parameters more finely in the later stage of training. Using β 1 =0.9,β 2 =0.99 and updated using the Adam optimizer. All experiments were performed on an NVIDIA RTX 4090Ti GPU. The peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) were used as evaluation indicators for quantitative results analysis, where the RGB image format was first converted to the YCbCr format and then the indicators were calculated on the Y channel.

[0099] The loss function of this model adopts L1 loss as shown in formula (7).

[0100] Results evaluation:

[0101] Objective quantitative evaluation usually uses Peak Signal-Noise Ratio (PSNR) and Structure-Similarity Index (SSIM) as evaluation criteria.

[0102] PSNR is simply defined by the mean square error (MSE). The mathematical expression of PSNR is as follows:

[0103] As shown in formula 11 and 12:

[0104]

[0105] Among them, f MSE represents the root mean square error between the original image and the reconstructed image, f m represents the maximum possible value of a pixel (for example, for an 8-bit image, f m =2 8 -1 = 255. A higher PSNR value indicates a smaller distortion of the reconstructed image, expressed in dB.

[0106] SSIM is an indicator to measure the structural similarity of two images. Structural similarity can be divided into three attributes: brightness, contrast, and image structure from the perspective of image composition. The value range of SSIM is [-1, 1]. The closer it is to 1, the more similar the reconstructed image (SR) is to the original high-resolution image (HR). The mathematical expression of SSIM is as shown in Equation 12-Equation 19:

[0107]

[0108] SSIM=f L (I SR , I HR )*f C (I SR , I HR )

[0109] *f S (I SR , I HR ) (19)

[0110] Among them, μ x represents the image mean, σ x represents the image variance, σ xy represents the covariance between the SR image and the HR image. L (I SR , I HR ), f C (I SR , I HR ) and f S (I SR , I HR ) represent the brightness similarity, contrast similarity, and structure similarity of the SR image and the HR image respectively. 1 , C 2 and C 3 It is a constant set to avoid the denominator in the fraction being 0, as shown in Equation 3.10-Equation 3.12:

[0111] C 1 =(K 1 L) 2 (20)

[0112] C 2 =(K 2 L) 2 (twenty one)

[0113]

[0114] Among them, K 1 Usually the value is 0.1, K 2 The value is usually 0.03, and the L value is the range of pixels. For example, for an 8-bit image, the value of L is 255.

[0115] Experimental results:

[0116] For the traditional single image super-resolution (SISR), this paper selects some classic or advanced methods for comparison. They all use the DIV2K dataset as the training dataset and the Benchmark dataset as the test dataset, including Set5, B100, Set14, Urabn100, and Manga109 datasets.

[0117] Quantitative results:

[0118] The x2, x3, and x4 super-resolution results for the test data are shown in Tables 1 to 3 below (the bold data are the best in the same group).

[0119] The table shows the quantitative comparison results of this method with other selected methods: IGNN, HAN, SMSR, NLSN, SwinIR methods. It can be seen that the proposed method outperforms other methods in 27 of the 30 indicators including PSNR and SSIM of three super-resolution multiples on all five benchmark datasets. Since the image style of Manga109 data is anime images and is different from the trained DIV2K dataset, which has a slight impact on the model, the remaining 3 items are ranked second, but overall, the proposed method is superior to other comparison methods. This is due to the dual-branch combination design in the PAFM module, which effectively captures local knowledge and global connectivity, enabling it to achieve superior results on various test sets, proving the potential of this method in SR tasks.

[0120] Table 1 - Super-resolution results x2

[0121]

[0122] Table 2 - Super-resolution results x3

[0123]

[0124] Table 3 - Super-resolution results x4

[0125]

[0126]

[0127] Qualitative comparison:

[0128] The results are as follows Figure 3 , Figure 4As shown in the figure, the SR quality of these different methods is qualitatively compared. Considering that SMSR is not an officially provided x4 test model, it is not included in the comparison. The image block on the right is the enlarged result of the red frame captured from the overall image. HR is the original high-resolution image block. For image_012 of Urban100, it can be seen from the figure that HAN, NLSN, SwinIR, etc., obviously have ripple-like error lines in the construction of details. IGNN is not particularly obvious, but compared with the method of this paper, the line texture of the method of this paper is relatively clearer. For Barbara.png of Set14, it is very difficult to restore the grid texture. Other methods restore it to stripes. Compared with other methods, the method proposed in this paper also shows the result of the grid. Therefore, when other methods cannot restore the correct texture or produce severely blurred images, the BET method of this paper obtains clearer and correct textures than other methods. The qualitative comparison of vision also proves the superiority of the method of this paper in single image super-resolution.

Claims

1. A Transformer image super-resolution reconstruction method based on branch enhancement, characterized in that: The following steps are involved: Step 1: Shallow feature extraction: Given a low-resolution input image I LR (H×W×C in )H, W and C in are the height, width and number of input channels of the image respectively, and a 3×3 convolution layer H is used. sf (·) Extract shallow feature F0, see formula (1); F0=H sf (I LR ) (1); Step 2: Deep feature extraction; Extract deep features F from shallow features F0 df (H×W×C), see formula (2); F df =H df (F0) (2); Among them, H df (·) is composed of m residual BET modules cascaded, with H BET (·) indicates that the residual connection accelerates the convergence speed by directly passing the input to the subsequent layer, see formula (3); H df(i) =H BET (F i-1 )i=1,2…k (3); BET is composed of n PAFM modules stacked and connected to a 3x3Conv. PAFM (·) and H Conv (·) indicates, see formula (4); H BET (F i-1 )=H PAFM (F i-1 )+H Conv (F i-1 )i=1,2…k (4); The PAFM consists of a CNN auxiliary branch H CNN (·)(Including SE (Squeeze and Excitation) module and ChannelAttention module, W-MSA branch H W-MSA (·) Two-way branching, α is the weight factor of the CNN branch, therefore; H PAFM (F j-1 )=α·H CNN (F j-1 )+H W-MSA (F j-1 )j=1,2…k (5): Step 3: high-resolution image reconstruction; Reconstruct high-quality images by aggregating shallow features and deep features. SR , where H RE (·) is the function of the reconstruction module. The shallow features mainly contain low-frequency information, and the deep features contain the lost high-frequency information. At the same time, sub-pixel convolution is used to upsample the features to reconstruct a high-quality image I SR See formula (6); I SR =H RE (F0+F df ) (6).

2. According to claim 1, a Transformer image super-resolution reconstruction method based on branch enhancement is characterized in that: For super-resolution reconstruction of SR images, the loss function optimizes the network parameters by minimizing the L1 pixel loss. and are the i-th reconstructed image and the high-resolution original image respectively; Among them, the L1 loss function calculates the sum of the absolute differences between the actual value and the target value. In such supervised image super-resolution tasks, the goal is to make the generated super-resolution reconstructed image (SR) as close as possible to the real high-resolution image (HR). The L1 loss is used to calculate the error between the values ​​of the corresponding pixel positions of SR and HR.

3. According to the Transformer image super-resolution reconstruction method based on branch enhancement in claim 1, it is characterized in that: The shallow feature extraction module mainly includes a 3x3 convolution and ReLU activation function. By sliding the convolution kernel on the input image, the dot product between the convolution kernel and the local area of ​​the input image is calculated. The result of the convolution operation is represented as a new feature map. After the convolution operation, the nonlinear transformation is performed through the ReLU activation function to increase the expression ability of the model.

4. The Transformer image super-resolution reconstruction method based on branch enhancement according to claim 1, characterized in that: The main structure of deep feature extraction includes the cascade of several BET modules and the residual connection of each module. Each BET is mainly composed of the cascade of several PAFM modules, and finally connected to a 3x3 convolution module and residual connection.

5. The Transformer image super-resolution reconstruction method based on branch enhancement according to claim 1, characterized in that: The PAFM module is a parallel image processing module, including a convolution branch and a W-MSA branch, which is used to extract and enhance image features and improve the adaptability of the model to different types of inputs.

6. The Transformer image super-resolution reconstruction method based on branch enhancement according to claim 5, characterized in that: In the PAFM module: First, through Layer Norm, the mean of the feature is set to 0 and the standard deviation is set to 1, which helps to speed up the training of the network. After that, the input image is copied to two parallel paths, each of which performs different feature processing tasks. The convolution branch mainly includes the Squeeze and Excitation (SE) module and the Channel Attention (CA) module. The SE module adjusts the channel weights of image features, increases the weights of important channels, and reduces the weights of unimportant channels, so that the network can focus more on important features. It mainly includes two 3x3Conv layers with a GLUE activation function in the middle. The CA module further enhances the focus on important channel features. By calculating the attention weights, the network can identify which channel features are more important. It includes a global average pooling layer, followed by a convolutional layer plus a ReLU activation function and a convolutional layer plus a Sigmoid activation function.

7. The Transformer image super-resolution reconstruction method based on branch enhancement according to claim 1, characterized in that: The high-resolution image reconstruction described above fuses shallow features, deep features, and initial input features through residual connections, ensuring an effective combination of low-frequency information and high-frequency details. The fused features are upsampled by sub-pixel convolution to improve resolution.

8. The Transformer image super-resolution reconstruction method based on branch enhancement according to claim 7 is characterized in that: The sub-pixel convolution increases the number of channels and then upsamples the feature map from a lower resolution to a higher resolution using a pixel shuffle operation. The pixel shuffle operation is a special transformation that rearranges pixels at the same spatial position in multiple channels into a larger spatial dimension.

Images