A Transformer single image super-resolution reconstruction method based on cross-scale token interaction

Abstract

The present invention discloses a Transformer single image super-resolution reconstruction method based on cross-scale token interaction, comprising: 1. Obtaining a low-resolution image, and using a shallow feature extraction module SFE to extract preliminary shallow features from the low-resolution image; 2. Building a backbone network based on a number of deep feature extraction blocks DFE, wherein the deep feature extraction block DFE includes a number of cross-scale token interaction Transformer layers, and inputting shallow features into the backbone network to extract deep features; 3. Fusing the extracted shallow features and deep features to reconstruct a high-resolution image. The present invention realizes the interaction between token information at different levels, fully mines and fuses cross-scale information in the image, and strengthens the interaction of global information. The multi-scale information extraction module is designed to first pool the original pixel tokens, generate regional tokens and global tokens, thereby realizing the effective extraction of multi-scale information.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing and machine vision, and in particular relates to a Transformer single image super-resolution reconstruction method based on cross-scale token interaction. Background Art

[0002] In the field of digital image processing and computer vision, image super-resolution (SR) technology is dedicated to restoring high-resolution (HR) images from low-resolution (LR) images, thereby improving image quality and details. Due to imaging equipment limitations, storage and transmission constraints, and external interference, obtaining high-quality images faces many challenges, and image super-resolution technology can not only effectively reduce costs and shorten time, but also has a wide range of applications in medical imaging, satellite photography, security monitoring, industrial detection and other fields.

[0003] Image super-resolution reconstruction is an ill-posed inverse problem with great uncertainty. The main methods are divided into traditional methods and deep learning-based methods. Traditional methods include interpolation, reconstruction, and traditional machine learning methods. Interpolation methods amplify images by estimating missing pixel values, but often lead to blurred results; reconstruction methods use multiple low-resolution images to reconstruct high-resolution images, but the computational complexity is high; traditional machine learning methods rely on manually designed features, and the effect is limited by feature selection. Although these methods have their own advantages and disadvantages, they are limited in processing high-frequency details and complex image structures. Deep learning-based methods automatically learn the complex mapping relationship between LR and HR images through deep neural networks, significantly improve image quality, and are excellent in generating clear and delicate high-resolution images.

[0004] As deep learning has achieved great success in the field of computer vision, Dong et al. [Dong C, Loy CC, He K, et al. Image super-resolution using deep convolutional networks[J].IEEE transactions on pattern analysis and machine intelligence, 2015, 38(2):295-307.] first introduced deep learning methods into the image super-resolution reconstruction task in 2014. With the powerful learning ability of neural networks, they achieved results that were superior to traditional methods. Subsequently, researchers proposed a series of continuously optimized algorithm models, from the earliest SRCNN (Super-Resolution Convolutional Neural Network) model based on convolutional neural networks, to the SRGAN (Super-Resolution Generative Adversarial Network) model based on generative adversarial networks, to the TTSR (Texture Transformer Network for SR) model based on Transformer. Image super-resolution reconstruction technology based on deep learning has made continuous breakthroughs. However, these methods also have some problems and challenges. Although SRCNN is simple and easy to use, it is difficult to capture high-frequency details and complex textures. SRGAN introduces an adversarial training mechanism, which generates higher quality images, but the training process is unstable and prone to artifacts. TTSR captures long-distance dependencies through Transformer, enhancing details and texture reconstruction, but the computational complexity is high, resulting in long training and inference time and high resource consumption.

[0005] Since Transformer-based models generally have large memory and computational overhead, Swin Transformer-based models with smaller memory and computational overhead have become the mainstream of research. Swin Transformer uses a sliding window mechanism to perform self-attention operations within each window and capture global information by shifting the window. This design greatly reduces the computational complexity and memory consumption, making Swin Transformer more efficient and lightweight while maintaining high performance. Most of the current research is based on the SwinIR (Image Restoration Using SwinTransformer) baseline model. Although Swin Transformer has significant advantages, it also has some problems and limitations. First, although the sliding window mechanism improves computational efficiency, it still has deficiencies in cross-window information interaction and global information interaction, resulting in poor results when processing large-scale images. In addition, the sliding window operation may cause boundary effects, affecting the continuity and consistency of the reconstruction results. Second, the existing SwinIR model often fails to fully utilize the cross-scale similarity information in the image, which is crucial for further improving the image super-resolution reconstruction effect. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a Transformer single image super-resolution reconstruction method based on cross-scale token interaction, to achieve interaction between token information at different levels, to fully mine and fuse cross-scale information in the image, and to strengthen the interaction of global information.

[0007] Technical solution: A Transformer single image super-resolution reconstruction method based on cross-scale token interaction of the present invention comprises the following steps:

[0008] Step 1: Obtain a low-resolution image and use a shallow feature extraction module SFE to extract preliminary shallow features from the low-resolution image;

[0009] Step 2: Build a backbone network based on several deep feature extraction blocks DFE, where the deep feature extraction block DFE includes several cross-scale token interaction transformer layers CSTITL (Cross-Scale Token Interaction Transformer Layer), and input shallow features into the backbone network to extract deep features;

[0010] Step 3: Fuse the extracted shallow features and deep features to reconstruct a high-resolution image.

[0011] Furthermore, step 1 is as follows: use a layer Standard convolution transforms low-resolution images Map to the high-level feature space to obtain shallow features , where C represents the number of channels, Represents the feature map size, the formula is as follows:

[0012]

[0013] Furthermore, step 2 specifically includes the following steps:

[0014] Step 201: Each deep feature extraction block outputs , and finally a convolutional layer is used to obtain the deep feature map , the formula is as follows:

[0015]

[0016]

[0017] Step 202, the deep feature extraction block in the backbone network includes L cross-scale token interaction Transformer layers CSTITL, which are used to achieve global and multi-level information fusion, and a 3×3 convolutional layer, which focuses on the aggregation of local information. The formula is as follows:

[0018]

[0019] Step 203: Considering the extraction of multi-scale information, the multi-scale information extraction module MSIE in the cross-scale token interaction Transformer layer CSTITL extracts the input feature map , first perform step-by-step pooling downsampling to obtain the regional token feature map and the global token feature map , and then calculate the sliding window self-attention WSA to obtain ;

[0020] Step 204: The cross-scale information fusion module CSIF in the cross-scale token interaction transformer layer CSTITL first fuses the cross-scale information of the pixel token and the region token. First, the pixel tokens corresponding to the pixel feature map (i, j) are taken out. region tokens, with pixel tokens as query Q, region tokens as key K and value V. Before calculating the feature map, the feature dimensions of Q and K are compressed through a linear layer to reduce the amount of calculation, and then the attention calculation is performed to obtain the pixel tokens that fuse the region information. , supplement the regional information in the feature map;

[0021] Step 205: Then, the cross-scale information fusion module CSIF is used to fuse the cross-scale information of the pixel token and the global token. The fusion process is similar to the interaction between the pixel token and the regional token. The only difference is that each pixel token must calculate the attention with all the global tokens, thereby obtaining the pixel token that fuses the global information. ;

[0022] Step 206: Get Finally, they are added and sent to the deep convolution gated forward propagation layer DWConvFFL to extract deep features.

[0023] Further, step 203 is specifically as follows: firstly, a step-by-step pooling downsampling is performed to obtain a regional token feature map and the global token feature map , considering that no additional parameters are added, the parameters of the shared linear layer are given by the following formula:

[0024]

[0025]

[0026]

[0027] Window attention is about to be given an input Divide into Non-overlapping windows, and then calculate the standard self-attention restoration of each window respectively, and input the window feature map , the calculation formula of self-attention is as follows:

[0028]

[0029] Where R is the learnable relative position encoding, Q is the query, K is the key, V is the value, T represents the matrix transpose operation, and d represents the scaling factor.

[0030] Furthermore, in step 204, the pixel token is used as the query Q, the region token is used as the key K and the value V, and the feature dimensions of Q and K are compressed by a linear layer before calculating the feature map to reduce the amount of calculation, and then the attention calculation is performed to obtain the pixel token that integrates the region information. , the specific formula is as follows:

[0031]

[0032]

[0033] in, It means taking the pth token from the pixel token. Indicates the value taken from the region token and The closest n in space 2 The process is repeated until all pixel tokens are calculated.

[0034] Further, in step 206, the deep convolution gated forward propagation layer DWConvFFL is the element-by-element product of two parallel paths of the linear transformation layer, one of which is activated by the GELU nonlinearity, and a deep convolution is added therein to encode the information of spatially adjacent pixel positions, and the formula is as follows:

[0035]

[0036]

[0037]

[0038] in, Representing element-wise products, the depthwise convolutional gated prior propagation controls the flow of information across the layers in our pipeline, allowing each level to focus on details that are complementary to other levels.

[0039] Furthermore, step 3 is specifically as follows: the shallow features obtained in step 2 The deep features obtained in step 3 Feature fusion is performed through skip connections, and then sub-pixel convolution upsampling is used to obtain a reconstructed high-resolution image. , the formula is as follows:

[0040]

[0041] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0042] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0043] The present invention also discloses a computer program product, comprising a computer program / instruction, which implements the steps of the method of the present invention when executed by a processor.

[0044] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0045] 1. In terms of multi-scale information extraction, a multi-scale information extraction module is designed to first pool the original pixel tokens to generate regional tokens and global tokens, and then calculate the (sliding) window self-attention (WSA) respectively, so as to achieve effective extraction of multi-scale information and strengthen the interaction of global information;

[0046] 2. In terms of cross-scale information fusion, a cross-scale information fusion module is designed to perform attention calculation on cross-scale tokens, realize the interaction between token information at different levels, fully mine and fuse cross-scale information in the image, and strengthen the interaction of global information;

[0047] 3. In terms of lightweight optimization, linear layer parameters are shared during multi-scale information extraction to reduce the number of parameters. Before calculating the cross-scale token feature map, the query (Q) and key (K) values ​​are compressed through a linear layer to reduce the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The overall framework diagram of the model proposed by the present invention;

[0049] Figure 2 is a multi-scale information extraction module of the present invention;

[0050] Figure 3 The cross-scale information interaction method proposed by the present invention;

[0051] Figure 4 Schematic diagram of the deep convolutional gated forward propagation layer of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0053] The present invention provides a Transformer single image super-resolution reconstruction method based on cross-scale token interaction. The overall framework of the model is as follows: Figure 1 As shown in (a) in the figure, the following steps are included:

[0054] Step 1: Use the shallow feature extraction module (SFE) to extract Extract preliminary feature information from the dataset, providing a basis for subsequent deep feature extraction.

[0055] Further, the step 1 is specifically as follows:

[0056] Step 101, the specific process is to use a layer Standard convolution transforms low-resolution images Map to the high-level feature space to obtain shallow features , where C represents the number of channels, Represents the feature map size, the formula is as follows:

[0057]

[0058] Step 2: The shallow features obtained in step 1 The deep feature information is then sent to the backbone network (consisting of B deep feature extraction blocks DFE) for deep feature extraction to enhance the feature information in order to obtain a more accurate reconstruction effect.

[0059] Further, the step 2 is specifically as follows:

[0060] Step 201, each deep feature extraction block outputs , and finally a convolutional layer is used to obtain the deep feature map , the formula is as follows:

[0061]

[0062]

[0063] Step 202, such as Figure 1 As shown in (b), the deep feature extraction block in the backbone network contains L cross-scale token interaction Transformer layers (CSTITL) to achieve global and multi-level information fusion, and a 3×3 convolutional layer that focuses on the aggregation of local information. The formula is as follows:

[0064]

[0065] Step 203, considering the extraction of multi-scale information, the multi-scale information extraction module (MSIE) in the cross-scale token interaction transformer layer (CSTITL) extracts the input feature map , first perform step-by-step pooling downsampling to obtain the regional token feature map and the global token feature map , and then calculate the (sliding) window self-attention (WSA) to obtain , considering that no additional parameters are added, here they share the parameters of the linear layer, the formula is as follows:

[0066]

[0067]

[0068]

[0069] Window attention is about to be given an input Divide into Non-overlapping windows, and then calculate the standard self-attention restoration of each window respectively, and input the window feature map , the calculation formula of self-attention is as follows:

[0070]

[0071] Where R is a learnable relative position code, the process is as follows Figure 2 As shown;

[0072] Step 204, such as Figure 1 As shown in (c) of the figure, considering the fusion of cross-scale information, the cross-scale information fusion module (CSIF) in the cross-scale token interaction transformer layer (CSTITL) first fuses the cross-scale information of pixel tokens and region tokens. First, the pixel tokens corresponding to the pixel feature map (i, j) are taken out. region tokens, with pixel tokens as queries (Q), region tokens as keys (K) and values ​​(V). It is worth noting that before calculating the feature map, the feature dimensions of Q and K are compressed through a linear layer to reduce the amount of calculation, and then the attention calculation is performed to obtain the pixel token that integrates the region information. , the regional information in the feature map is supplemented, and the calculation formula is as follows:

[0073]

[0074]

[0075] Step 205, then use the cross-scale information fusion module (CSIF) to fuse the cross-scale information of the pixel token and the global token, such as Figure 3 As shown in the figure, the interaction between pixel tokens and region tokens is similar. The only difference is that each pixel token must calculate attention with all global tokens, thus obtaining a pixel token that integrates global information. .

[0076] Step 206, get After that, they are added and sent to the deep convolution gated forward propagation layer (DWConvFFL), as shown in Figure 4 As shown in Figure 1, this introduces nonlinear transformation to the features and improves the expressiveness of the network model. Specifically, the gating mechanism is expressed as the element-by-element product of two parallel paths of the linear transformation layer, one of which is activated by the GELU nonlinearity, and a deep convolution is added to encode the information of spatially adjacent pixel positions. The formula is as follows:

[0077]

[0078]

[0079]

[0080] in Representing element-wise products, the depthwise convolutional gated prior propagation controls the flow of information across the layers in our pipeline, allowing each level to focus on details that are complementary to other levels.

[0081] Step 3: Fuse the extracted shallow features and deep features to reconstruct a high-resolution image.

[0082] Further, the step 3 is specifically as follows:

[0083] Step 301: The shallow features obtained in step 2 The deep features obtained in step 3 Feature fusion is performed through skip connections, and then sub-pixel convolution upsampling is used to obtain a reconstructed high-resolution image. , the formula is as follows:

[0084]

[0085] Table 1 Comparison between the present invention and the current mainstream method

[0086]

[0087] As shown in Table 1, under moderate parameters, most indicators of the present invention on the benchmark test dataset are better than those of the current mainstream methods, and a significant improvement is achieved on the Manga109 cartoon image dataset, which fully proves the effectiveness of the present invention.

Claims

1. A Transformer single image super-resolution reconstruction method based on cross-scale token interaction, characterized in that: The steps include: Step 1: Obtain a low-resolution image and use a shallow feature extraction module SFE to extract preliminary shallow features from the low-resolution image; Step 2: construct a backbone network based on several deep feature extraction blocks DFE, wherein the deep feature extraction block DFE includes several cross-scale token interaction Transformer layers CSTITL, and inputs shallow features into the backbone network to extract deep features; Step 3: Fusing the extracted shallow features and deep features to reconstruct a high-resolution image; Step 2 specifically includes the following steps: Step 201: Each deep feature extraction block outputs There are K deep feature extraction blocks in total, and finally a deep feature map is obtained through a convolution layer , the formula is as follows: Step 202, the deep feature extraction block in the backbone network includes L cross-scale token interaction Transformer layers CSTITL, which are used to achieve global and multi-level information fusion, and a 3×3 convolutional layer, which focuses on the aggregation of local information. The formula is as follows: Step 203: Considering the extraction of multi-scale information, the multi-scale information extraction module MSIE in the cross-scale token interaction Transformer layer CSTITL extracts the input feature map , first perform step-by-step pooling downsampling to obtain the regional token feature map and the global token feature map , and then calculate the sliding window self-attention WSA to obtain ; Step 204: The cross-scale information fusion module CSIF in the cross-scale token interaction transformer layer CSTITL first fuses the cross-scale information of the pixel token and the region token. First, the pixel tokens corresponding to the pixel feature map (i, j) are taken out. region tokens, with pixel tokens as query Q, region tokens as key K and value V. Before calculating the feature map, the feature dimensions of Q and K are compressed through a linear layer to reduce the amount of calculation, and then the attention calculation is performed to obtain the pixel tokens that fuse the region information. , supplement the regional information in the feature map; Step 205: Then, the cross-scale information fusion module CSIF is used to fuse the cross-scale information of the pixel token and the global token. The fusion process is similar to the interaction between the pixel token and the regional token. The only difference is that each pixel token must calculate the attention with all the global tokens, thereby obtaining the pixel token that fuses the global information. ; Step 206: Get Finally, they are added and sent to the deep convolution gated forward propagation layer DWConvFFL to extract deep features; Step 203 is as follows: first, perform step-by-step pooling downsampling to obtain a regional token feature map. and the global token feature map , considering that no additional parameters are added, the parameters of the shared linear layer are given by the following formula: Window attention is about to be given an input Divide into Non-overlapping windows, and then calculate the standard self-attention restoration of each window respectively, and input the window feature map , the calculation formula of self-attention is as follows: Where R is the learnable relative position encoding, Q is the query, K is the key, V is the value, T represents the matrix transpose operation, and d represents the scaling factor; In step 204, the pixel token is used as the query Q, the region token is used as the key K and the value V. Before calculating the feature map, the feature dimensions of Q and K are compressed by a linear layer to reduce the amount of calculation, and then the attention calculation is performed to obtain the pixel token that integrates the region information. , the specific formula is as follows: in, It means taking the pth token from the pixel token. Indicates the value taken from the region token and The closest n in space 2 The process is repeated until all pixel tokens are calculated.

2. According to claim 1, a Transformer single image super-resolution reconstruction method based on cross-scale token interaction is characterized in that: Step 1 is as follows: Use a layer Standard convolution transforms low-resolution images Map to the high-level feature space to obtain shallow features , where C represents the number of channels, Represents the feature map size, the formula is as follows:

3. According to the Transformer single image super-resolution reconstruction method based on cross-scale token interaction according to claim 1, it is characterized in that: In step 206, the deep convolution gated forward propagation layer DWConvFFL is the element-by-element product of two parallel paths of the linear transformation layer, one of which is activated by the GELU nonlinearity, and a deep convolution is added to encode the information of spatially adjacent pixel positions. The formula is as follows: in, Representing element-wise products, the depthwise convolutional gated prior propagation controls the flow of information across the layers in our pipeline, allowing each level to focus on details that are complementary to other levels.

4. According to claim 1, a Transformer single image super-resolution reconstruction method based on cross-scale token interaction is characterized in that: Step 3 is as follows: the shallow features obtained in step 1 The deep features obtained in step 2 Feature fusion is performed through skip connections, and then sub-pixel convolution upsampling is used to obtain a reconstructed high-resolution image. , the formula is as follows:

5. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

6. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images