Remote sensing image super-resolution reconstruction method and system based on residual hierarchical Transform

Abstract

The invention provides a remote sensing image super-resolution reconstruction method and system based on residual hierarchical Transform, and relates to the technical field of digital image processing, and the method comprises the following steps: obtaining a low-resolution image; performing shallow feature extraction on the input low-resolution image through convolutional layer operation to obtain a shallow feature image; secondly, on the basis of the design concept that residual blocks are nested in residual groups, a multi-scale feature calibration mechanism, a Transform space-channel attention mechanism based on a hierarchical window and a space-gate feed-forward network are combined, and a depth feature image is obtained; and finally, performing fusion and up-sampling on the shallow feature image and the depth feature image through convolution and sub-pixel convolution operation to obtain a final remote sensing image super-resolution reconstruction result. By means of the scheme, the remote dependency relationship in the remote sensing image can be effectively modeled, the feature extraction capacity can be improved, and the problem that large-scale window self-attention calculation is high in complexity in a Transform mechanism can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a remote sensing image super-resolution reconstruction method and system based on residual hierarchical Transformer. Background Art

[0002] Remote sensing images acquired based on satellite technology and drone aerial photography technology are important media for earth observation and are widely used in many fields, such as urban monitoring, forestry inspection, military reconnaissance, disaster detection, etc. Generally speaking, the higher the spatial resolution of the corresponding remote sensing image, the more detailed the ground object information can be reflected. However, this type of image will be affected by factors such as the atmosphere and long-distance imaging during the acquisition process, resulting in the problem of low spatial resolution of the acquired image. From a hardware perspective, designing higher-quality optical sensors or aerial cameras can obtain clearer images, but this inevitably leads to expensive manufacturing and maintenance costs. Therefore, how to improve the spatial resolution of remote sensing images from the software algorithm level has become one of the current remote sensing vision tasks with important application prospects and research value.

[0003] Image super-resolution (SR) reconstruction technology aims to reconstruct a low-resolution (LR) image into a clear and realistic high-resolution (HR) image through computer algorithms. Due to the powerful feature extraction capability of Transformer, many Transformer-based SR reconstruction algorithms have emerged in recent years. The current main research algorithms can be divided into two categories: Convolutional Neural Network (CNN)-based and Transformer-based methods. CNN-based networks are better at extracting local features of images and have difficulty modeling long-range dependencies. Transformer networks use self-attention mechanisms to learn global sequence information and effectively model long-range dependencies between images, but have limitations in local feature extraction. At the same time, the computational complexity of Transformer grows quadratically with the increase in the size of the input image, which limits the practical application of this method on large-scale remote sensing images.

[0004] In view of this, how to provide a remote sensing image super-resolution reconstruction method that takes into account both reconstruction effect and model calculation complexity is an issue that technicians in this field urgently need to solve. Summary of the invention

[0005] The present invention aims to provide a remote sensing image super-resolution reconstruction method and system based on residual hierarchical Transformer to solve the shortcomings of existing methods such as lack of long-range dependency between images, limitations in feature extraction, and high computational complexity.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer, comprising:

[0008] S1, obtain low-resolution images;

[0009] S2, performing shallow feature extraction on the low-resolution image through a convolution operation to obtain a shallow feature image;

[0010] S3, based on the design concept of nested residual blocks in residual groups, combined with multi-scale feature calibration mechanism, hierarchical window-based Transformer space-channel attention mechanism and space-gate feedforward network, deep features are extracted from the shallow feature image to obtain a deep feature image;

[0011] S4, fusing and upsampling the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

[0012] Preferably, the formula for shallow feature extraction is:

[0013] F0=f Conv3×3 (I LR );

[0014] Wherein, F0 is the shallow feature image; f Conv3×3 (·) indicates a convolution operation with a kernel size of 3×3; I LR is the low-resolution image.

[0015] Preferably, when performing deep feature extraction, a residual-in-residual connection structure is adopted, that is, a design principle of nesting residual blocks in residual groups; extraction is performed through N1 RG residual groups and a convolutional layer to enhance feature reuse at different levels. The calculation formula is as follows:

[0016]

[0017] Among them, F i represents the output of the i-th RG residual group, and represent the i-1th and ith RG residual group operations respectively; F0 represents the shallow feature image; FDF Representing the depth feature image; represents the output of the N1th RG residual group operation; f Conv3×3 Indicates a convolution operation with a convolution kernel size of 3×3;

[0018] Among them, each RG residual group consists of N2 RHTB blocks and a convolutional layer, and the calculation formula is as follows:

[0019]

[0020] Among them, F i,j represents the output of the jth RHTB residual block operation of the i-th RG residual group, F i,0 represents the input of the i-th RG residual group, F i,out represents the convolution output of the i-th RG residual group, and They represent the j-1th and jth RHTB residual block operations in the i-th RG residual group respectively; represents the output of the N2-th RHTB residual block operation of the i-th RG residual group; F i,0 Represents the input of the i-th RG residual group.

[0021] Preferably, the RHTB residual block includes two sub-residual modules, the first residual module includes a sequentially connected Layer normalization layer, a multi-scale feature calibration MSFM mechanism and a hierarchical window-based spatial channel attention S-CAHW mechanism; the second sub-residual module includes a sequentially connected Layer normalization layer and a spatial gate feedforward network SGFN; wherein the multi-scale feature calibration MSFM mechanism uses the fusion of multi-scale features to learn the long-range dependency between images, and the fusion learning process is specifically as follows: the input feature map is divided into several parts by channel segmentation to obtain feature sub-maps; each feature sub-map is then down-sampled and up-sampled in different spaces and then channel-joined, and then a convolution layer is used to convolve the result of the channel jointing, and the distribution of the estimated attention feature map is used to calibrate the feature map, thereby obtaining the output feature map of the MSFM;

[0022] The spatial-channel attention S-CAHW mechanism uses the output feature map of the MSFM as the input feature, generates a multi-head query matrix and a value matrix after non-overlapping window division and channel segmentation, and then fuses the spatial features and channel features in the hierarchical window to obtain the output feature F S-CAHW , to alleviate the problem of heavy computational burden caused by fixed large-size window setting in Transformer mechanism;

[0023] The spatial-gated feed-forward network SGFN is used to output the feature F of S-CAHW S-CAHWThe features fused with the input of the current RHTB block are then used as input after the output of the Layer normalization layer in the second sub-residual module. After the first linear projection, they are split into two parts. Then, the gate mechanism is used to model the nonlinear spatial features, and then a second linear projection is performed to obtain the output features of SGFN, which are used to supplement the nonlinear features.

[0024] Preferably, the calculation steps of the multi-scale feature calibration MSFM mechanism are as follows:

[0025] Step 1: Definition is the feature map with a total number of channels C, where H×W is the size of a single feature map; the channel segmentation technology is used to divide X into four parts X1, X2, X3, and X4 to reduce the amount of calculation in subsequent steps. The calculation formula is as follows:

[0026] X1,X2,X3,X4=f split (X);

[0027] Among them, f split (·) represents the channel splitting operation,

[0028] Step 2: Perform different spatial downsampling and upsampling processing on X1, X2, X3, and X4. Since smaller feature maps have larger receptive fields, after convolution with the same convolution kernel size, the receptive fields of each pixel in feature maps of different scales are different. In order to facilitate the subsequent feature map channel splicing operation, the previously downsampled feature maps are upsampled back in turn through the bilinear interpolation algorithm. The calculation formula is as follows:

[0029]

[0030] Among them, f DW-Conv3×3 (·) represents a depthwise separable convolution operation with a kernel size of 3×3; f DS-2 (·),f DS-4 (·) and f DS-8 (·) respectively represent the downsampling of the input feature size using the maximum pooling operation to and f US-2 (·),f US-4 (·) and f US-8 (·) represents the bilinear interpolation algorithm with upsampling factors of 2, 4, and 8, respectively;

[0031] They represent the processing results of spatial downsampling and upsampling at different scales respectively;

[0032] Step 3: Get the multi-scale features Perform channel splicing operations and use a convolutional layer to perceive local and global correlations. The calculation formula is as follows:

[0033]

[0034] in, is the result after feature fusion, f concat (·) represents the channel concatenation operation, f Conv1×1 (·) indicates a convolution operation with a kernel size of 1×1;

[0035] Step 4: Use GELU nonlinear activation function to estimate The attention feature map distribution is used, and X is adaptively calibrated through the attention feature map to obtain the output result of the final MSFM module. The calculation formula is as follows:

[0036]

[0037] Among them, f GELU (·) represents the GELU nonlinear activation function, ⊙ represents the element-by-element multiplication operation of the feature map, Represents the output of the MSFM module.

[0038] Preferably, the calculation steps of the spatial channel attention S-CAHW mechanism are as follows:

[0039] Step 1: Input features Perform non-overlapping window division to obtain features Where h and w are the height and width of the window respectively, N = HW / hw, which represents the number of windows; channel segmentation is used to divide F p Divide into two parts to generate the query matrix and value matrix of multiple heads. The calculation formula is as follows:

[0040] Q,V=f split (F p );

[0041] in, represent the query matrix and value matrix respectively; f split (·) indicates channel splitting operation;

[0042] Step 2: Input Q and V into the spatial attention SAHW and channel attention CAHW mechanisms under the hierarchical window to extract multi-scale spatial and channel features. The calculation formula is as follows:

[0043] F SAHW =H WM (H SAHW (Q,V));

[0044] F CAHW =HWM (H CAHW (Q,V));

[0045] Among them, H SAHW (·) and H CAHW (·) respectively represent the spatial attention mechanism operation and channel attention mechanism operation under the hierarchical window, H WM (·) indicates the window merging operation, Respectively represent the spatial and channel features extracted under the hierarchical window;

[0046] Step 3: F SAHW and F CAHW Perform channel splicing operation, and then perform feature space transformation on the spliced ​​features as the final output of the S-CAHW module. The calculation formula is as follows:

[0047] F S-CAHW =f Linear (f concat (F SAHW ,F CAHW ));

[0048] Among them, f Linear (·) represents a fully connected layer with unchanged feature channel dimension, Represents the output of the S-CAHW module.

[0049] Preferably, the calculation steps of the spatial-gated feedforward network SGFN are as follows:

[0050] Step 1: Define the input features of the SGFN as X in Perform a linear projection operation, the formula is:

[0051]

[0052] in, represents the first linear projection operation, is the output feature map after the linear projection operation, C′ represents the number of output channels after the linear projection operation; f GELU Represents the GELU nonlinear activation function;

[0053] Step 2: Use channel segmentation technology to divide X′ into two parts X′1 and X′2 to reduce the computational complexity in subsequent steps. The formula is:

[0054] X′1,X′2=f split (X′);

[0055] in,

[0056] Step 3: Use the gate mechanism to model the nonlinear spatial characteristics of X′1 and X′2 to reduce channel redundancy. The formula is:

[0057] X′ SG =X′1⊙(f DW-Conv3×3 (X′2));

[0058] Among them, X′ SG represents the output feature of the gate mechanism for X′1 and X′2; ⊙ represents the element-by-element multiplication operation of the feature map; f DW-Conv3×3 (·) represents a depthwise separable convolution operation with a kernel size of 3×3;

[0059] Step 4: X′ SG Perform a secondary linear projection operation to ensure that the number of channels of the output feature is consistent with that of the input feature. The calculation formula is as follows:

[0060]

[0061] in, represents the output of SGFN, Represents the second linear projection operation.

[0062] Preferably, the formula for fusing and upsampling the shallow feature image and the deep feature image is as follows:

[0063] I SR =f subpixel (f Conv3×3 (F0+F DF ));

[0064] Among them, f subpixel (·) represents the sub-pixel convolution operation; F0 is the shallow feature image; F DF is the depth feature image; represents the reconstructed image; s represents the upsampling factor; C is the total number of channels, and H×W is the size of a single feature map.

[0065] The present invention also provides a remote sensing image super-resolution reconstruction system based on residual hierarchical Transformer, comprising:

[0066] Input module, used to obtain low-resolution images;

[0067] A shallow feature extraction module, used to perform shallow feature extraction on the low-resolution image through a convolution operation to obtain a shallow feature image;

[0068] A deep feature extraction module is used to extract deep features from the shallow feature image based on the design concept of nested residual blocks in residual groups, combined with a multi-scale feature calibration mechanism, a hierarchical window-based Transformer space-channel attention mechanism, and a space-gate feedforward network to obtain a deep feature image;

[0069] The reconstruction module is used to fuse and upsample the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

[0070] The present invention also provides a remote sensing image super-resolution reconstruction device based on a residual hierarchical Transformer, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a remote sensing image super-resolution reconstruction method based on a residual hierarchical Transformer as described above.

[0071] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a remote sensing image super-resolution reconstruction method based on a residual hierarchical Transformer as described above is implemented.

[0072] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0073] (1) The system of the present invention is mainly composed of a shallow feature extraction module, a deep feature extraction module and a reconstruction module. The shallow feature extraction module extracts shallow features from the input low-resolution image to provide a basis for subsequent processing; the deep feature extraction module constructs a "Residual-in-Residual" connection structure, that is, the design concept of nesting residual blocks in residual groups to enhance feature reuse at different levels; the reconstruction module uses global residual learning to fuse shallow and deep features to accelerate gradient propagation, and implements pixel-level upsampling in the spatial dimension to generate high-quality reconstructed images.

[0074] (2) The present invention introduces a multi-scale feature calibration mechanism in the residual block, which establishes long-distance dependencies in remote sensing images through multi-scale feature extraction. At the same time, due to the use of channel segmentation and spatial downsampling operations, it does not bring too much computational cost.

[0075] (3) The present invention designs a spatial-channel attention mechanism under a Transformer hierarchical window in the residual block. First, the fixed window commonly used in self-attention calculation is replaced with an expanded hierarchical window to gradually increase the receptive field of the network and extract multi-scale features. Secondly, by using spatial attention and channel attention, the long-distance dependencies of remote sensing images are learned to the greatest extent from both the channel and spatial dimensions to restore the texture details of the reconstructed image. In addition, in order to alleviate the problem of high complexity in large-scale window self-attention calculation, the present invention only uses the query matrix and the value matrix when calculating attention, reducing the information redundancy caused by the key matrix.

[0076] (4) The present invention introduces a spatial-gated feed-forward network in the residual block, which obtains more nonlinear features through a simple gating mechanism while reducing channel redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A schematic diagram of a remote sensing image super-resolution reconstruction method based on a residual hierarchical Transformer provided in Example 1 of the present invention.

[0079] Figure 2 A flowchart of a remote sensing image super-resolution reconstruction method based on a residual hierarchical Transformer provided in Example 1 of the present invention.

[0080] Figure 3 A schematic diagram of the structure of a residual hierarchical Transformer block (RHTB) provided in Embodiment 1 of the present invention.

[0081] Figure 4 This is a schematic diagram of the multi-scale feature calibration (MSFM) structure provided in Example 1 of the present invention.

[0082] Figure 5 Schematic diagram of the overall structure of the spatial-channel attention with hierarchical windows (S-CAHW) provided in Example 1 of the present invention.

[0083] Figure 6 A schematic diagram of the hierarchical window spatial attention (SAHW) structure in the i-th residual hierarchical Transformer block provided in Example 1 of the present invention.

[0084] Figure 7 A schematic diagram of the hierarchical window channel attention (CAHW) structure in the i-th residual hierarchical Transformer block provided in Example 1 of the present invention.

[0085] Figure 8 A schematic diagram of the structure of a spatial-gated feedforward network (SGFN) provided in Example 1 of the present invention.

[0086] Fig. 9 A schematic diagram of a remote sensing image super-resolution reconstruction system based on a residual hierarchical Transformer provided in Example 2 of the present invention.

[0087] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0089] Embodiment 1

[0090] Embodiment 1 of the present invention provides a remote sensing image super-resolution reconstruction method based on a residual hierarchical Transformer, which can be implemented by a remote sensing image super-resolution reconstruction device based on a residual hierarchical Transformer (hereinafter referred to as a reconstruction device), and in particular, is executed by one or more processors in the reconstruction device.

[0091] In this embodiment, the reconstruction device may be an electronic device equipped with a processor, which carries a computer program of the remote sensing image super-resolution reconstruction method based on the residual hierarchical Transformer and can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0092] like Figure 1-Figure 2As shown in Figure 1, the remote sensing image super-resolution reconstruction system based on residual hierarchical Transformer is mainly composed of shallow feature extraction module, deep feature extraction module and reconstruction module. The input image is I LR After passing through the shallow feature extraction module, the shallow feature image F0 is output. After passing through the deep feature extraction module, the shallow feature image F0 is output as the deep feature image F DF , the shallow feature image F0 and the deep feature image F DF Input to the reconstruction module and output the final remote sensing image super-resolution reconstruction result I SR .

[0093] Specifically, a remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer includes steps S1 to S4.

[0094] S1, obtain low-resolution images.

[0095] In this embodiment, let the input low-resolution image be I LR .

[0096] S2, through convolution operation, shallow feature extraction is performed on the low-resolution image to obtain a shallow feature image.

[0097] Specifically, I LR The formula for outputting shallow feature F0 through the shallow feature extraction module is:

[0098] F0=f Conv3×3 (I LR );

[0099] Wherein, F0 is the shallow feature image; f Conv3×3 (·) indicates a convolution operation with a kernel size of 3×3; I LR is the low-resolution image.

[0100] S3, based on the design concept of nested residual blocks in residual groups, combined with multi-scale feature calibration mechanism, hierarchical window-based Transformer space-channel attention mechanism and space-gate feedforward network, deep features are extracted from the shallow feature image to obtain a deep feature image.

[0101] Specifically, the shallow feature F0 is output as a deep feature F after passing through the deep feature extraction module. DF , the calculation formula is as follows:

[0102] F DF =H DF (F0);

[0103] Among them, F DF is the depth feature image; HDF (·) indicates the deep feature extraction module.

[0104] like Figure 2 As shown in the figure, the deep feature extraction module adopts the "Residual-in-Residual" connection structure, that is, the design concept of nesting residual blocks in residual groups to enhance the reuse of features at different levels; the deep feature extraction module consists of N1 RG residual groups (ResidualGroup, RG) and a convolutional layer, and the calculation formula is as follows:

[0105]

[0106] in, and denote the i-1th and i-th RG residual group operations respectively, and F i represents the output of the i-th RG residual group.

[0107] Specifically, each RG residual group consists of N2 residual hierarchical transformer blocks (RHTB) and a convolutional layer, and the calculation formula is as follows:

[0108]

[0109]

[0110] Among them, F i,0 represents the input of the i-th RG residual group, F i,out represents the output of the i-th RG residual group, F i,j The jth RHTB residual block output of the i-th RG residual group is, and They represent the j-1th and jth RHTB residual block operations in the i-th RG residual group, respectively.

[0111] Specifically, Figure 3 As shown in the figure, each RHTB residual block contains two sub-residual modules. The first residual module includes a sequentially connected Layer normalization layer, a multi-scale feature calibration MSFM (Multi-Scale Feature Modulation) mechanism, and a spatial channel attention S-CAHW (Spatial-Channel Attention based on Hierarchical Windows) mechanism; the second sub-residual module includes a sequentially connected Layer normalization layer and a spatial-gate feed-forward network SGFN (Spatial-Gate Feed-forward Network).

[0112] Specifically, Figure 4 As shown, the MSFM uses the fusion of multi-scale features to learn the long-range dependencies between images. The specific calculation steps are as follows:

[0113] Step 1: Definition is the feature map with a total number of channels C, where H×W is the size of a single feature map; the channel segmentation technology is used to divide X into four parts X1, X2, X3, and X4 to reduce the amount of calculation in subsequent steps. The calculation formula is as follows:

[0114] X1,X2,X3,X4=f split (X);

[0115] Among them, f split (·) represents the channel splitting operation, Represents the feature map after X segmentation;

[0116] Step 2: Perform different spatial downsampling and upsampling processing on X1, X2, X3, and X4. Since smaller feature maps have larger receptive fields, after convolution with the same convolution kernel size, the receptive fields of each pixel in feature maps of different scales are different. In order to facilitate the subsequent feature map channel splicing operation, the previously downsampled feature maps are upsampled back in turn through the bilinear interpolation algorithm. The calculation formula is as follows:

[0117]

[0118] Among them, f DW-Conv3×3 (·) represents a depthwise separable convolution operation with a kernel size of 3×3, f DS-2 (·),f DS-4 (·) and f DS-8 (·) respectively represent the downsampling of the input feature size using the maximum pooling operation to and f US-2 (·),f US-4 (·) and f US-8 (·) represents the bilinear interpolation algorithm with upsampling factors of 2, 4, and 8, respectively;

[0119] Step 3: Get the multi-scale features Perform channel splicing operations and use a convolutional layer to perceive local and global correlations. The calculation formula is as follows:

[0120]

[0121] in, is the result after feature fusion, f concat(·) represents the channel concatenation operation, f Conv1×1 (·) indicates a convolution operation with a kernel size of 1×1;

[0122] Step 4: Use GELU nonlinear activation function to estimate The attention feature map distribution is used, and X is adaptively calibrated through the attention feature map to obtain the output result of the final MSFM module. The calculation formula is as follows:

[0123]

[0124] Among them, f GELU (·) represents the GELU nonlinear activation function, ⊙ represents the element-by-element multiplication operation of the feature map, Represents the output of the MSFM module.

[0125] Specifically, Figure 5 As shown in FIG. 1 , the S-CAHW combines the spatial and channel features in the hierarchical window and alleviates the problem of heavy computational burden caused by the fixed large-size window setting in the Transformer mechanism. The specific calculation steps are as follows:

[0126] Step 1: Input features Perform non-overlapping window division to obtain features Where h and w are the height and width of the window respectively, and N = (HW / hw) represents the number of windows. p Divide into two parts to generate the multi-head query matrix Q and value matrix V, the calculation formula is as follows:

[0127] Q,V=f split (F p );

[0128] in,

[0129] Step 2: Input Q and V into the spatial attention SAHW and channel attention CAHW mechanisms under the hierarchical window to extract multi-scale spatial and channel features. The calculation formula is as follows:

[0130] F SAHW =H WM (H SAHW (Q,V));

[0131] D CAHW =H WM (H CAHW (Q,V));

[0132] Among them, H SAHW (·) and H CAHW(·) respectively represent the spatial attention mechanism operation and channel attention mechanism operation under the hierarchical window, H WM (·) indicates the window merging operation, Respectively represent the spatial and channel features extracted under the hierarchical window;

[0133] Step 3: F SAHW and F CAHW Perform channel splicing operation, and then perform feature space transformation on the spliced ​​features as the final output of the S-CAHW module. The calculation formula is as follows:

[0134] F S-CAHW =f Linear (f concat (F SAHW ,F CAHW ));

[0135] Among them, f Linear (·) represents a fully connected layer with unchanged feature channel dimension, Represents the output of the S-CAHW module.

[0136] Specifically, Figure 6 As shown, in another preferred embodiment, the calculation process of the spatial attention SAHW (Spatial Attention based on Hierarchical Windows) mechanism based on hierarchical windows in the i-th RHTB residual block is as follows:

[0137] Define the base height and width of the non-overlapping partition window as h B and w B , the height h of the window in the i-th RHTB residual block i and width i The calculation is as follows:

[0138] h i =α i h B ;

[0139] w i =α i w B ;

[0140] Among them, α i represents the hierarchical window scaling coefficient in the i-th RHTB residual block; for the value matrix in the i-th RHTB residual block Perform linear projection operation on spatial features, and the calculation formula is as follows:

[0141]

[0142] in, represents the linear projection operation of spatial features in the i-th RHTB residual block, and the projection value matrix V ↓,i h in ↓ and w ↓ The definition is as follows:

[0143]

[0144] When α i When ≤1, the network can retain the fine-grained features of small windows; when α i When >1, the network can extract high-level information from a large window;

[0145] The final output of SAHW in the i-th RHTB residual block is calculated as follows:

[0146]

[0147] Among them, B represents the learnable relative position encoding, represents the number of channels of each attention head, where M represents the number of self-attention heads.

[0148] Specifically, Figure 7 As shown, in another preferred embodiment, the final output of the channel attention CAHW (Channel Attention based on Hierarchical Windows) in the i-th RHTB residual block is calculated as follows:

[0149]

[0150] Among them, D i =h i w i .

[0151] Specifically, Figure 8 As shown, the calculation steps of the spatial-gated feedforward network SGFN are as follows:

[0152] Step 1: Define the input features of the SGFN as X in Perform a linear projection operation, the formula is:

[0153]

[0154] in, represents the first linear projection operation, is the output feature map after the linear projection operation, where C′ represents the number of output channels after the linear projection operation;

[0155] Step 2: Use channel segmentation technology to divide X′ into two parts X′1 and X′2. The formula is:

[0156] X′1,X′2=f split (X′);

[0157] in,

[0158] Step 3: Use the gate mechanism to model the nonlinear spatial characteristics of X′1 and X′2. The formula is:

[0159] X′ SG =X′1⊙(f DW-Conv3×3 (X′2));

[0160] Among them, X′ SG represents the output characteristics of the gate mechanism for X′1 and X′2;

[0161] Step 4: X′ SG Perform a secondary linear projection operation to ensure that the number of channels of the output feature is consistent with that of the input feature. The calculation formula is as follows:

[0162]

[0163] in, represents the output of SGFN, Represents the second linear projection operation.

[0164] S4, fusing and upsampling the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

[0165] After the above steps, the shallow feature image and the deep feature image are finally fused and upsampled. The formula is as follows:

[0166] I SR =f subpixel (f Conv3×3 (F0+F DF ));

[0167] Among them, f subpixel (·) represents sub-pixel convolution operation; Represents the obtained reconstructed image, that is, the super-resolution reconstructed image, and s represents the upsampling factor.

[0168] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0169] (1) The system of the present invention is mainly composed of a shallow feature extraction module, a deep feature extraction module and a reconstruction module. The shallow feature extraction module extracts shallow features from the input low-resolution image to provide a basis for subsequent processing; the deep feature extraction module constructs a "Residual-in-Residual" connection structure, that is, the design concept of nesting residual blocks in residual groups to enhance feature reuse at different levels; the reconstruction module uses global residual learning to fuse shallow and deep features to accelerate gradient propagation, and implements pixel-level upsampling in the spatial dimension to generate high-quality reconstructed images.

[0170] (2) The present invention introduces a multi-scale feature calibration mechanism in the residual block, which establishes long-distance dependencies in remote sensing images through multi-scale feature extraction. At the same time, due to the use of channel segmentation and spatial downsampling operations, it does not bring too much computational cost.

[0171] (3) The present invention designs a spatial-channel attention mechanism under a Transformer hierarchical window in the residual block. First, the fixed window commonly used in self-attention calculation is replaced with an expanded hierarchical window to gradually increase the receptive field of the network and extract multi-scale features. Secondly, by using spatial attention and channel attention, the long-distance dependencies of remote sensing images are learned to the greatest extent from both the channel and spatial dimensions to restore the texture details of the reconstructed image. In addition, in order to alleviate the problem of high complexity in large-scale window self-attention calculation, the present invention only uses the query matrix and the value matrix when calculating attention, reducing the information redundancy caused by the key matrix.

[0172] (4) The present invention introduces a spatial-gated feed-forward network in the residual block, which obtains more nonlinear features through a simple gating mechanism while reducing channel redundancy.

[0173] Embodiment 2

[0174] like Fig. 9 As shown, the second embodiment of the present invention further provides a remote sensing image super-resolution reconstruction system based on residual hierarchical Transformer, including:

[0175] Input module, used to obtain low-resolution images;

[0176] A shallow feature extraction module, used to perform shallow feature extraction on the low-resolution image through a convolution operation to obtain a shallow feature image;

[0177] A deep feature extraction module is used to extract deep features from the shallow feature image based on the design concept of nested residual blocks in residual groups, combined with a multi-scale feature calibration mechanism, a hierarchical window-based Transformer space-channel attention mechanism, and a space-gate feedforward network to obtain a deep feature image;

[0178] The reconstruction module is used to fuse and upsample the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

[0179] Embodiment 3

[0180] The third embodiment of the present invention also provides a remote sensing image super-resolution reconstruction device based on a residual hierarchical Transformer, which includes a memory and a processor, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement the remote sensing image super-resolution reconstruction method based on the residual hierarchical Transformer as described above.

[0181] Embodiment 4

[0182] The fourth embodiment of the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the remote sensing image super-resolution reconstruction method based on the residual hierarchical Transformer as described above is implemented.

[0183] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0185] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0186] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0187] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0188] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0189] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer, characterized in that: include: S1, obtain low-resolution images; S2, performing shallow feature extraction on the low-resolution image through a convolution operation to obtain a shallow feature image; S3, based on the design concept of nested residual blocks in residual groups, combined with multi-scale feature calibration mechanism, hierarchical window-based Transformer space-channel attention mechanism and space-gate feedforward network, deep features are extracted from the shallow feature image to obtain a deep feature image; S4, fusing and upsampling the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

2. According to claim 1, a remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer is characterized in that ,The formula for shallow feature extraction is: F0=f Conv3×3 (IN LR ); Wherein, F0 is the shallow feature image; f Conv3×3 (·) indicates a convolution operation with a kernel size of 3×3; I LR is the low-resolution image.

3. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 1 is characterized in that ,When performing deep feature extraction, the Residual-in-Residual connection structure is adopted, that is, the design concept of nesting residual blocks in the residual group, and extraction is performed through N1 RG residual groups and a convolutional layer to enhance the feature reuse at different levels. The calculation formula is as follows: Among them, F i represents the output of the i-th RG residual group, and represent the i-1th and ith RG residual group operations respectively; F0 represents the shallow feature image; F DF Representing the depth feature image; represents the output of the N1th RG residual group operation; f Conv3×3 Indicates a convolution operation with a convolution kernel size of 3×3; Among them, each RG residual group consists of N2 RHTB blocks and a convolutional layer, and the calculation formula is as follows: Among them, F i,j represents the output of the jth RHTB residual block operation of the i-th RG residual group, F i,0 represents the input of the i-th RG residual group, F i,out represents the convolution output of the i-th RG residual group, and They represent the j-1th and jth RHTB residual block operations in the i-th RG residual group respectively; represents the output of the N2th RHTB residual block operation of the i-th RG residual group.

4. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 3 is characterized in that ,The RHTB residual block includes two sub-residual modules, the first residual module includes a sequentially connected Layer normalization layer, a multi-scale feature calibration MSFM mechanism and a hierarchical window-based spatial channel attention S-CAHW mechanism; the second sub-residual module includes a sequentially connected Layer normalization layer and a spatial-gated feedforward network SGFN; Among them, the multi-scale feature calibration MSFM mechanism uses the fusion of multi-scale features to learn the long-range dependency between images. The fusion learning process is specifically as follows: the input feature map is divided into several parts by channel segmentation to obtain feature sub-maps; each feature sub-map is then down-sampled and up-sampled in different spaces and then channel-joined, and then a convolution layer is used to convolve the result of channel jointing, and the distribution of the estimated attention feature map is used to calibrate the feature map, thereby obtaining the output feature map of MSFM; The spatial-channel attention S-CAHW mechanism uses the output feature map of the MSFM as the input feature, generates a multi-head query matrix and a value matrix after non-overlapping window division and channel segmentation, and then fuses the spatial features and channel features in the hierarchical window to obtain the output feature F S-CAHW , to alleviate the problem of heavy computational burden caused by fixed large-size window setting in Transformer mechanism; The spatial-gated feed-forward network SGFN is based on the output feature F of S-CAHW. S-CAHW The features fused with the input of the current RHTB block are then used as input after the output of the Layer normalization layer in the second sub-residual module. After the first linear projection, they are split into two parts. Then, the gate mechanism is used to model the nonlinear spatial features, and then a second linear projection is performed to obtain the output features of SGFN, which are used to supplement the nonlinear features.

5. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 4 is characterized in that ,The calculation steps of the multi-scale feature calibration MSFM mechanism are as follows: Step 1: Definition is the feature map with a total number of channels C, where H×W is the size of a single feature map; the channel segmentation technology is used to divide X into four parts X1, X2, X3, and X4 to reduce the amount of calculation in subsequent steps. The calculation formula is as follows: X1,X2,X3,X4=f split (X); Among them, f split (·) represents the channel splitting operation, Step 2: Perform different spatial downsampling and upsampling processing on X1, X2, X3, and X4; in order to facilitate the channel splicing operation of the subsequent feature map, the feature map that has undergone the downsampling operation is sequentially upsampled through the bilinear interpolation algorithm. The calculation formula is as follows: Among them, f DW-Conv3×3 (·) represents a depthwise separable convolution operation with a kernel size of 3×3; f DS-2 (·),f DS-4 (·) and f DS-8 (·) respectively represent the downsampling of the input feature size using the maximum pooling operation to and f US-2 (·),f US-4 (·) and f US-8 (·) represents the bilinear interpolation algorithm with upsampling factors of 2, 4, and 8, respectively; They represent the processing results of spatial downsampling and upsampling at different scales respectively; Step 3: Get the multi-scale features Perform channel splicing operations and use a convolutional layer to perceive local and global correlations. The calculation formula is as follows: in, is the result after feature fusion, f concat (·) represents the channel concatenation operation, f Conv1×1 (·) indicates a convolution operation with a kernel size of 1×1; Step 4: Use GELU nonlinear activation function to estimate The attention feature map distribution is used, and X is adaptively calibrated through the attention feature map to obtain the output result of the final MSFM module. The calculation formula is as follows: Among them, f GELU (·) represents the GELU nonlinear activation function, ⊙ represents the element-by-element multiplication operation of the feature map, Represents the output of the MSFM module.

6. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 4 is characterized in that ,The calculation steps of the spatial-channel attention S-CAHW mechanism are as follows: Step 1: Input features Divide the window without overlapping to get the features Among them, h and w are the height and width of the window respectively, C is the total number of channels, and N = HW / hw, which represents the number of windows. p Divide into two parts to generate the query matrix and value matrix of multiple heads. The calculation formula is as follows: Q,V=f split (F p ); in, represent the query matrix and value matrix respectively; f split (·) indicates channel splitting operation; Step 2: Input Q and V into the spatial attention SAHW and channel attention CAHW mechanisms under the hierarchical window to extract multi-scale spatial and channel features. The calculation formula is as follows: F SAHW =H WH (H SAHW (Q,V)); F CAHW =H WM (H CAHW (Q,V)); Among them, H SAHW (·) and H CAHW (·) respectively represent the spatial attention mechanism operation and channel attention mechanism operation under the hierarchical window, H WM (·) indicates the window merging operation, Respectively represent the spatial and channel features extracted under the hierarchical window; Step 3: F SAHW and F CAHW Perform channel splicing operation, and then perform feature space transformation on the spliced ​​features as the final output of the S-CAHW module. The calculation formula is as follows: F S-CAHW =f Linear (f concat (F SAHW ,F CAHW )); Among them, f Linear (·) represents a fully connected layer with unchanged feature channel dimension, Represents the output of the S-CAHW module.

7. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 4 is characterized in that ,The calculation steps of the space-gated feedforward network SGFN are as follows: Step 1: Define the input features of the SGFN as X in Perform a linear projection operation, the formula is: in, represents the first linear projection operation, is the output feature map after the linear projection operation, C′ represents the number of output channels after the linear projection operation; f GELU Represents the GELU nonlinear activation function; Step 2: Use channel segmentation technology to divide X′ into two parts X′1 and X′2 to reduce the computational complexity in subsequent steps. The formula is: X′1,X′2=f split (X′); in, Step 3: Use the gate mechanism to model the nonlinear spatial characteristics of X′1 and X′2 to reduce channel redundancy. The formula is: in, represents the output feature of the gate mechanism for X′1 and X′2; ⊙ represents the element-by-element multiplication operation of the feature map; f DW-Conv3×3 (·) represents a depthwise separable convolution operation with a kernel size of 3×3; Step 4: X′ SG Perform a secondary linear projection operation to ensure that the number of channels of the output feature is consistent with that of the input feature. The calculation formula is as follows: in, represents the output of SGFN, Represents the second linear projection operation.

8. The remote sensing image super-resolution reconstruction method based on residual hierarchical Transformer according to claim 1 is characterized in that ,The formula for fusing and upsampling the shallow feature image and the deep feature image is as follows: I SR =f subpixel (f Conv3×3 (F0+F DF )); Among them, f subpixel (·) represents the sub-pixel convolution operation; F0 is the shallow feature image; F DF is the depth feature image; represents the reconstructed image; s represents the upsampling factor; C is the total number of channels, and H×W is the size of a single feature map.

9. A remote sensing image super-resolution reconstruction system based on residual hierarchical Transformer, characterized in that: include: Input module, used to obtain low-resolution images; A shallow feature extraction module, used to perform shallow feature extraction on the low-resolution image through a convolution operation to obtain a shallow feature image; A deep feature extraction module is used to extract deep features from the shallow feature image based on the design concept of nested residual blocks in residual groups, combined with a multi-scale feature calibration mechanism, a hierarchical window-based Transformer space-channel attention mechanism, and a space-gate feedforward network to obtain a deep feature image; The reconstruction module is used to fuse and upsample the shallow feature image and the deep feature image through convolution and sub-pixel convolution operations to obtain a final remote sensing image super-resolution reconstruction result.

Images