A remote sensing image super-resolution reconstruction method based on multi-stage space-frequency joint

Abstract

The present invention provides a remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination, which uses a dual-branch space-frequency information extraction network to simultaneously obtain the global spectrum features of the remote sensing image in the frequency domain and the spatial detail features in the spatial domain; uses an adaptive space-frequency selection attention network to adaptively select and fuse the space-frequency two-branch features; connects the above two networks in series and regards them as the basic feature extraction components in the multi-stage U-type encoding-decoding network architecture, thereby forming the final multi-stage space-frequency combination remote sensing super-resolution network, which can effectively mine the multi-level space-frequency combination features in the remote sensing image and improve the accuracy of the remote sensing image super-resolution reconstruction task. The present invention makes up for the limitation that the existing remote sensing super-resolution network only focuses on the extraction of spatial domain features, has the ability to model the global correlation within the image from the frequency domain perspective, enhances the information expression of the fused features for the remote sensing image, and effectively improves the reconstruction accuracy of the remote sensing image super-resolution task.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically to a remote sensing image super-resolution reconstruction method. Background Art

[0002] High-resolution remote sensing images play an important role in resource exploration, environmental monitoring, target detection, target recognition, etc. However, due to the limitations of imaging equipment and poor imaging conditions in space, the quality of remote sensing images is often affected by factors such as motion blur and atmosphere. Improving its resolution from the hardware is time-consuming and costly. Remote Sensing Image Super-Resolution (RSISR) technology can obtain higher-resolution images above the resolution limit of the imaging system through image processing software algorithms without changing the limitations of physical imaging equipment. It has received widespread attention in remote sensing applications.

[0003] In recent years, deep convolutional neural network methods based on spatial domain have made great breakthroughs in RSISR tasks. However, due to the complexity and diversity of texture and structural information inside remote sensing images, feature mining of remote sensing images only from the perspective of spatial domain convolution cannot fully perceive the global degradation phenomenon inside remote sensing images, resulting in poor network reconstruction and insufficient texture details. Subsequently, a large number of scholars began to introduce the Tansformer self-attention mechanism to capture the global self-similarity of remote sensing image features, which alleviated the above problems to a certain extent and improved the performance of RSISR tasks. However, this type of method only focuses on capturing the self-similarity of image features, and does not model the global degradation correlation of images and the coherence of texture contours, resulting in the network being limited to the mining of feature self-similarity and ignoring the mining of feature global correlation, which has limited performance improvement on super-resolution tasks. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination. In order to alleviate the problems of inadequate global feature capture, insufficient reconstruction accuracy, and unclear texture of existing remote sensing image super-resolution technology, the present invention proposes a remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination. A dual-branch space-frequency information extraction network is used to simultaneously obtain the global spectral features of the remote sensing image in the frequency domain and the spatial detail features in the spatial domain; an adaptive space-frequency selection attention network is used to adaptively select and fuse the space-frequency two-branch features; the above two networks are connected in series and regarded as the basic component of feature extraction in the multi-stage U-type encoding-decoding network architecture, thereby forming the final multi-stage space-frequency combination remote sensing super-resolution network, which can effectively mine the multi-level space-frequency combination features in the remote sensing image and improve the accuracy of the remote sensing image super-resolution reconstruction task.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination, the steps are as follows:

[0007] Step 1: Image preprocessing: Use bilinear interpolation to upsample the low-resolution remote sensing image to the required super-resolution image output size to reduce excessive information loss;

[0008] Step 2: Construct a multi-stage U-type encoding-decoding network architecture, input the preprocessed upsampled image into the U-type encoding-decoding network architecture for feature extraction, and obtain the final super-resolution reconstruction result;

[0009] The U-shaped encoding-decoding network architecture uses four encoding networks and three down-sampling networks to extract four different scales of encoding feature maps {E1, E2, E3, E4}, and uses four decoding networks and three up-sampling networks to take the minimum scale encoding feature E4 as input, and gradually decode and restore it into the corresponding scale decoding feature map {D1, D2, D3, D4}. In this process, the decoding features of each level are directly connected and summed with the corresponding encoding features at the same scale as the input of the next level decoding network; the encoding network and the decoding network both use the same dual-branch space-frequency information extraction network design, and the D4 feature map output by the last layer of the decoding network is adjusted through a 3×3 convolution layer The output channel dimension is added to the bilinear interpolation up-sampled image to obtain the final super-resolution reconstruction result; the dual-branch space-frequency information extraction network includes three parts: a space-domain information extraction network, a frequency-domain information extraction network, and an adaptive space-frequency selection attention network;

[0010] Step 3: For the dual-branch space-frequency information extraction network, given the initial feature input X, X is sent to the spatial domain information extraction network branch in the dual-branch space-frequency information extraction network to obtain the corresponding spatial domain feature output S under the current feature scale;

[0011] The spatial information extraction network branch is composed of a residual unit and a spatial enhanced self-attention unit connected in series; the residual unit uses a 3×3 convolution operation and a jump connection to capture the subtle structural texture in the local area of ​​the feature map from the initial feature input X as an implicit spatial detail feature K; the spatial enhanced self-attention unit generates a coarse spatial self-attention map M from the spatial detail feature K, and obtains a spatial feature output S=K+K·M after selectively enhancing important features in different local areas by multiplying the spatial detail feature K and the spatial self-attention map M and then adding them;

[0012] Step 4: For the dual-branch space-frequency information extraction network, given the initial feature input X, X is sent to the frequency domain information extraction network branch in the dual-branch space-frequency information extraction network to obtain the corresponding frequency domain feature output J under the current feature scale;

[0013] The frequency domain information extraction network branch is composed of a channel self-attention module and a frequency domain convolution module connected in series; the channel self-attention module aggregates the channel information of the initial feature input X, learns the dependency weight map Q between the channels, and performs weighted summation on the feature map of each channel according to Q, so as to adjust the importance of each channel and obtain the feature output X′ after channel aggregation; then the feature output X′ is Fourier transformed to obtain the amplitude feature A and phase feature P corresponding to X′ in the frequency domain, and the frequency domain convolution module is used to perform small kernel convolution on the amplitude feature representing texture information and the phase feature representing contour information, respectively, to calculate the new frequency domain amplitude feature A′ and phase feature P′, and the amplitude feature A′ and phase feature P′ in the frequency domain are converted back to the spatial domain by inverse Fourier transform, and after obtaining the feature O, O is directly connected and added to the feature X′ through a direct residual structure, and the information is further aggregated to output the frequency domain feature output J corresponding to the current feature scale;

[0014] Step 5: For the i-th layer dual-branch space-frequency information extraction network, given the spatial feature output S in step 3 and the frequency feature output J in step 4, S and J are sent to the adaptive space-frequency selection attention network branch in the dual-branch space-frequency information extraction network, and the spatial feature attention weights and frequency feature attention weights required to be retained for each spatial pixel position are adaptively generated. The generated space-frequency two-branch attention weights are weighted summed with the corresponding space-frequency features to obtain the space-frequency fusion feature output R at the current feature scale;

[0015] If the adaptive space-frequency selection attention network branch is in the encoding stage, the feature output R is sent to the downsampling network as the input of the dual-branch space-frequency information extraction network of the next feature scale; if the dual-branch space-frequency information extraction network of this layer is in the decoding stage, the feature R is sent to the upsampling network and added to the encoding feature of the corresponding scale as the input of the dual-branch space-frequency information extraction network of the next feature scale;

[0016] Step 6: Construct and obtain a high-resolution remote sensing image dataset. The high-resolution remote sensing image dataset is divided into three parts: training, verification, and testing. The corresponding high-resolution image and low-resolution image pairs are obtained through bicubic interpolation. HR / I LR ; Construct the super-resolution image reconstruction loss function L, and the calculation formula is: Where N represents the number of samples, represents the true high-resolution image label of the jth sample, The low-resolution image of the jth sample is input into the super-resolution image output by the U-type encoding-decoding network mentioned in step 2;

[0017] After each round of training, the validation data set is used to calculate the reconstruction loss function. The smaller the function value, the better the super-resolution reconstruction performance of the trained network model on the validation data set. Based on this, the model weights corresponding to the smallest reconstruction loss function value in 600 rounds of training are selected and saved to complete the model training. During the test, the selected model weights are loaded, and the low-resolution image is input into the U-type encoding-decoding network mentioned in step 2 to obtain the corresponding super-resolution image output, thus completing the super-resolution reconstruction task of the remote sensing image.

[0018] In step 4, for the frequency domain information extraction network in the dual-branch space-frequency information extraction network, given the original input feature vector First, the channel attention mechanism is used to aggregate the channel information of the input features, and the 3×3 convolutional layer is used to map the features to obtain a new feature vector The global average pooling operation is used to compress the global spatial information of each feature channel, and then two 1×1 convolutional layers are used to learn the dependencies between channels, and the importance weights of each channel dimension are output through the Sigmoid layer. Multiply the feature vector Y to get the adaptive channel weighted feature information and add it to the original input feature vector X, and output the final adaptive channel excitation feature information X′=X+Y·Q;

[0019] The feature vector after excitation Use Fourier transform F(·) to transform to the frequency domain, as shown in formula (2):

[0020]

[0021] The amplitude A(X′)(u,v) representing the texture information and the phase P(X′)(u,v) representing the contour information are calculated respectively, as shown in formulas (3) and (4):

[0022]

[0023]

[0024] Secondly, two small kernel residual structures consisting of 3×3 convolution-Leaky-Relu activation-3×3 convolution are used to abstract the texture features and contour features of the frequency domain amplitude information and phase information, respectively, to obtain new amplitude features A′(X′)(u,v) and phase features P′(X′)(u,v); finally, the spatial domain O=F is transformed back through the inverse Fourier transform. -1(A′(X′)(u,v),P′(X′)(u,v)), after 3×3 convolution, is directly added to the input feature X′ to further aggregate information and output the final frequency domain branch feature J=Conv 3×3 (O)+X′.

[0025] In step 5, two 1×1 convolutions are first used to respectively transform the spatial branch features Frequency domain branch characteristics After feature mapping, concatenate to get feature vector As shown in formulas (5) and (6):

[0026]

[0027]

[0028] The concatenated features are averagely pooled along the channel dimension P avg (·) and the maximum pooling P max (·), as shown in formula (7):

[0029] SA avg =P avg (U),SA max =P max (U) (7)

[0030] The result of pooling the two items The pooled information is further summarized through a 7×7 large kernel convolution and a Sigmoid activation function σ(·), as shown in formula (8):

[0031] SA=σ(Conv 7×7 ([SA avg ,SA max ])) (8)

[0032] Output the space-frequency two-branch attention weights that need to be retained at each spatial pixel position, and perform weighted summation of the solved weight coefficients and the input features, as shown in formula (9):

[0033]

[0034] Get the final space-frequency joint output result

[0035] In step 6, the number of training rounds is set to 600, the initial learning rate is set to 0.0001, and the learning rate is decayed at a rate of 0.5 in the 300th single cycle.

[0036] In step 6, the batch size is set to 4.

[0037] In step 6, the Adam optimizer is used to update the network parameters.

[0038] The beneficial effect of the present invention is that the dual-branch space-frequency information extraction module makes up for the limitation of the existing remote sensing super-resolution network that only focuses on spatial domain feature extraction, further expands the network's ability to extract and learn frequency domain features, and has the ability to model the global correlation within the image from a frequency domain perspective;

[0039] The present invention realizes the adaptive complementary fusion of frequency domain information and spatial domain information through the design of an adaptive space-frequency selection attention module, enhances the information expression of the fusion features for remote sensing images, and effectively improves the reconstruction accuracy of remote sensing image super-resolution tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the present invention.

[0041] Figure 2 Schematic diagram of frequency domain feature extraction module.

[0042] Figure 3 Schematic diagram of the spatial feature extraction module.

[0043] Figure 4 Schematic diagram of the space-frequency joint attention module.

[0044] Figure 5 This is a design diagram of the multi-stage space-frequency joint network architecture. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0046] The present invention provides a remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination, such as Figure 1 As shown, the specific steps of the method include:

[0047] Step 1, obtain a high-resolution remote sensing image dataset: the training dataset is 945 high-quality non-degraded remote sensing images in the UCMerced dataset, the validation dataset is 105 high-quality non-degraded remote sensing images in the UCMerced dataset, and the test dataset is 1050 high-quality non-degraded remote sensing images in the UCMerced dataset.

[0048] Step 2: Take the original image in the dataset in step 1 as the real high-resolution label image I HR , the corresponding low-resolution image I is obtained by bicubic interpolation after being downsampled 8 times LR , forming the high-resolution / low-resolution image pairs required for training, validation, and testing.

[0049] Step 3, construct a U-shaped multi-stage encoding-decoding network architecture. The low-resolution LR image is input into the U-shaped network for feature extraction to obtain feature maps of different sizes; the U-shaped encoding-decoding network architecture uses four encoding networks and three downsampling networks to extract four different sizes of encoding feature maps {E1, E2, E3, E4}, and uses four decoding networks and three upsampling networks to gradually decode and restore the smallest scale encoding feature E4 as input to the corresponding scale decoding feature map {D1, D2, D3, D4}. In this process, the decoding features of each level are directly connected and summed with the corresponding encoding features as the input of the next decoding network; the encoding network and the decoding network both use the same dual-branch space-frequency information extraction network design, and the D4 feature map output by the last layer of the decoding network is adjusted through a 3×3 convolution layer The output channel dimension is added to the bilinear interpolation upsampled image to obtain the final super-resolution reconstruction result; the dual-branch space-frequency information extraction network consists of a spatial information extraction network, a frequency information extraction network, and an adaptive space-frequency selection attention network.

[0050] As an optional embodiment, the implementation process for constructing a multi-stage U-type encoding-decoding network is as follows: Figure 5 As shown. Low resolution image I LR First, bilinear interpolation is used to obtain the desired super-resolution image output size. Then a 3×3 convolution is used to Convert from RGB pixel space to low-level feature embedding This patent sets the value of C to 64. Next, the shallow feature X0 is sent to a four-level encoder Enc(·) composed of a dual-branch space-frequency information extraction network and three downsampling networks Down(·), and the feature is converted into multi-scale features {E1, E2, E3, E4} of different resolutions, as shown in formula (10). Next, E4 is sent to a four-level decoder Dec(·) also composed of a dual-branch space-frequency information extraction network and three upsampling networks Up(·), and the output of each level of the decoder is added to the corresponding encoder output of the same resolution as the input of the next level of the decoder, gradually restoring and enriching the feature representation of the remote sensing image {D1, D2, D3, D4}, as shown in formula (11). The feature G7 of the final decoded output is sent to the 3×3 convolution Conv 3×3 (·) is directly connected with the interpolated image to obtain the final network super-resolution reconstructed image output

[0051]

[0052] Where E i Represents the encoding feature output of the i-th level, Enc i(·) represents the encoding network of the i-th level, which is composed of a dual-branch space-frequency information extraction network. Down(·) represents the downsampling network, which is composed of a 3×3 convolution, invPixelShuffle, and 1×1 convolution in series.

[0053]

[0054] Where D i Represents the decoded feature output of the i-th level, Dec i (·) represents the decoding network of the i-th level, which is composed of a dual-branch space-frequency information extraction network. Up(·) represents the upsampling network, which is composed of a series of 1×1 convolution, pixel reorganization (PixelShuffle), and PReLU activation function.

[0055] As an optional embodiment, the implementation process of the spatial domain information extraction network in the dual-branch spatial frequency information extraction network is as follows: Figure 2 As shown in the residual module section, given the input feature vector A small residual module consisting of two 3×3 convolutions-Relu activation-3×3 convolutions and an outer 3×3 convolution direct connection is selected as the basic spatial feature extraction module to extract effective spatial detail features. Secondly, the enhanced spatial attention unit is used to generate a rough spatial self-attention map M from the feature K, and the important features in different local areas are selected for enhancement by aggregating the context information at the spatial and channel levels, such as Figure 2 As shown in the spatial enhancement module on the right, first use 1×1 convolution to compress the feature channel C to C / / 4, then use a 3×3 convolution layer with a stride of 2 and a 7×7 maximum pooling layer to expand the spatial receptive field and compress the spatial information, then use a bilinear interpolation upsampling layer to restore the compressed features to the original spatial dimension, use 1×1 convolution to restore the channel layer from C / / 4 to C, and finally use the Sigmoid activation function to calculate the spatial attention weight. The weighted summation with the input features is used to obtain the final spatially enhanced feature output S = K + K·M.

[0056] As an optional embodiment, the implementation process of the frequency domain information extraction network in the dual-branch space-frequency information extraction network is as follows: Figure 3 As shown in (above), given the original input feature vector First, the channel attention mechanism is used to aggregate the channel information of the input features, and the 3×3 convolutional layer is used to map the features to obtain a new feature vector The global average pooling operation is used to compress the global spatial information of each feature channel, and then two 1×1 convolutional layers (equivalent to fully connected layers) are used to learn the dependencies between channels, and the importance weights of each channel dimension are output through the Sigmoid layer. The adaptive channel weighted feature information is obtained by multiplying the feature vector Y and then added to the original input feature vector X to output the final adaptive channel excitation feature information X′=X+Y·Q.

[0057] Then Figure 3 As shown in (below), the eigenvector after excitation The Fourier transform F(·) is used to transform to the frequency domain, as shown in formula (2), and the amplitude A(X′)(u,v) representing the texture information and the phase P(X′)(u,v) representing the contour information are calculated respectively, as shown in formulas (3) and (4); secondly, two small kernel residual structures consisting of 3×3 convolution-Leaky-Relu activation-3×3 convolution are used to abstract the texture features and contour features of the frequency domain amplitude information and phase information respectively, and the new amplitude features A′(X′)(u,v) and phase features P′(X′)(u,v) are obtained; finally, the spatial domain O=F is returned by inverse Fourier transform. -1 (A′(X′)(u,v),P′(X′)(u,v)), after 3×3 convolution, is directly added to the input feature X′ to further aggregate information and output the final frequency domain branch feature J=Conv 3×3 (O)+X′.

[0058]

[0059]

[0060]

[0061] As an optional embodiment, the construction and implementation process of the adaptive space-frequency selection attention network in the dual-branch space-frequency information extraction network is as follows: Figure 4 First, two 1×1 convolutions are used to respectively transform the spatial branch features. Frequency domain branch characteristics After feature mapping, concatenate to get feature vector As shown in formulas (5) and (6). The concatenated features are average pooled along the channel dimension P avg (·) and the maximum pooling P max (·), as shown in formula (7). The result of pooling the two items The pooled information is further summarized through a 7×7 large kernel convolution and a Sigmoid activation function σ(·), as shown in formula (8). The spatial and frequency branch attention weights required to be retained at each spatial pixel position are output, and the weight coefficients solved are weighted and summed with the input features to obtain the final spatial and frequency joint output result. As shown in formula (9).

[0062]

[0063]

[0064] SA avg =P avg (U),SA max =P max (U) (7)

[0065] SA=σ(Conv 7×7 ([SA avg ,SA max ])) (8)

[0066]

[0067] Step 4: After the network is built in step 3, train the remote sensing image super-resolution model based on multi-stage space-frequency joint: use the L1 loss function between the super-reconstructed image and the true value image to optimize the entire network, set the hyperparameters: 600 training rounds, the initial learning rate is 0.0001, and the learning rate is decayed by 0.5 in the 300th cycle, and the batch size is 4. Use the Adam optimizer to update the network parameters so that the loss function converges to the minimum and complete the model training.

[0068] Step 5: Use the validation data set in step 2 to select the model weight with the best super-resolution performance in step 4 and save it for later reconstruction of high-resolution remote sensing images.

[0069] The effects of the present invention are further described below in conjunction with simulation experiments.

[0070] 1. Simulation experiment conditions:

[0071] The hardware platform of the simulation experiment of the present invention includes: a processor of Intel(R) Core(TM) i9-9820X ​​CPU, a main frequency of 3.0 GHz, a memory of 32.0 GB, and a graphics card of 1080Ti.

[0072] Software platform: Win10 operating system, Pytorch 1.13.0.

[0073] The images of the test sample set used in the simulation experiment of the present invention are 1050 remote sensing images in the UCMerced dataset, which contain 21 categories of scene images, with image sizes of 256×256 pixels and image format of .tif. The 1050 images are downsampled by bicubic interpolation to obtain the test sample set.

[0074] 2. Simulation content and results analysis:

[0075] The simulation experiment of the present invention adopts the networks trained by the present invention and three prior arts (MHAN method, HSENet method, TransENet method), respectively performs super-resolution reconstruction on the input test sample sets, and obtains super-resolution result images.

[0076] In order to verify the simulation experiment effect of the present invention, the images reconstructed from the test sample set by the present invention and the three prior arts are respectively evaluated by using the Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measurement (SSIM). The quality of the super-resolution simulation results of the simulation experiment of the present invention is evaluated. The higher the two evaluation indicators are, the better the reconstructed image effect is.

[0077] The present invention and three existing advanced methods are used in the simulation experiment:

[0078] The MHAN method in the prior art refers to the enhanced mixed high-order attention network super-resolution reconstruction method for test samples adopted in Zhang et al. (D. Zhang, J. Shao, X. Li, and H. T. Shen, “Remote sensing image super-resolution via mixed high-order attention network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 6, pp. 5183–5196, 2020).

[0079] The HSENet method of the prior art refers to the hybrid multi-scale self-similarity mining network super-resolution reconstruction method adopted in “S.Lei and Z.Shi, “Hybrid-scale self-similarity exploitation for remote sensing image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–10, 2021.) published by Lei et al.

[0080] The TransENet method of the prior art refers to the multi-stage super-resolution reconstruction method based on the Transformer self-similarity mining network adopted in Lei et al. (S.Lei, Z.Shi, and W.Mo, "Transformer-based multistage enhancement for remote sensing image super-resolution," IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp.1–11, 2021.).

[0081] The comparison results of the average values ​​of all objective evaluation indicators in the test sample set 1 reconstructed by the present invention and the above three existing methods are shown in Table 1:

[0082] Table 1 Objective evaluation value table of images reconstructed by the present invention and the comparative method

[0083]

[0084] It can be seen from Table 1 that the average values ​​of the two evaluation indicators after the test sample WHU-RS19 is reconstructed by the present invention are higher than those of the three methods in the above-mentioned prior art, indicating that the image reconstructed by the present invention is closer to the original high-resolution image.

Claims

1. A remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination, characterized in that The steps include: Step 1: Image preprocessing: Use bilinear interpolation to upsample the low-resolution remote sensing image to the required super-resolution image output size to reduce excessive information loss; Step 2: Construct a multi-stage U-type encoding-decoding network architecture, input the preprocessed upsampled image into the U-type encoding-decoding network architecture for feature extraction, and obtain the final super-resolution reconstruction result; The U-shaped encoding-decoding network architecture uses four encoding networks and three down-sampling networks to extract four different scales of encoding feature maps {E1, E2, E3, E4}, and uses four decoding networks and three up-sampling networks to take the minimum scale encoding feature E4 as input, and gradually decode and restore it into the corresponding scale decoding feature map {D1, D2, D3, D4}. In this process, the decoding features of each level are directly connected and summed with the corresponding encoding features at the same scale as the input of the next level decoding network; the encoding network and the decoding network both use the same dual-branch space-frequency information extraction network design, and the D4 feature map output by the last layer of the decoding network is adjusted through a 3×3 convolution layer The output channel dimension is added to the bilinear interpolation up-sampled image to obtain the final super-resolution reconstruction result; the dual-branch space-frequency information extraction network includes three parts: a space-domain information extraction network, a frequency-domain information extraction network, and an adaptive space-frequency selection attention network; Step 3: For the dual-branch space-frequency information extraction network, given the initial feature input X, X is sent to the spatial domain information extraction network branch in the dual-branch space-frequency information extraction network to obtain the corresponding spatial domain feature output S under the current feature scale; The spatial information extraction network branch is composed of a residual unit and a spatial enhanced self-attention unit connected in series; the residual unit uses a 3×3 convolution operation and a jump connection to capture the subtle structural texture in the local area of ​​the feature map from the initial feature input X as an implicit spatial detail feature K; the spatial enhanced self-attention unit generates a coarse spatial self-attention map M from the spatial detail feature K, and obtains a spatial feature output S=K+K·M after selectively enhancing important features in different local areas by multiplying the spatial detail feature K and the spatial self-attention map M and then adding them; Step 4: For the dual-branch space-frequency information extraction network, given the initial feature input X, X is sent to the frequency domain information extraction network branch in the dual-branch space-frequency information extraction network to obtain the corresponding frequency domain feature output J under the current feature scale; The frequency domain information extraction network branch is composed of a channel self-attention module and a frequency domain convolution module connected in series; the channel self-attention module aggregates the channel information of the initial feature input X, learns the dependency weight map Q between the channels, and performs weighted summation on the feature map of each channel according to Q, so as to adjust the importance of each channel and obtain the feature output X′ after channel aggregation; then the feature output X′ is Fourier transformed to obtain the amplitude feature A and phase feature P corresponding to X′ in the frequency domain, and the frequency domain convolution module is used to perform small kernel convolution on the amplitude feature representing texture information and the phase feature representing contour information, respectively, to calculate the new frequency domain amplitude feature A′ and phase feature P′, and the amplitude feature A′ and phase feature P′ in the frequency domain are converted back to the spatial domain by inverse Fourier transform, and after obtaining the feature O, O is directly connected and added to the feature X′ through a direct residual structure, and the information is further aggregated to output the frequency domain feature output J corresponding to the current feature scale; Step 5: For the i-th layer dual-branch space-frequency information extraction network, given the spatial feature output S in step 3 and the frequency feature output J in step 4, S and J are sent to the adaptive space-frequency selection attention network branch in the dual-branch space-frequency information extraction network, and the spatial feature attention weights and frequency feature attention weights required to be retained for each spatial pixel position are adaptively generated. The generated space-frequency two-branch attention weights are weighted summed with the corresponding space-frequency features to obtain the space-frequency fusion feature output R at the current feature scale; If the adaptive space-frequency selection attention network branch is in the encoding stage, the feature output R is sent to the downsampling network as the input of the dual-branch space-frequency information extraction network of the next feature scale; if the dual-branch space-frequency information extraction network of this layer is in the decoding stage, the feature R is sent to the upsampling network and added to the encoding feature of the corresponding scale as the input of the dual-branch space-frequency information extraction network of the next feature scale; Step 6: Construct and obtain a high-resolution remote sensing image dataset. The high-resolution remote sensing image dataset is divided into three parts: training, verification, and testing. The corresponding high-resolution image and low-resolution image pairs are obtained through bicubic interpolation. HR / I LR ; Construct the super-resolution image reconstruction loss function L, and the calculation formula is: Where N represents the number of samples, represents the true high-resolution image label of the jth sample, The low-resolution image of the jth sample is input into the super-resolution image output by the U-type encoding-decoding network mentioned in step 2; After each round of training, the validation data set is used to calculate the reconstruction loss function. The smaller the function value, the better the super-resolution reconstruction performance of the trained network model on the validation data set. Based on this, the model weight corresponding to the minimum reconstruction loss function value in 600 rounds of training is selected and saved to complete the model training; During the test, the selected model weights are loaded, and the low-resolution image is input into the U-type encoding-decoding network mentioned in step 2 to obtain the corresponding super-resolution image output, thus completing the super-resolution reconstruction task of the remote sensing image.

2. The remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination according to claim 1 is characterized in that: In step 4, for the frequency domain information extraction network in the dual-branch space-frequency information extraction network, given the original input feature vector First, the channel attention mechanism is used to aggregate the channel information of the input features, and the 3×3 convolutional layer is used to map the features to obtain a new feature vector The global average pooling operation is used to compress the global spatial information of each feature channel, and then two 1×1 convolutional layers are used to learn the dependencies between channels, and the importance weights of each channel dimension are output through the Sigmoid layer. Multiply the feature vector Y to get the adaptive channel weighted feature information and add it to the original input feature vector X, and output the final adaptive channel excitation feature information X′=X+Y·Q; The feature vector after excitation Use Fourier transform F(·) to transform to the frequency domain, as shown in formula (2): The amplitude A(X′)(u,v) representing the texture information and the phase P(X′)(u,v) representing the contour information are calculated respectively, as shown in formulas (3) and (4): Secondly, two small kernel residual structures consisting of 3×3 convolution-Leaky-Relu activation-3×3 convolution are used to abstract the texture features and contour features of the frequency domain amplitude information and phase information, respectively, to obtain new amplitude features A′(X′)(u,v) and phase features P′(X′)(u,v); finally, the spatial domain O=F is transformed back through the inverse Fourier transform. -1 (A′(X′)(u,v),P′(X′)(u,v)), after 3×3 convolution, is directly added to the input feature X′ to further aggregate information and output the final frequency domain branch feature J=Conv 3×3 (O)+X′.

3. The remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination according to claim 1 is characterized in that: In step 5, two 1×1 convolutions are first used to respectively transform the spatial branch features Frequency domain branch characteristics After feature mapping, concatenate to get feature vector As shown in formulas (5) and (6): The concatenated features are averagely pooled along the channel dimension P avg (·) and the maximum pooling P max (·), as shown in formula (7): SA avg =P avg (U), SA max =P max (U) (7) The result after pooling the two items The pooled information is further summarized through a 7×7 large kernel convolution and a Sigmoid activation function σ(·), as shown in formula (8): SA=σ(Conv 7×7 ([SADDLE avg ,SADDLE max ])) (8) Output the space-frequency two-branch attention weights that need to be retained at each spatial pixel position, and perform weighted summation of the solved weight coefficients and the input features, as shown in formula (9): Get the final space-frequency joint output result 4. The remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination according to claim 1 is characterized in that: In step 6, the number of training rounds is set to 600, the initial learning rate is set to 0.0001, and the learning rate is decayed at a rate of 0.5 in the 300th single cycle.

5. The remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination according to claim 1 is characterized in that: In step 6, the batch size is set to 4.

6. The remote sensing image super-resolution reconstruction method based on multi-stage space-frequency combination according to claim 1 is characterized in that: In step 6, the Adam optimizer is used to update the network parameters.

Images