A remote sensing image super-resolution reconstruction method

Abstract

The application discloses a remote sensing image super-resolution reconstruction method, and relates to the technical field of image processing; and comprises the following steps: preparing a data set; constructing a network model; training the network model; fine-tuning the network model; and solidifying the network model.The technical scheme of the application designs a multi-dimensional interaction and conditional discrimination structure and training mode, introduces high and low dimensional feature interaction in the generator, promotes the generation of images with more abundant details, introduces gradient and semantic information in the discriminator, guides the generator to learn more fine-grained gradient and semantic perception texture, so that the generator can generate more realistic images, and the quality of the super-resolution reconstructed images is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for super-resolution reconstruction of remote sensing images. Background Technology

[0002] Super-resolution (SR) technology for remote sensing images is a technique that converts low-resolution (LR) images into high-resolution (HR) images. Its purpose is to increase the image's resolution, making it clearer, more detailed, and more realistic. Compared to natural images, remote sensing images have a wider field of view, capturing more ground information and possessing richer texture information. However, due to the massive amount of data in remote sensing images, processing requires significant computational resources and time. Furthermore, in some cases, limitations in sensor performance, data transmission, and storage may result in low resolution of remote sensing images, making it difficult to present sufficient detail and clarity, thus limiting their practical applications. Super-resolution technology can synthesize high-resolution images from multiple low-resolution images, significantly improving the detail and clarity of remote sensing images. This technology is of great significance for improving the accuracy of remote sensing image analysis, object recognition capabilities, and monitoring effects, and can be widely applied in fields such as land cover classification, building detection, and 3D reconstruction.

[0003] Chinese patent publication number "CN113034361A" is titled "A Method for Super-Resolution Reconstruction of Remote Sensing Images Based on Improved ESRGAN." This technology improves the super-resolution reconstruction network for remote sensing images, comprising a generator network and a discriminator network. The generator network consists of 64 3×3 convolutional layers, 23 RRDB modules, and a LeakyReLU activation function. The discriminator network consists of 6 layers, employing a fully convolutional network with even-sized kernels, and adding BN layers and LeakyReLU activation layers between these layers. The first layer of the discriminator network receives the original low-resolution remote sensing image realA, the image magnified by bicubic interpolation, and the image fakeB output by the generator network as input, after channel merging. By alternately training the generator and discriminator networks and updating their parameters, an improvement to the remote sensing image super-resolution reconstruction network model is achieved. This method uses a single convolutional network, resulting in a lack of realism in the reconstruction effect and insufficient texture detail in the generated image.

[0004] In summary, how to solve the problems of unrealistic reconstruction results and insufficient detail and texture in current remote sensing image super-resolution reconstruction methods is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The technical solution of this invention to solve the above-mentioned technical problems is to provide a method for super-resolution reconstruction of remote sensing images, comprising the following steps:

[0006] Step 1, Prepare the dataset: Obtain the remote sensing image dataset and divide it into training set, validation set, and test set according to the proportions;

[0007] Step 2, Construct the network model: Construct an adversarial network model including a generator and a discriminator; the generator includes an initial module, a low-dimensional feature guidance group, a high-dimensional feature guidance group, a multi-dimensional interaction module, and a reconstruction module. The low-dimensional feature guidance group includes a bi-branch local module and a max-pooling layer, and the high-dimensional feature guidance group includes a global attention block and a sequence upsampling block; the discriminator includes a gradient information branch, a semantic information branch, a gradient-semantic fusion block, and multiple convolutional layers to evaluate the realism of the generated image;

[0008] Step 3, train the network model: train the generator and discriminator networks using the loss function until the number of training iterations reaches the initial threshold or the value of the loss function reaches the preset range, at which point the network model training is complete.

[0009] Furthermore, in step 2, the generator includes an initial module, three low-dimensional feature guidance groups, three high-dimensional feature guidance groups, three multi-dimensional interaction modules, and a reconstruction module. A skip connection is established between the low-dimensional feature guidance groups and the multi-dimensional interaction modules, and a global residual connection is introduced before the reconstruction module.

[0010] Furthermore, the initial module includes an upsampling operation and a 3×3 convolution. The upsampling operation employs a bicubic interpolation method. The low-dimensional feature guidance group includes three bi-branch local modules and a max-pooling layer, and utilizes local residual connections to fuse features for extracting low-dimensional multi-scale features from low-resolution images. The bi-branch local modules use channel segmentation technology to divide the features into two branches for processing. Branch one consists of a 3×3 convolutional layer, a 5×5 convolutional layer, a 7×7 convolutional layer, an L-shaped function, a channel concatenation operation, and a 1×1 convolutional layer. Features are fused through local residual connections. 3×3, 5×5, and 7×7 convolutional layers are used to obtain multi-scale information of the input features, and a 1×1 convolutional layer is used for depth feature extraction. Branch 2 consists of a 3×3 convolutional layer, a channel attention layer, and a spatial attention layer, and features are fused through local residual connections to extract local details in the channel and spatial dimensions. The output features of Branch 1 and Branch 2 are concatenated through channels and then passed through a 1×1 convolution to serve as the output of the final dual-branch local feature module. The max pooling layer is used to output the low-dimensional components of the features.

[0011] Furthermore, the high-dimensional feature guidance group includes a global attention block and a sequence upsampling block; the global attention block includes a spatial self-attention mechanism and a channel self-attention mechanism, which are used to extract global features. Through the first dimension transformation layer, the dimension is transformed from C×H×W to (HW)×C. Under the action of the 1×1 convolutional layer, dimension reduction is performed to generate Q (query), K (key) and V (value) feature matrices. The Q feature matrix and the K feature matrix are multiplied together.

[0012] The spatial dimension attention weight matrix is ​​calculated using the softmax function, and the V1 feature matrix is ​​obtained by multiplying the spatial dimension attention weight matrix with the V feature matrix.

[0013] Then, the Q, K, and V1 feature matrices are transformed through a second dimension transformation layer (converting the dimension from (HW)×C to C×(HW)) to obtain the Q′, K′, and V′ feature matrices. The Q′ feature matrix and the K′ feature matrix are then multiplied.

[0014] The attention weight matrix of the channel dimension is calculated using softmax. The attention weight matrix of the channel dimension is multiplied by the V feature matrix to obtain the attention output sequence. The sequence upsampling block includes a multilayer perceptron, a dimension transformation layer and a pixel rearrangement layer, which are used to output the high-dimensional components of the features. The multilayer perceptron doubles the channel dimension by introducing a nonlinear transformation. The dimension transformation layer transforms the dimension from C×(HW) to C×H×W. The pixel rearrangement layer uses the PixelShuffle upsampling method.

[0015] The multidimensional interaction module includes a 1×1 convolutional layer, a 5×5 convolutional layer, a 7×7 convolutional layer, an average pooling layer, a max pooling layer, channel concatenation, channel segmentation, a sigmoid function, and pixel-level multiplication. The 1×1 convolutional layer is used to reduce the number of channels and lower the computational cost of the process. Features are concatenated through 5×5 and 7×7 convolutional layers, then pooled and sigmoid functions are used to learn the weights of the mixed features. The weights represent the importance of features in different receptive fields. The calculated weights are multiplied by the low-dimensional and high-dimensional guided features respectively to obtain the selected dimension features from the mixed features. Finally, the features of the two branches are added to achieve adaptive fusion interaction between low-dimensional and high-dimensional features.

[0016] The reconstruction module includes a 3×3 convolutional layer and an L-shaped function, which integrates global residual features to better reconstruct the details and textures of the image.

[0017] Furthermore, in step 2, the discriminator includes a 4×4 convolutional layer, an L-shaped function, a gradient information branch, a semantic information branch, and three gradient-semantic fusion blocks;

[0018] The gradient information branch includes a gradient extraction block, a 4×4 convolutional layer, and an L-shaped function, used to extract gradient feature information; the 4×4 convolutional layer is used to adjust the scale of the gradient features, and the gradient extraction block uses gradient operators in the vertical and horizontal directions to obtain gradient information;

[0019] The semantic information branch consists of semantic extraction blocks, and a pre-trained CLIP "RN50" is used as a semantic extractor to obtain semantic information;

[0020] The gradient-semantic fusion block uses the input features obtained through convolutional layer 1, L-shaped function, and dimension transformation layer 1 as the content context information in the cross-attention mechanism. The semantic features obtained through the semantic feature optimization block are used as the semantic features in the cross-attention mechanism. The gradient features obtained through the gradient feature optimization block are used as the gradient features in the cross-attention mechanism. The content context information and semantic features are used to obtain distorted semantic perception image features through the cross-attention mechanism. The content context information and gradient features are used to obtain distorted gradient perception image features through the cross-attention mechanism. The distorted gradient perception image features and distorted semantic perception image features are used to obtain gradient-semantic perception image features through the cross-attention mechanism. These features are then processed through a layer normalization layer, G-shaped function, and dimension transformation layer 3 to obtain perception image features. The perception image features are connected with the original enhanced features and finally output by convolutional layer 2.

[0021] The gradient feature optimization block includes a 3×3 convolutional layer, a spatial attention layer, a dimension transformation layer, a layer normalization layer one, a residual self-attention mechanism, and a layer normalization layer two. Through the spatial attention layer and the residual self-attention mechanism, the gradient information of global spatial features is extracted.

[0022] The semantic feature optimization block includes a grouping normalization layer, a dimension transformation layer, a layer normalization layer one, a residual self-attention mechanism, and a layer normalization layer two. The residual self-attention mechanism is used to extract global semantic features.

[0023] Furthermore, step 3 also includes the following steps:

[0024] Selecting loss functions and determining evaluation metrics: Pixel loss, perceptual loss, and adversarial loss are used as loss functions, and value signal-to-noise ratio, structural similarity, and perceptual image similarity are used as evaluation metrics.

[0025] Furthermore, it also includes the following steps:

[0026] Step 4, fine-tune the network model: use the validation set to adjust the network model and optimize the network model parameters;

[0027] Step 5, solidify the network model: After fine-tuning is completed, solidify the fine-tuned network parameters and determine the final remote sensing image super-resolution network parameters.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This invention generates a super-resolution image using a generator, and then inputs the super-resolution image and the real image together into a discriminator to identify the authenticity of the image. The use of generative adversarial networks in the field of remote sensing super-resolution solves the problem of inaccurate super-resolution images.

[0030] 2. This invention designs a gradient-semantic fusion block, which uses a cross-attention mechanism to distort semantic perception and gradient texture from the image into the discriminator, thereby guiding the generator to learn finer-grained gradients and semantically perceptual textures, enabling the generator to generate more realistic images.

[0031] 3. This invention designs a dual-branch local module, which extracts rich local feature information through multi-scale and channel spatial attention dual branches, making the local details of the reconstructed image richer.

[0032] 4. This invention designs a multi-dimensional interaction module that adaptively obtains low-dimensional and high-dimensional features through convolution and pooling operations, realizing the interactive fusion of different dimensions, obtaining richer features, and making the reconstructed image of higher quality with more and richer details.

[0033] 5. This invention designs a composite loss function, which consists of three parts: adversarial loss, pixel loss, and perceptual loss. Compared with pixel loss alone, this composite loss makes up for the problem of overly smooth images caused by pixel loss. It can also learn the implicit relationship between low-resolution and high-resolution images, which helps to promote the network to generate high-quality super-resolution images. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the steps of the remote sensing image super-resolution reconstruction method described in this invention;

[0036] Figure 2 This is a schematic diagram of the generator structure in the method of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of the initial module in the method of the present invention;

[0038] Figure 4This is a schematic diagram of the structure of the dual-branch local module in the method of the present invention;

[0039] Figure 5 This is a schematic diagram of the global attention block structure in the method of the present invention;

[0040] Figure 6 This is a schematic diagram of the structure of the sequence upsampling block in the method of the present invention;

[0041] Figure 7 This is a schematic diagram of the structure of the multi-dimensional interaction module in the method of the present invention;

[0042] Figure 8 This is a schematic diagram of the reconstruction module in the method of the present invention;

[0043] Figure 9 This is a schematic diagram of the discriminator in the method of the present invention;

[0044] Figure 10 This is a schematic diagram of the gradient-semantic fusion block in the method of the present invention;

[0045] Figure 11 This is a schematic diagram of the cross-attention mechanism in the method of the present invention;

[0046] Figure 12 This is a schematic diagram of the gradient feature optimization block in the method of the present invention;

[0047] Figure 13 This is a schematic diagram of the residual self-attention mechanism in the method of the present invention;

[0048] Figure 14 This is a schematic diagram of the semantic feature optimization block in the method of the present invention. Detailed Implementation

[0049] This invention proposes a method for super-resolution reconstruction of remote sensing images, aiming to solve the problems of unrealistic reconstruction results and insufficient detail and texture in current super-resolution reconstruction methods for remote sensing images.

[0050] The method for super-resolution reconstruction of remote sensing images proposed in this invention will be described below in specific embodiments:

[0051] Example 1:

[0052] A method for super-resolution reconstruction of remote sensing images, such as Figure 1 As shown, it includes the following steps:

[0053] Step 1, Prepare the dataset: Obtain the remote sensing image dataset and divide it into training set, validation set, and test set according to the proportions;

[0054] Obtain remote sensing image dataset 1 and dataset 2; divide the data in dataset 1 into training set and validation set according to a certain ratio, while dataset 2 is used as test set. Then, perform bicubic downsampling on the high-resolution images in the two datasets to generate low-resolution images, thereby constructing the required image pairs. Finally, perform augmentation preprocessing on the obtained image pair dataset to expand the dataset.

[0055] Step 2, Construct the network model: Construct an adversarial network model including a generator and a discriminator; the generator includes an initial module, a low-dimensional feature guidance group, a high-dimensional feature guidance group, a multi-dimensional interaction module, and a reconstruction module. The low-dimensional feature guidance group includes a bi-branch local module and a max-pooling layer, and the high-dimensional feature guidance group includes a global attention block and a sequence upsampling block; the discriminator includes a gradient information branch, a semantic information branch, a gradient-semantic fusion block, and multiple convolutional layers to evaluate the realism of the generated image;

[0056] Step 3, train the network model: select the loss function and determine the evaluation index: use pixel loss, perceptual loss and adversarial loss as loss functions, and use value signal-to-noise ratio, structural similarity and perceptual image similarity as evaluation index;

[0057] The generator and discriminator networks are trained using a loss function until the number of training iterations reaches the initial threshold or the value of the loss function reaches a preset range, at which point the network model training is complete.

[0058] Step 4, fine-tune the network model: use the validation set to adjust the network model and optimize the network model parameters;

[0059] Step 5, solidify the network model: After fine-tuning is completed, solidify the fine-tuned network parameters and determine the final remote sensing image super-resolution network parameters.

[0060] Furthermore, in step 2, the generator includes an initial module, three low-dimensional feature guidance groups, three high-dimensional feature guidance groups, three multi-dimensional interaction modules, and a reconstruction module. A skip connection is established between the low-dimensional feature guidance groups and the multi-dimensional interaction modules, and a global residual connection is introduced before the reconstruction module.

[0061] Furthermore, the initial module includes an upsampling operation and a 3×3 convolution. The upsampling operation employs bicubic interpolation. The low-dimensional feature guidance group comprises three bi-branch local modules and a max-pooling layer, utilizing local residual connections to fuse features for extracting low-dimensional multi-scale features from low-resolution images. The bi-branch local modules use channel segmentation to divide features into two branches for processing. Branch one consists of a 3×3 convolutional layer, a 5×5 convolutional layer, a 7×7 convolutional layer, an L-shaped function, a channel concatenation operation, and a 1×1 convolutional layer, and... Local residual connections are used to fuse features. 3×3, 5×5, and 7×7 convolutional layers are used to obtain multi-scale information of the input features, and a 1×1 convolutional layer is used for depth feature extraction. Branch 2 consists of a 3×3 convolutional layer, a channel attention layer, and a spatial attention layer, and fuses features through local residual connections to extract local details in the channel and spatial dimensions. The output features of Branch 1 and Branch 2 are concatenated through channels and then passed through a 1×1 convolution to serve as the output of the final dual-branch local feature module. Max pooling layers are used to output the low-dimensional components of the features.

[0062] Furthermore, the high-dimensional feature guidance group includes a global attention block and a sequence upsampling block; the global attention block includes a spatial self-attention mechanism and a channel self-attention mechanism, which are used to extract global features. Through the first dimension transformation layer, the dimension is transformed from C×H×W to (HW)×C. Under the action of the 1×1 convolutional layer, dimensionality reduction is performed to generate Q (query), K (key) and V (value) feature matrices. The Q feature matrix and the K feature matrix are multiplied together.

[0063] The spatial dimension attention weight matrix is ​​calculated using the softmax function, and the V1 feature matrix is ​​obtained by multiplying the spatial dimension attention weight matrix with the V feature matrix.

[0064] Next, the Q, K, and V1 feature matrices are transformed from (HW)×C to C×(HW) through dimension transformation layer two, resulting in the Q′, K′, and V′ feature matrices. The Q′ feature matrix and the K′ feature matrix are then multiplied.

[0065] The attention weight matrix of the channel dimension is calculated using softmax. The attention weight matrix of the channel dimension is multiplied by the V feature matrix to obtain the attention output sequence. The sequence upsampling block includes a multilayer perceptron, a dimension transformation layer and a pixel rearrangement layer, which are used to output the high-dimensional components of the features. The multilayer perceptron doubles the channel dimension by introducing a non-linear transformation. The dimension transformation layer transforms the dimension from C×(HW) to C×H×W. The pixel rearrangement layer uses the PixelShuffle upsampling method.

[0066] The multidimensional interaction module includes 1×1 convolutional layers, 5×5 convolutional layers, 7×7 convolutional layers, average pooling layers, max pooling layers, channel concatenation operations, channel segmentation operations, sigmoid functions, and pixel-level multiplication operations. The 1×1 convolutional layers are used to reduce the number of channels and lower the computational cost of the process. Features are concatenated through 5×5 and 7×7 convolutional layers, and then pooled and sigmoid functions are used to learn the weights of the mixed features. The weights represent the importance of features under different receptive fields. The calculated weights are multiplied by low-dimensional and high-dimensional guided features respectively to obtain the selected dimension features from the mixed features. Finally, the features of the two branches are added to achieve adaptive fusion interaction between low-dimensional and high-dimensional features.

[0067] The reconstruction module includes 3×3 convolutional layers and L-shaped functions, integrating global residual features to better reconstruct the details and textures of the image.

[0068] Furthermore, in step 2, the discriminator includes a 4×4 convolutional layer, an L-shaped function, a gradient information branch, a semantic information branch, and three gradient-semantic fusion blocks;

[0069] The gradient information branch includes a gradient extraction block, a 4×4 convolutional layer, and an L-shaped function, which are used to extract gradient feature information; the 4×4 convolutional layer is used to adjust the scale of the gradient features; and the gradient extraction block uses gradient operators in the vertical and horizontal directions to obtain gradient information.

[0070] The semantic information branch consists of semantic extraction blocks, and a pre-trained CLIP "RN50" is used as the semantic extractor to obtain semantic information;

[0071] The gradient-semantic fusion block uses the input features obtained through convolutional layer 1, L-shaped function, and dimension transformation layer 1 as the content context information in the cross-attention mechanism. The semantic features obtained through the semantic feature optimization block are used as the semantic features in the cross-attention mechanism. The gradient features obtained through the gradient feature optimization block are used as the gradient features in the cross-attention mechanism. The content context information and semantic features are combined through the cross-attention mechanism to obtain distorted semantic perception image features. The content context information and gradient features are combined through the cross-attention mechanism to obtain distorted gradient perception image features. The distorted gradient perception image features and distorted semantic perception image features are combined through the cross-attention mechanism to obtain gradient-semantic perception image features. These features are then combined through a layer normalization layer, G-shaped function, and dimension transformation layer 3 to obtain perception image features. The perception image features are connected with the original enhanced features and finally output by convolutional layer 2.

[0072] The gradient feature optimization block includes a 3×3 convolutional layer, a spatial attention layer, a dimension transformation layer, a layer normalization layer 1, a residual self-attention mechanism, and a layer normalization layer 2. Through the spatial attention layer and the residual self-attention mechanism, the gradient information of global spatial features is extracted.

[0073] The semantic feature optimization block includes a grouping normalization layer, a dimension transformation layer, a layer normalization layer 1, a residual self-attention mechanism, and a layer normalization layer 2. The residual self-attention mechanism is used to extract global semantic features.

[0074] Example 2:

[0075] A method for super-resolution reconstruction of remote sensing images specifically includes the following steps:

[0076] Step 1, prepare the dataset;

[0077] The first remote sensing dataset is the AID dataset, containing 10,000 images covering 30 scene categories (airports, bare land, baseball fields, beaches, bridges, centers, churches, commercial areas, densely populated areas, deserts, farmland, forests, industrial areas, grasslands, medium-sized houses, mountains, parks, parking lots, playgrounds, ponds, ports, train stations, resorts, rivers, schools, sparsely populated areas, squares, stadiums, storage tanks, and viaducts). Each category has approximately 200-420 images, with an image size of 600 pixels × 600 pixels. The dataset is cleaned, resulting in 5,000 images, which are then divided into training and validation sets in a 7:3 ratio. The second remote sensing dataset is the UCMLU dataset, with images at a size of 256 pixels × 256 pixels. It contains 21 scene categories, with 100 images per category, for a total of 2,100 images. This dataset is cleaned, resulting in 1,500 images, and will be used as the test set. The cleaned dataset is downsampled by 4x bicubic to obtain low-resolution images, which are then used to construct paired datasets for training, validation, and testing.

[0078] In step 1, image augmentation is performed on the images in the dataset. For the same pair of images, the high-resolution image is randomly flipped and cropped to a size of 256×256, and the same operation is performed on the low-resolution image to obtain an input image size of 64×64, which is used as the input to the entire network. The random size and position can be implemented by software algorithms. The purpose of processing the images in the dataset through image augmentation is to enhance the robustness of the network and improve its generalization ability.

[0079] Step 2, Construct the network model; Construct a remote sensing image super-resolution reconstruction network including a generator and a discriminator;

[0080] Generator network models, such as Figure 2As shown, the generator consists of an initial module, low-dimensional feature guidance group one, low-dimensional feature guidance group two, low-dimensional feature guidance group three, high-dimensional feature guidance group one, multi-dimensional interaction module one, high-dimensional feature guidance group two, high-dimensional feature guidance group three, multi-dimensional interaction module three, and a reconstruction module.

[0081] Initial module structure, such as Figure 3 As shown, the image is first enlarged to the target size by 4x upsampling, and then the expressive power of the model is increased by dimensionality increase, so that the network can better learn the complex features of the input data. It consists of upsampling operation and 3×3 convolutional layer. The upsampling operation adopts the bicubic interpolation method. The convolutional kernel size of the convolutional layer is 3×3, the stride is 1, and the padding is 1.

[0082] Low-dimensional feature-guided groups, such as Figure 2 As shown in Figure 1, the low-to-medium dimensional feature guidance group can efficiently extract low-dimensional features at various scales. It consists of three bi-branch local modules and a max-pooling layer. The specific structure of the bi-branch local modules is as follows: Figure 4 As shown, channel segmentation technology is first used to divide the features into two branch features. and The processing is performed as follows: Branch 1 consists of 3×3 convolutional layers, 5×5 convolutional layers, 7×7 convolutional layers, L-shaped functions, channel connection operations, and 1×1 convolutional layers, and fuses features through local residual connections. The 3×3, 5×5, and 7×7 convolutional layers are used to obtain multi-scale information of the input features, and the 1×1 convolutional layer is used for depth feature extraction; Branch 2 consists of 3×3 convolutional layers, channel attention layers, and spatial attention layers, and fuses features through local residual connections; Branch 1 and Branch 2 are then combined. The output features of path 2 are concatenated through channels and finally passed through a 1×1 convolution to become the output of the final dual-branch local feature module. Specifically, the 3×3 convolutional layer has a kernel size of 3×3, a stride of 1, and padding of 1; the 5×5 convolutional layer has a kernel size of 5×5, a stride of 1, and padding of 2; the 7×7 convolutional layer has a kernel size of 7×7, a stride of 1, and padding of 3; and the 1×1 convolutional layer has a kernel size of 1×1, a stride of 1, and padding of 0. This process can be expressed by the following formula:

[0083] X in = [X1, X2],

[0084] F out =Conv 1×1 ([Branch1(X1),Branch2(X2)]),

[0085] in, The features representing input branch one, This represents the features of input branch two. This represents the input characteristics of a bi-branch local module. The output features of the two-branch local modules are represented by Branch1(·)(branch one), which represents convolution, L-shaped functions, and all other operations within Branch1(·)(branch one). Branch2(·)(branch two) represents convolution, channel attention, spatial attention, and all other operations within Branch2(·)(branch two). 1×1 This represents a 1×1 convolution operation;

[0086] High-dimensional feature-guided groups, such as Figure 2 As shown in Figure 1, the high-dimensional feature guidance group can efficiently extract high-dimensional features across the entire domain. It consists of a global attention block and a sequence upsampling block. The structure of the global attention block is as follows: Figure 5 As shown, the input features enter the global attention block. First, they pass through a dimension transformation layer (converting the dimension from C×H×W to (HW)×C), and then undergo dimensionality reduction under the action of a 1×1 convolutional layer to generate Q (query), K (key), and V (value) feature matrices. The Q feature matrix and the K feature matrix are multiplied, and then the spatial attention weight matrix is ​​calculated using the softmax function. The spatial attention weight matrix is ​​multiplied by the V feature matrix to obtain the V1 feature matrix. Finally, the Q, K, and V1 feature matrices are passed through a dimension transformation layer (converting the dimension from (HW)×C to C×(HW)) to obtain the Q′, K′, and V′ features. The feature matrix, Q′ feature matrix, and K′ feature matrix are multiplied together, and then softmax is used to calculate the channel attention weight matrix. The channel attention weight matrix is ​​multiplied with the V′ feature matrix to obtain the attention output sequence. The sequence upsampling module consists of a multilayer perceptron, a dimension transformation layer, and a pixel rearrangement layer, which is used to output the high-dimensional components of the features. The multilayer perceptron doubles the channel dimension by introducing a nonlinear transformation. The dimension transformation layer transforms the dimension from C×(HW) to C×H×W. The pixel rearrangement layer uses the PixelShuffle upsampling method. Spatial self-attention and channel self-attention can be designed as follows:

[0087]

[0088] in, K is the scaling factor. T Let K' be the transpose of matrix K. T It is the transpose of matrix K′;

[0089] Sequence upsampling blocks, such as Figure 6 As shown, it consists of a multilayer perceptron, a dimension transformation layer, and a pixel rearrangement layer, which are used to output high-dimensional components of features. The multilayer perceptron doubles the channel dimension by introducing a nonlinear transformation. The dimension transformation layer transforms the dimension from C×(HW) to C×H×W. The pixel rearrangement layer uses the PixelShuffle upsampling method.

[0090] The structure of the multi-dimensional interactive module is as follows Figure 7 As shown, the algorithm consists of 1×1 convolutional layers, 5×5 convolutional layers, 7×7 convolutional layers, average pooling layers, max pooling layers, feature concatenation operations, feature segmentation operations, a sigmoid function, and multiplication operations. The 1×1 convolutional layers are used to reduce the number of channels and lower the computational cost of the process. Features are concatenated through the 5×5 and 7×7 convolutional layers according to the channel dimension, and then pooling and the sigmoid function are used to calculate the weights of the mixed features. The weights represent the importance of features under different receptive fields. The calculated weights are multiplied by the low-dimensional guided features and the high-dimensional guided features respectively to obtain the selected dimension features from the mixed features. Finally, the features of the two branches are added to achieve adaptive fusion interaction between low-dimensional and high-dimensional features. Specifically, the kernel size of the 1×1 convolutional layer is 1×1, the stride is 1, and the padding is 0; the kernel size of the 5×5 convolutional layer is 5×5, the stride is 1, and the padding is 2; and the kernel size of the 7×7 convolutional layer is 7×7, the stride is 1, and the padding is 3.

[0091] The structure of the reconstruction module is as follows: Figure 8 As shown, it consists of 3×3 convolutional layers and L-shaped functions, with a kernel size of 3×3, a stride of 1, and padding of 1. The reconstruction module better reconstructs the details and textures of the image;

[0092] Discriminator network model such as Figure 9 As shown, it consists of a 4×4 convolutional layer, an L-shaped activation function, a gradient extraction branch, a semantic extraction branch, and three gradient-semantic fusion blocks. The 4×4 convolutional layer has a kernel size of 4×4, a stride of 2, and padding of 1. By introducing gradient and semantic information through gradient and semantic information branches, the discriminator can better focus on the details of the image and improve the performance of the generator, resulting in more realistic and detailed images.

[0093] The gradient information branch consists of a gradient extraction block, a 4×4 convolutional layer, and an L-shaped function. It is used to extract gradient feature information. The 4×4 convolutional layer is used to adjust the scale of the gradient features, with a kernel size of 4×4, a stride of 2, and padding of 1. The gradient extraction block uses gradient operators in the vertical and horizontal directions to obtain gradient information. The horizontal and vertical convolutions used are as follows:

[0094] The horizontal convolution kernel is:

[0095]

[0096] The vertical convolution kernel is:

[0097]

[0098] The formulas used for calculating the gradient in the horizontal direction, the gradient in the vertical direction, and the gradient magnitude are as follows:

[0099]

[0100] Where i∈[0,1,2] represents the three input channels, and ∈ is a small constant to prevent division by zero errors.

[0101] The semantic information branch consists of semantic extraction blocks, and a pre-trained CLIP "RN50" is used as the semantic extractor to obtain semantic information;

[0102] The structure of the gradient-semantic fusion block is as follows: Figure 10 As shown, the system consists of Convolutional Layer 1, L-shaped function, Dimension Transformation Layer 1, Cross-Attention Mechanism, Gradient Feature Optimization Block, Semantic Feature Optimization Block, Layer Normalization Layer, G-shaped function, and Dimension Transformation Layer 2. Convolutional Layer 1 has a 4×4 kernel size, a stride of 2, and padding of 1. Convolutional Layer 2 has a 3×3 kernel size, a stride of 1, and padding of 1. The input features, after passing through Convolutional Layer 1, L-shaped function, and Dimension Transformation Layer 1 (converting the dimension from C×H×W to (HW)×C), are used as the content context information in the Cross-Attention Mechanism. The semantic features, after passing through the Semantic Feature Optimization Block, are used as the semantic features in the Cross-Attention Mechanism. Gradient features... The features obtained after gradient feature optimization are used as gradient features in the cross-attention mechanism. Content context information and semantic features are processed by the cross-attention mechanism to obtain distorted semantic-aware image features. Content context information and gradient features are processed by the cross-attention mechanism to obtain distorted gradient-aware image features. Distorted gradient-aware image features and distorted semantic-aware image features are processed by the cross-attention mechanism to obtain gradient-semantic-aware image features. These features are then processed by layer normalization, G-type function, and dimension transformation layer 2 (converting the dimension from (HW)×C to C×H×W) to obtain perceptual image features, which are then concatenated with the original enhanced features. Finally, the features are output by convolutional layer 2. The structure of the cross-attention mechanism is as follows: Figure 11 As shown, semantic / gradient features are reduced in dimensionality by a 1×1 convolutional layer to generate a Q (query) feature matrix. Context input features are reduced in dimensionality by a 1×1 convolutional layer to generate K (key) and V (value) feature matrices. The Q matrix and K matrix are multiplied together, and then the softmax function is used to calculate the spatial dimension attention weight matrix. The spatial dimension attention weight matrix is ​​multiplied with the V feature matrix to obtain the context output feature matrix.

[0103] The structure of the gradient feature optimization block is as follows: Figure 12As shown, it consists of a 3×3 convolutional layer, a spatial attention layer, a dimension transformation layer, a layer normalization layer one, a residual self-attention mechanism, and a layer normalization layer two. The 3×3 convolutional layer has a 3×3 kernel size, a stride of 1, and padding of 1. The spatial attention layer and the residual self-attention mechanism extract gradient information of global spatial features. The structure of the residual self-attention mechanism is as follows: Figure 13 As shown, the input features are reduced in dimensionality by a 1×1 convolutional layer to generate Q (query), K (key), and V (value) feature matrices. The Q matrix and K matrix are multiplied, and then the softmax function is used to calculate the spatial attention weight matrix. The spatial attention weight matrix is ​​multiplied by the V feature matrix to obtain the V2 feature matrix. The V2 feature matrix is ​​then multiplied by the original input features after passing through a 1×1 convolutional layer to obtain the final output features.

[0104] The structure of the semantic feature optimization block is as follows: Figure 14 As shown, it consists of a grouping normalization layer, a dimension transformation layer, a layer normalization layer one, a residual self-attention mechanism, and a layer normalization layer two. The residual self-attention mechanism is used to extract global semantic features.

[0105] To ensure the robustness of the network, retain more structural information, and fully extract image features, this invention uses two sets of activation functions: the L-type function and the G-type function. The definitions of the L-type function and the G-type function are as follows:

[0106]

[0107] Step 3, train the network model; first, select an appropriate loss function and determine the evaluation metric: based on the network model in Step 2, select an appropriate loss function to minimize the difference between the super-resolution image reconstructed by the network and the high-resolution image, and determine the evaluation metric to evaluate the network performance; the loss function used by the generator consists of three types: pixel loss, adversarial loss, and perceptual loss; the discriminator only uses adversarial loss;

[0108] In supervised image super-resolution tasks, in order to make the generated image (SR) as close as possible to the real high-resolution image (HR), L1 loss is used to calculate the error of the values ​​at corresponding pixel positions in SR and HR, i.e., pixel loss, as shown in the following formula:

[0109]

[0110] Where x is the input low-resolution image, y is the high-resolution image, and G(·) is the generator.

[0111] To help the generator network improve the quality of the super-reconstructed image, correct reconstruction details, and learn the implicit relationship between the super-reconstructed image and the real image, an adversarial loss is adopted, as shown in the following formula:

[0112]

[0113] Where x is the input low-resolution image, y is the high-resolution image, and G(·) and D(·) are the generator and discriminator, respectively.

[0114] The generator improves image quality by capturing semantic information and structural features of the image. It uses a pre-trained VGG-19 network to calculate the feature difference between the generated image and the real image, i.e., the perceptual loss, as shown in the following formula:

[0115]

[0116] Where l represents the convolutional layer of the VGG network, λ l M represents the weights of the l-th layer. l N l The sub-table represents the size of the feature map of layer l. φ l (G(x)) and φ l (y) represents the feature representation of the generated image and the real image at layer l, where i and j represent the pixel indices in the feature map.

[0117] The generator's total loss function is shown below:

[0118] L G =L1+λ a L adv +λ p L per ,

[0119] Where, λ a , λ p λ and λ' represent the weights of different loss components in the generator's total loss. The weight settings are based on preliminary experiments on the training dataset. In this experiment, λ is set to... a =0.1, λ p =0.5. By optimizing the final overall loss function, the network can learn sharper edges and more detailed textures, resulting in more realistic and visually better reconstructed super-resolution images.

[0120] The discriminator loss uses adversarial loss, and its formula is as follows:

[0121]

[0122] Where x is the input low-resolution image, y is the high-resolution image, and G(·) and D(·) are the generator and discriminator, respectively.

[0123] The evaluation metrics selected are Peak Signal-to-Noise Ratio (PSNR), Structural Similarity, and Perceptual Image Similarity. PSNR measures the difference between the super-resolution image and the ground truth image; a higher PSNR indicates better super-resolution image quality. Structural Similarity measures contour preservation during the super-resolution process by calculating the structural similarity between the super-resolution image and the ground truth image; a higher structural similarity indicates smaller differences. Perceptual Image Similarity focuses on the perceptual similarity between the super-resolution image and the ground truth image; a lower perceptual image similarity indicates greater similarity between the two images. The definitions of PSNR, Structural Similarity, and Perceptual Image Similarity are as follows:

[0124]

[0125] Where, μ x μ y Let x and y represent the mean values ​​of the image, respectively. and Let σ represent the standard deviations of the image x and y, respectively. xy Let C1 and C2 represent the covariance of the images x and y, where C1 and C2 are constants, and w l These are trainable weight parameters;

[0126] Secondly, train the remote sensing super-resolution network model: start training the network using the loss function, and continue training until the number of training iterations reaches a set threshold or the value of the loss function reaches a set range. The model parameters are then considered to be trained successfully, and the model parameters are saved. Select the test dataset from dataset one to test the network model, and evaluate the model performance using the evaluation metrics selected above.

[0127] All experiments were conducted on an AutoDL cloud server, with an NVIDIA RTX 3080Ti GPU (12GB) used for algorithm acceleration. The training cycle was set to 200 epochs, and the learning rate for both the generator and discriminator was set to 1e-4. The Adam optimizer was chosen as the network parameter optimizer due to its simplicity, computational efficiency, low memory requirements, and the fact that parameter updates are unaffected by gradient scaling, resulting in relatively stable parameters. The network was considered essentially trained when the discriminator's ability to identify fake images and the generator's ability to deceive the discriminator reached a balance.

[0128] Step 4: Fine-tune the network model; use the validation set of dataset 1 from step 1 to adjust the network, optimize the network model parameters, and further improve the performance of the super-resolution reconstruction network.

[0129] Step 5: Solidify the network model; After the fine-tuning in Step 4 is completed, solidify the fine-tuned network parameters and determine the final super-resolution network parameters; When subsequent work requires super-resolution reconstruction of remote sensing images, directly use the low-resolution remote sensing image as the input of the super-resolution network to obtain a better super-resolution result.

[0130] The implementations of convolution, activation functions, multilayer perceptrons, feature concatenation, max pooling, average pooling, CLIP RN50 semantic extraction, gradient operators, instance normalization, layer normalization, channel attention, spatial attention, and group normalization are algorithms well-known to those skilled in the art, and the specific processes and methods can be found in relevant textbooks or technical literature.

[0131] This invention constructs a remote sensing image super-resolution reconstruction method based on multi-dimensional interaction and conditional discrimination, which can directly super-reconstruct low-resolution remote sensing images into high-resolution images. Under the same conditions, the feasibility and superiority of this method are further verified by calculating the image correlation index with existing methods. The existing technical solution is a generative adversarial network. The generator adopts an RRDB structure, i.e., dense residual connections, and the two ends of the residual edges are connected in a concat manner, consisting of convolutional layers and 23 RRDBs. The discriminator adopts a SeD structure, i.e., a semantic-aware discriminator, consisting of 3 semantic-aware fusion blocks and 2 convolutional layers. This method comes from the SeD semantic-aware discriminator for image super-resolution proposed by the University of Science and Technology of China. The paper is SeD: Semantic-Aware Discriminator for Image Super-Resolution. The correlation index comparison between existing technologies and the method proposed in this invention is shown below.

[0132]

[0133] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for super-resolution reconstruction of remote sensing images, characterized in that, The method comprises the following steps: Step 1, preparing a data set: obtaining a remote sensing image data set, and dividing the data set into a training set, a validation set and a test set in proportion; Step 2, constructing a network model: constructing a generative adversarial network model comprising a generator and a discriminator; the generator comprises an initial module, a low-dimensional feature guide group, a high-dimensional feature guide group, a multi-dimensional interaction module and a reconstruction module, the low-dimensional feature guide group comprises a double-branch local module and a max-pooling layer, and the high-dimensional feature guide group comprises a global attention block and a sequence upsampling block; the discriminator comprises a gradient information branch, a semantic information branch, a gradient-semantic fusion block and a plurality of convolution layers, and is used for evaluating the authenticity of a generated image; Step 3, training the network model: training the generator and the discriminator network using a loss function until the number of training reaches an initial set threshold or the value of the loss function reaches a preset range, and then the network model training is completed; In step 2, the discriminator comprises a 4x4 convolution layer, an L-shaped function, a gradient information branch, a semantic information branch and three gradient-semantic fusion blocks; The gradient information branch comprises a gradient extraction block, a 4x4 convolution layer and an L-shaped function, and is used for extracting gradient feature information; the 4x4 convolution layer is used for adjusting the size of the gradient feature, and the gradient extraction block obtains gradient information by using gradient operators in the vertical direction and the horizontal direction; The semantic information branch is composed of a semantic extraction block, and a pre-trained CLIP "RN50" is used as a semantic extractor to obtain semantic information; The gradient-semantic fusion block: the features obtained by the input features through a convolution layer one, an L-shaped function and a dimension conversion layer one are used as content context information in a cross-attention mechanism, the features obtained by the semantic features through a semantic feature optimization block are used as semantic features in the cross-attention mechanism, the features obtained by the gradient features through a gradient feature optimization block are used as gradient features in the cross-attention mechanism, the content context information and the semantic features are subjected to the cross-attention mechanism to obtain a distorted semantic perception image feature, the content context information and the gradient features are subjected to the cross-attention mechanism to obtain a distorted gradient perception image feature, the distorted gradient perception image feature and the distorted semantic perception image feature are subjected to the cross-attention mechanism to obtain a gradient-semantic perception image feature, and the gradient-semantic perception image feature is subjected to a layer normalization layer, a G-shaped function and a dimension conversion layer three to obtain a perception image feature, the perception image feature is connected with an original enhanced feature, and finally the features are output by a convolution layer two; The gradient feature optimization block comprises a 3x3 convolution layer, a spatial attention layer, a dimension conversion layer, a layer normalization layer one, a residual self-attention mechanism and a layer normalization layer two, and the gradient information of the global spatial feature is extracted through the spatial attention layer and the residual self-attention mechanism; The semantic feature optimization block comprises a grouping normalization layer, a dimension conversion layer, a layer normalization layer one, a residual self-attention mechanism and a layer normalization layer two, and the global semantic feature is extracted through the residual self-attention mechanism.

2. The remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, In step 2, the generator comprises an initial module, three low-dimensional feature guide groups, three high-dimensional feature guide groups, three multi-dimensional interaction modules and a reconstruction module, a skip connection is introduced between the low-dimensional feature guide group and the multi-dimensional interaction module, and a global residual connection is introduced before the reconstruction module.

3. The remote sensing image super-resolution reconstruction method according to claim 2, characterized in that, The initial module comprises an upsampling operation and a 3*3 convolution, the upsampling operation adopts a bicubic interpolation method, the low-dimensional feature guide group comprises three double-branch local modules and a max pooling layer, and local residual connections are used to fuse features, so as to extract low-dimensional multi-scale features of a low-resolution image; the double-branch local module uses a channel segmentation technology to divide features into two branch features and processes them; Branch one is composed of a 3*3 convolution layer, a 5*5 convolution layer, a 7*7 convolution layer, an L-shaped function, a channel splicing operation and a 1*1 convolution layer, and features are fused through a local residual connection, the 3*3 convolution layer, the 5*5 convolution layer and the 7*7 convolution layer are used to obtain multi-scale information of input features, and the 1*1 convolution layer is used for deep feature extraction; Branch two is composed of a 3*3 convolution layer, a channel attention layer and a spatial attention layer, and features are fused through a local residual connection, so as to extract local details in the channel dimension and the spatial dimension; after the output features of branch one and branch two are subjected to a channel splicing operation, they are subjected to a 1*1 convolution to serve as the output of the final double-branch local feature module; the max pooling layer is used to output low-dimensional components of features.

4. The remote sensing image super-resolution reconstruction method of claim 2, wherein, The high-dimensional feature guide group comprises a global attention block and a sequence upsampling block; the global attention block comprises a spatial self-attention mechanism and a channel self-attention mechanism, is used to extract global features, and is subjected to a dimension conversion layer one to convert the dimension from C*H*W to (H W)*C, is subjected to a 1*1 convolution layer to reduce the dimension to generate Q, K and V feature matrices, Q feature matrix and K feature matrix are multiplied; a spatial dimension attention weight matrix is calculated through a softmax function, the spatial dimension attention weight matrix and V feature matrix are multiplied to obtain V1 feature matrix; Q, K and V1 feature matrices are subjected to a dimension conversion layer two to convert the dimension from (H W)*C to C*(H W) to obtain Q', K' and V' feature matrices, Q' feature matrix and K' feature matrix are multiplied, a channel dimension attention weight matrix is calculated through a softmax function, the channel dimension attention weight matrix and V feature matrix are multiplied to obtain an attention output sequence; the sequence upsampling block comprises a multi-layer perception, a dimension conversion layer and a pixel rearrangement layer, is used to output high-dimensional components of features, the multi-layer perception doubles the channel dimension by introducing a nonlinear transformation, the dimension conversion layer converts the dimension from C*(H W) to C*H*W, and the pixel rearrangement layer uses a PixelShuffle upsampling method. The multi-dimensional interaction module comprises a 1*1 convolution layer, a 5*5 convolution layer, a 7*7 convolution layer, an average pooling layer, a maximum pooling layer, a channel splicing operation, a channel splitting operation, a Sigmoid function and a pixel-level multiplication operation, the 1*1 convolution layer is used to reduce the number of channels and reduce the calculation cost of the process, the features are respectively subjected to the 5*5 convolution layer and the 7*7 convolution layer, and then subjected to the channel splicing operation, and then subjected to the pooling and the Sigmoid function calculation to learn the weight of the mixed feature, the weight represents the importance of the feature under different receptive fields, the calculated weight is multiplied with the low-dimensional guide and the high-dimensional guide feature respectively, the selected dimensional feature is obtained from the mixed feature, finally, the features of the two branches are added, and the adaptive fusion interaction of the low-dimensional feature and the high-dimensional feature is realized; The reconstruction module comprises a 3*3 convolution layer and an L-shaped function, and integrates global residual features, so that the details and textures of the image are better reconstructed.

5. The remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, In step 3, the following steps are further included: selecting a loss function and determining an evaluation index: using a pixel loss, a perception loss and an adversarial loss as the loss function, and using a value signal-to-noise ratio, a structural similarity and a perception image similarity as the evaluation index.

6. The remote sensing image super-resolution reconstruction method according to claim 1, characterized in that, Further comprising the following steps: Step 4, fine-tuning the network model: using the verification set to adjust the network model and optimize the network model parameters; Step 5, solidifying the network model: after the fine-tuning is completed, the fine-tuned network parameters are solidified, and the final remote sensing image super-resolution network parameters are determined.

Images