Remote sensing image super-resolution method combining global and local features

Abstract

The invention provides a remote sensing image super-resolution method combining global and local features, and relates to the technical field of remote sensing image processing, and the method comprises the steps: obtaining an original low-resolution remote sensing image, carrying out the preprocessing, and constructing a data set; a remote sensing image reconstruction network based on Transfomer is constructed; the remote sensing image reconstruction network comprises a feature simplification feedforward network, a convolutional layer, a plurality of residual joint attention groups connected in sequence and a reconstruction module; training a remote sensing image reconstruction network through the data set; obtaining a to-be-reconstructed remote sensing image; obtaining a to-be-reconstructed remote sensing image; and inputting a remote sensing image to be reconstructed into the trained remote sensing image reconstruction network to obtain a reconstructed super-resolution image. According to the technical scheme, the CBAM and the W-MSA are integrated into the Transfomer model, so that global channel and spatial feature information can be effectively captured, and the local feature representation capability is high.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a method for super-resolution of remote sensing images that combines global and local features. Background Technology

[0002] High spatial resolution remote sensing images play a crucial role in various fields, such as urban planning and agricultural monitoring. However, acquiring high-resolution remote sensing images directly is costly due to hardware limitations. Compared to upgrading hardware, super-resolution technology offers a more flexible and economical solution. Image super-resolution aims to reconstruct high-quality, high-resolution images from low-quality, low-resolution images. Since SRCNN first applied deep convolutional neural networks (CNNs) to image super-resolution tasks and outperformed traditional SR methods, many CNN-based super-resolution methods have emerged and achieved significant progress in image super-resolution tasks. Although numerous studies have confirmed the effectiveness of convolutional neural networks in practical applications, the model still has significant limitations: firstly, the static weight characteristics make it difficult for CNNs to be flexibly adjusted for different instances; secondly, the sparse connection structure hinders the effective modeling of global dependencies.

[0003] In recent years, numerous methods based on the Transformer architecture have been applied to the field of super-resolution. These methods have continuously broken through technical bottlenecks and broken performance records in super-resolution tasks, fully demonstrating the powerful feature representation capabilities of the Transformer. For example, IPT is a pioneering Transformer model in the field of image inpainting. However, the standard self-attention mechanism suffers from a quadratic increase in computational complexity, which keeps the computational cost high and, to some extent, limits the application expansion of the Transformer in the super-resolution field. To overcome this obstacle, researchers have proposed a series of efficient attention mechanisms, such as local window attention and shift window attention, all aimed at alleviating the aforementioned computational challenges. Although these methods have achieved significant results in terms of both performance and computational efficiency, they limit the scope of attention to local regions, causing the model to focus on mining local correlations between pixels rather than capturing global dependencies. This, to some extent, deviates from the key elements of the Transformer architecture's success. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing Transformer-based remote sensing image super-resolution reconstruction methods cannot balance computational cost and reconstruction accuracy, and to provide a remote sensing image super-resolution method that combines global and local features.

[0005] The above-mentioned objective of this application is achieved through the following technical solution: S1: Acquire raw low-resolution remote sensing images and preprocess them to construct a dataset; S2: Construct a remote sensing image reconstruction network based on Transformer; the remote sensing image reconstruction network includes: a feature-reduced feedforward network, convolutional layers, multiple sequentially connected residual joint attention groups, and a reconstruction module; the convolutional layers, multiple sequential residual joint attention groups, and the reconstruction module are connected sequentially. S3: Train a remote sensing image reconstruction network using a dataset; S4: Obtain the remote sensing image to be reconstructed; input the remote sensing image to be reconstructed into the trained remote sensing image reconstruction network to obtain the reconstructed super-resolution image.

[0006] Optionally, step S1 includes: Data augmentation processing is performed on the raw low-resolution remote sensing image data.

[0007] Optionally, step S3 includes: S31: Extract shallow features from the preprocessed low-resolution remote sensing image; S32: Input the shallow features into the constructed residual joint attention group to obtain deep features; use the feature simplification feedforward network to remove feature redundancy in the deep features; S33: Input shallow and deep features into the constructed reconstruction module to generate high-resolution, high-quality images.

[0008] Optionally, step S31 includes: A single convolutional layer is used to extract shallow features from the preprocessed low-resolution remote sensing image. Where H, W, and C represent shallow features, respectively. Height, width, and number of channels.

[0009] Optionally, step S32 includes: The residual joint attention group consists of: six joint attention blocks (JABs), one overlapping cross attention block (OCAB), and a convolutional layer with residual connections; Six joint attention blocks (JABs), one overlapping cross attention block (OCAB), and a convolutional layer with residual connections are sequentially connected. Each Joint Attention Block (JAB) includes: a first LayerNorm layer, a Convolutional Block Attention Module (CBAM), a Window-based Multi-Head Self-Attention Mechanism (W-MSA) module, a second LayerNorm layer, and a multilayer perceptron; The entire processing procedure of the Joint Attention Block (JAB) is represented as follows:

[0010]

[0011]

[0012] in This represents the input features of the Joint Attention Block (JAB). This represents the LayerNorm layer; This represents the features output by the LayerNorm layer; and Indicates intermediate features; This represents the Convolutional Block Attention Module (CBAM). denoted as W-MSA (Window-based Multi-head Self-Attention Mechanism Module); MLP represents a multilayer perceptron; Y is the output feature of the Joint Attention Block (JAB). Input features are used to simplify the deep features of the feedforward network. After sequentially applying layer normalization, partial convolution, and linear mapping, feature redundancy in deep features is removed, as follows:

[0013] in , For output features The number of channels; Standardization; Indicates partial convolution; Represents a linear mapping; It is the GELU activation function; Will Divide the channel into equal parts as follows:

[0014] in Indicates a delimiter operation; and , which are the features after segmentation; right The output features are obtained by sequentially performing reconstruction, depthwise convolution, and flattening. ; Will and Perform multiplication and then apply a linear mapping to the result, as follows:

[0015] in ; Indicates the multiplication operation; Represents a linear mapping; Input features and Add them together to obtain the deep features of the output. .

[0016] Optionally, step S33 includes: Shallow and deep features are fused together using a residual connection to obtain fused features; The fused features are fed into the reconstruction module to generate high-resolution, high-quality images; The reconstruction module includes two convolutional layers and a pixel shuffling layer; the pixel shuffling layer is used to upsample the fused features.

[0017] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform a remote sensing image super-resolution method that combines global and local features.

[0018] A computer-readable storage medium storing instructions that, when executed, perform a remote sensing image super-resolution method combining global and local features.

[0019] The beneficial effects of the technical solution provided in this application are: This application integrates CBAM and W-MSA into the Transformer model, enabling it to effectively capture global channel and spatial feature information while possessing powerful local feature representation capabilities. A feature simplification feedforward network is also introduced. This network employs enhancement and simplification strategies to emphasize key effective information in the feature map while removing redundant features. This invention can reconstruct high-quality, high-resolution remote sensing images. Attached Figure Description

[0020] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a step diagram of an embodiment of this application; Figure 2 This is a module connection diagram in the embodiments of this application; Figure 3 This is a structural diagram of the residual joint attention group in the embodiments of this application; Figure 4 This is a structural diagram of the joint attention block in the embodiments of this application; Figure 5 This is a structural diagram of the overlapping attention block in the embodiments of this application; Figure 6 This is a structural diagram of the convolutional block attention module in the embodiments of this application; Figure 7This is a structural diagram of the feature-simplified feedforward network in the embodiments of this application; Figure 8 This is a schematic diagram of the electronic device structure in the embodiments of this application. Detailed Implementation

[0021] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] Embodiments of this application provide a method for super-resolution of remote sensing images that combines global and local features.

[0023] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a remote sensing image super-resolution method combining global and local features according to an embodiment of this application, including: S1: Acquire raw low-resolution remote sensing images and preprocess them to construct a dataset; As one embodiment, data is randomly selected from low-resolution remote sensing images processed according to a preset ratio to construct training, validation, and test sets. The residual joint attention group and reconstruction module are trained using the training set. The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the quality of the reconstructed images.

[0024] S2: Construct a remote sensing image reconstruction network based on Transformer; the remote sensing image reconstruction network includes: a feature-reduced feedforward network, convolutional layers, multiple sequentially connected residual joint attention groups, and a reconstruction module; the convolutional layers, multiple sequential residual joint attention groups, and the reconstruction module are connected sequentially. S3: Train a remote sensing image reconstruction network using a dataset; S4: Obtain the remote sensing image to be reconstructed; input the remote sensing image to be reconstructed into the trained remote sensing image reconstruction network to obtain the reconstructed super-resolution image.

[0025] Step S1 includes: Data augmentation processing is performed on the raw low-resolution remote sensing image data.

[0026] Step S3 includes: S31: Extract shallow features from the preprocessed low-resolution remote sensing image; S32: Input the shallow features into the constructed residual joint attention group to obtain deep features; use the feature simplification feedforward network to remove feature redundancy in the deep features; S33: Input shallow and deep features into the constructed reconstruction module to generate high-resolution, high-quality images.

[0027] As one embodiment, this method integrates a convolutional block attention module and a window-based self-attention mechanism into the Transformer, enabling it to capture not only global channel and spatial information but also excellent local feature representation capabilities. Furthermore, this method introduces a feature simplification feedforward network to reduce feature redundancy during image reconstruction. This achieves more efficient and accurate restoration of clear latent images.

[0028] Step S31 includes: A single convolutional layer is used to extract shallow features from the preprocessed low-resolution remote sensing image. Where H, W, and C represent shallow features, respectively. Height, width, and number of channels.

[0029] Step S32 includes: The residual joint attention group consists of: six joint attention blocks (JABs), one overlapping cross attention block (OCAB), and a convolutional layer with residual connections; Six joint attention blocks (JABs), one overlapping cross attention block (OCAB), and a convolutional layer with residual connections are sequentially connected. Each Joint Attention Block (JAB) includes: a Convolutional Block Attention Module (CBAM) and a window-based multi-head self-attention mechanism (W-MSA); The entire processing procedure of the Joint Attention Block (JAB) is represented as follows:

[0030]

[0031]

[0032] The entire processing procedure of the Joint Attention Block (JAB) is represented as follows:

[0033]

[0034]

[0035] in This represents the input features of the Joint Attention Block (JAB). This represents the LayerNorm layer; This represents the features output by the LayerNorm layer; and Indicates intermediate features; This represents the Convolutional Block Attention Module (CBAM). denoted as W-MSA (Window-based Multi-head Self-Attention Mechanism Module); MLP represents a multilayer perceptron; Y is the output feature of the Joint Attention Block (JAB). As one implementation, to simultaneously capture key feature information, JAB combines a Convolutional Block Attention Module (CBAM) and a Window-based Multi-head Self-attention (W-MSA) mechanism. The input feature X passes through a LayerNorm (LN) layer, with the information flow proceeding in parallel through CBAM and W-MSA. The output of CBAM is then multiplied by a constant. To avoid potential conflicts between CBAM and W-MSA in optimization and visual representation, shifted window self-attention (SW-MSA) is used intermittently in consecutive JABs. Ultimately, the constructed JAB (Joint Attention Block) effectively utilizes global channel and spatial information while possessing excellent local feature fitting capabilities.

[0036] Window-based self-attention computation can be represented as follows:

[0037] in, It is a learnable relative positional encoding; The dimension representing the query / key; The query, key, and value are obtained by linearly mapping the window features obtained from flattening each window.

[0038] As one example, a feature reduction feedforward network is used to further enhance valuable features and suppress less important features in the extracted deep features.

[0039] Input features are used to simplify the deep features of the feedforward network. After sequentially applying layer normalization, partial convolution, and linear mapping, feature redundancy in deep features is removed, as follows:

[0040] in , For output features The number of channels; Standardization; Indicates partial convolution; Represents a linear mapping; It is the GELU activation function; Will Divide the channel into equal parts as follows:

[0041] in Indicates a delimiter operation; and , which are the features after segmentation; right The output features are obtained by sequentially performing reconstruction, depthwise convolution, and flattening. ; Will and Perform multiplication and then apply a linear mapping to the result, as follows:

[0042] in ; Indicates the multiplication operation; Represents a linear mapping; Input features and Add them together to obtain the deep features of the output. .

[0043] Step S33 includes: Shallow and deep features are fused together using a residual connection to obtain fused features; The fused features are fed into the reconstruction module to generate high-resolution, high-quality images; The reconstruction module includes two convolutional layers and a pixel shuffling layer; the pixel shuffling layer is used to upsample the fused features.

[0044] As one example, Figure 2 This paper presents the architecture of a remote sensing image super-resolution method that combines global and local features. The method can be divided into three parts: shallow feature extraction, deep feature extraction, and image reconstruction. The shallow feature extraction stage typically captures image edges, textures, and other basic information, providing preliminary feature representations for subsequent deep feature extraction and reconstruction processes. The deep feature extraction stage uses a deep network structure to analyze the image in depth, capturing higher-level semantic information and detailed features, thereby enhancing image detail and quality, and supporting the final super-resolution reconstruction. The image reconstruction stage combines shallow and high-level features, using appropriate operations to generate a high-resolution, high-quality image with richer details and better visual effects.

[0045] As one example, for a given low-resolution image input H, W and These represent the height, width, and number of channels of the LR image, respectively. First, a convolutional layer is used to extract shallow features. C represents the number of channels in the feature map obtained after shallow feature extraction. Then, a series of Residual Joint Attention Groups (RJAG) and a Feature Refinement Feedforward Network (FRFN) are used to perform deep feature extraction. Then, we extract the shallow features... and deep features The features are fused using a residual connection. Finally, the fused features are fed into the image reconstruction module to generate a high-resolution, high-quality image.

[0046] Figure 3 It is the structure of a Residual Joint Attention Group (RJAG). Each RJAG contains six Joint Attention Blocks (JABs), one Overlapping Cross Attention Block (OCAB), and a convolutional layer with residual connections.

[0047] Figure 4 It is the structure of a joint attention block. As... Figure 4 As shown, the JAB combines a Convolutional Block Attention Module (CBAM) and a Window-based Multi-head Self-attention (W-MSA) mechanism. Specifically, the input feature X passes through the LayerNorm (LN) layer in parallel with both CBAM and W-MSA. The output of CBAM is multiplied by a constant α to avoid potential conflicts between CBAM and W-MSA in optimization and visual representation. Furthermore, shifted window self-attention (SW-MSA) is used at intervals in consecutive JABs. Ultimately, the constructed JAB effectively utilizes global channel and spatial information while possessing excellent local feature fitting capabilities.

[0048] Figure 5 It is a structure of overlapping attention blocks. The purpose of this module is to enhance the interaction between information across windows.

[0049] Figure 6 This is the structure of the convolutional block attention module. As... Figure 6 As shown, input features Fine features are obtained by sequentially processing through the channel attention module and the spatial attention module. .

[0050] Figure 7 It is the structure of a feature-reduced feedforward network. Just as... Figure 7 As shown, input features The feature map is obtained by sequentially applying layer normalization (LN), partial convolution (PConv), and linear mapping. After that, Divide the channel into equal parts Next, regarding The process involves sequentially reshaping, depth-wise convolution (DWConv), and flattening to obtain the final result. .Will and Perform multiplication, and then apply a linear mapping to the result to obtain... Finally, the input features and Add to obtain output features .

[0051] A more specific embodiment of the present invention is as follows: This remote sensing image super-resolution experiment was conducted on a Linux operating system and an NVIDIA GeForce RTX 3090 GPU.

[0052] The experimental data used the AID dataset, which is a remote sensing image dataset containing scene images in 30 categories, with approximately 220–420 images in each category, for a total of 10,000 images, each with a pixel size of approximately 600×600.

[0053] Evaluation Metric: Peak Signal-to-Noise Ratio (PSNR). PSNR is an objective metric used to measure the difference between two images. It is primarily used to evaluate the effectiveness of image compression, transmission, or reconstruction algorithms. A higher PSNR value indicates that the two images are more similar and that there is less quality loss. Its calculation formula is as follows:

[0054]

[0055] in It is the super-resolution image. This represents the target high-resolution image. M is the maximum possible pixel value in the image.

[0056] The Structural Similarity Index Measure (SSIM) is based on a perceptual model of the human visual system (HVS) and is used to measure the similarity between two images in terms of brightness, contrast, and structure. Unlike PSNR, SSIM more closely resembles human visual perception and more accurately reflects image quality. Its calculation formula is as follows:

[0057] in , and These represent similarity in brightness, contrast, and structure, respectively.

[0058] The experimental results, comparing the super-resolution performance of two deep learning methods with the method of this invention on the AID dataset, are shown in Table 1. The deep learning methods include NLSN and SwinIR.

[0059] Table 1. Comparison of super-resolution performance of different methods on the AID dataset

[0060] In Table 1, the bolded values ​​represent the optimal index values. As can be seen from Table 1, the proposed invention method achieves the best results in both PSNR and SSIM on the AID dataset. Specifically, it reaches 31.52 dB in PSNR and 0.8254 dB in SSIM.

[0061] This application also discloses an electronic device. (See reference...) Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0062] The communication bus 502 is used to enable communication between these components.

[0063] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0064] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0065] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned method for super-resolution of remote sensing images combining global and local features.

[0066] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0067] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A remote sensing image super-resolution method combining global and local features, characterized in that, The method comprises the following steps: S1: obtaining an original low-resolution remote sensing image and preprocessing, constructing a data set; S2: constructing a remote sensing image reconstruction network based on Transfomer; the remote sensing image reconstruction network comprises: a feature reduction feedforward network, a convolutional layer, a plurality of sequentially connected residual joint attention groups, and a reconstruction module; The convolutional layer, the plurality of sequentially connected residual joint attention groups, and the reconstruction module are sequentially connected in order; S3: training the remote sensing image reconstruction network through the data set; S4: obtaining a remote sensing image to be reconstructed; inputting the remote sensing image to be reconstructed into the trained remote sensing image reconstruction network to obtain a reconstructed high-resolution image. 2.The method of claim 1, wherein, Step S1 comprises: The original low-resolution remote sensing image data is subjected to data enhancement processing.

3. The method of claim 1, wherein, Step S3 comprises: S31: extracting shallow features of the preprocessed low-resolution remote sensing image; S32: inputting the shallow features into the constructed residual joint attention group to obtain deep features; and removing feature redundancy in the deep features by using the feature reduction feedforward network; S33: inputting the shallow features and the deep features into the constructed reconstruction module to generate a high-resolution and high-quality image.

4. The remote sensing image super-resolution method combining global and local features according to claim 3, wherein, Step S31 comprises: adopting a convolutional layer to extract shallow features of the preprocessed low-resolution remote sensing image where H, W and C represent height, width and channel number of the shallow features respectively.

5. The method of claim 3, wherein, Step S32 comprises: The residual joint attention group comprises: six joint attention blocks JAB, one overlapping cross attention block OCAB, and one convolutional layer with a residual connection; The six joint attention blocks JAB, the one overlapping cross attention block OCAB, and the one convolutional layer with a residual connection are sequentially connected in order; Each joint attention block JAB comprises: a first LayerNorm layer, a convolutional block attention module CBAM, a window-based multi-head self-attention mechanism module W-MSA, a second LayerNorm layer, and a multi-layer perceptron; The entire processing process of the joint attention block JAB is represented as follows: wherein denotes the input features of the joint attention block JAB; denotes a LayerNorm layer; denotes the features output by the LayerNorm layer; and denotes the intermediate features; denotes a convolutional block attention module CBAM; denotes a window-based multi-head self-attention mechanism module W-MSA; MLP denotes a multi-layer perceptron, and Y is the output feature of the joint attention block JAB. Deep features of an input feature reduction feedforward network The feature redundancy in the deep features is removed by sequentially applying layer normalization, partial convolution, and linear mapping as follows: wherein , is the number of output features ; denotes standardization denotes partial convolution represents linear mapping is the GELU activation function will be described below. The passage is equally divided as follows: wherein denotes a split operation; and are the split features; To reconstruct, depthwise convolution, and flatten in sequence to obtain output features ; are multiplied together and the result of the multiplication is linearly mapped as follows: and are multiplied together and the result of the multiplication is linearly mapped as follows: wherein ; denotes a multiplication operation; represents a linear mapping; The input features are added to obtain the deep features of the output .​​ 6. The method of claim 1, wherein, Step S33 comprises: The shallow features and the deep features are fused by a residual connection to obtain fused features; The fused features are sent to the reconstruction module to generate a high-resolution and high-quality image; The reconstruction module comprises: two convolutional layers and one pixel shuffle layer; the pixel shuffle layer is used for up-sampling the fused features.

7. An electronic device, comprising: The electronic device comprises a processor, a memory, a user interface, and a network interface; the memory is used to store instructions; the user interface and the network interface are used to communicate with other devices; and the processor is used to execute the instructions stored in the memory, so that the electronic device executes the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and when the instructions are executed by a computer, the method according to any one of claims 1-6 is executed.

Images