Remote sensing image super-resolution method based on frequency-space collaborative cross attention network

Abstract

The invention discloses a remote sensing image super-division method based on a frequency-space cooperative cross attention network. The method comprises the following steps: constructing a low-resolution and high-resolution paired remote sensing image training sample set; constructing a super-division network model based on frequency-space cooperative cross attention, and performing training optimization; the network model comprises a shallow feature extraction stage, a deep feature extraction stage and a reconstruction stage which are connected in sequence; the deep feature extraction stage comprises a double-domain intensive attention module and a multi-level space cross attention module; in the reconstruction stage, a double-path strategy is adopted, and cross-level information complementation and enhancement are realized in combination with a multi-level channel cross attention module; and inputting a low-resolution image to the trained network model, and reconstructing a high-resolution image. According to the invention, the global context and local details of the remote sensing image can be captured at the same time, and the high-resolution remote sensing image with high visual quality and accurate structure is efficiently generated.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision interdisciplinary technology, specifically involving a deep learning method for super-resolution reconstruction of remote sensing images, and particularly a remote sensing image super-resolution method based on a multi-level cross-attention network structure that integrates spatial and frequency domains. Background Technology

[0002] Image super-resolution technology aims to recover detailed information from high-resolution images by extracting them from low-resolution images. It provides more refined information on the structure, texture, and spatial distribution of surface objects, playing an irreplaceable role in visual tasks. However, limited by factors such as satellite sensors, atmospheric conditions, and data transmission, directly acquired remote sensing images often suffer from insufficient spatial resolution. Algorithm-based image super-resolution reconstruction has become a cost-effective solution.

[0003] Existing super-resolution methods are mainly divided into three categories: interpolation-based methods, model-based reconstruction methods, and learning-based methods. Interpolation-based methods, such as bicubic interpolation and bilinear interpolation, mainly fill pixel values ​​using mathematical interpolation formulas. They have low computational complexity and strong real-time performance, but the reconstruction results suffer from problems such as blurred details and smoothed edges, failing to meet the high-precision interpretation requirements of remote sensing images. Model-based reconstruction methods mainly rely on manually designed prior knowledge to build and optimize models. Their performance is highly dependent on the rationality of the prior assumptions, and their generalization ability is significantly insufficient. With the development of convolutional neural networks, some scholars have improved the reconstruction effect in natural image super-resolution tasks by utilizing the powerful feature extraction and feature representation capabilities of convolutional neural networks.

[0004] However, the strong heterogeneity of texture distribution and large differences in the scale of surface objects in remote sensing images make it difficult for natural image super-resolution models to adapt to remote sensing images, resulting in reconstruction results that are prone to detail loss and color distortion. Although the rapid development of remote sensing imaging technology in recent years has significantly improved image resolution, the demand for high spatiotemporal resolution images in many remote sensing-related applications, such as target detection, fine classification of ground features, and detection of subtle changes, still cannot be fully met. Current remote sensing image super-resolution methods generally rely excessively on shallow features and cannot effectively balance the relationship between global dependency modeling and computational efficiency. Efficiently recovering high-resolution images with clear details and spectral fidelity from low-resolution images remains a challenge.

[0005] In recent years, researchers have attempted to incorporate frequency domain information into super-resolution networks to enhance global modeling capabilities. For example, patent CN118052712A discloses a multi-stage space-frequency joint remote sensing image super-resolution reconstruction method, which utilizes a dual-branch network to extract spatial and frequency domain features separately, and then fuses them through an adaptive space-frequency selection attention network. Patent CN120599489A proposes an image super-resolution reconstruction method based on dual-domain feature fusion and implicit representation, which extracts frequency domain features through Haar wavelet transform and fuses them with spatial domain features through cross-attention. These methods have made some progress in frequency domain utilization, but still have the following shortcomings: 1) The extraction methods for frequency domain features are relatively simple (such as using only wavelet transform), which fails to fully explore the efficiency of Fourier transform in global dependency modeling; 2) Attention mechanisms are mostly computed within features at the same level, lacking cross-level spatial-channel collaborative interaction, resulting in insufficient fusion of shallow details and deep semantic information; 3) There is still room for improvement in balancing model complexity and reconstruction accuracy, making it difficult to deploy in real time in resource-constrained scenarios.

[0006] Mathematically, the degradation problem of high-resolution images can be modeled as follows:

[0007] in This represents the observed low-resolution image. For the corresponding high-resolution image, where and Height and width, The number of feature channels, This represents the convolution operation. Indicates the fuzzy kernel. The multiple is downsampling operation, This represents the noise component.

[0008] Therefore, how to design a remote sensing image super-resolution method that can efficiently capture global dependencies and finely restore local details while balancing computational efficiency and reconstruction quality remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] To address the shortcomings of existing remote sensing image super-resolution techniques, this paper proposes a remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network, which combines deep learning methods. This method can simultaneously capture the global context and local details of remote sensing images, and efficiently generate high-resolution remote sensing images with high visual quality and accurate structure.

[0010] The technical solution provided by this invention is a remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network, comprising: Construct a training sample set of remote sensing images paired with low-resolution and high-resolution images; A frequency-space collaborative cross-attention super-resolution network model is constructed, and the network model is trained and optimized based on a sample set. The network model is an end-to-end deep learning model, and its overall architecture includes a shallow feature extraction stage, a deep feature extraction stage, and a reconstruction stage connected in sequence. The deep feature extraction stage includes a dual-domain dense attention module and a multi-level spatial interaction attention module. The dual-domain dense attention module uses Fourier transform to realize attention calculation. The multi-level spatial cross-attention module has position sensitivity and models local structure and fine texture. The reconstruction stage adopts a dual-path strategy and combines it with the multi-level channel cross-attention module to achieve cross-level information complementarity and enhancement, generating super-resolution results. Input low-resolution images into a trained frequency-space co-current attention super-resolution network model to reconstruct high-resolution images.

[0011] Moreover, the deep feature extraction stage includes multiple stacked dual-domain dense attention modules and a multi-level spatial cross attention module. The shallow features are extracted from the remote sensing image through multiple dual-domain dense attention modules in the frequency domain and spatial domain. After the dimensions are adjusted by splicing and convolution, they are input into the multi-level spatial cross attention module together with the initial shallow features from the shallow feature extraction stage to obtain deep features.

[0012] Furthermore, the dual-domain dense attention module includes a dense connection block and a Fourier attention block. The dense connection block adopts a dense connection strategy, where the input of each layer is a concatenation of the features output by all previous layers along the channel dimension. The result of the last concatenation is then input into the Fourier attention block after pointwise convolution. The Fourier attention block transforms the key and query to the frequency domain using a fast Fourier transform for similarity calculation, and then multiplies the result by the value after inverse Fourier transform and layer normalization to obtain the module output.

[0013] Furthermore, the multi-level spatial cross-attention module receives shallow features from the shallow feature extraction stage and deep features from the stacked output of the dual-domain dense attention module as two inputs. First, layer normalization is performed on the two inputs respectively, and then channel adjustment and spatial feature extraction are performed through convolution. A query tensor is generated from the shallow features, and a key tensor and a value tensor are generated from the deep features. The tensors are reshaped into a sequence dominated by the spatial dimension. The matrix product of the transpose of the key tensor and the query tensor is calculated and processed by an activation function to generate a spatial cross-attention map. The attention map is applied to the value tensor, and after being reshaped back to the original dimension, residual aggregation is performed. Then, feature extraction is performed sequentially through layer normalization, channel compression convolution, SimpleGate nonlinear activation, and channel restoration convolution, and residual concatenation is performed again with the residual aggregated features to output deep features.

[0014] Moreover, the reconstruction stage adopts a dual-path strategy, including an upsampling path and an interpolation path. The upsampling path sequentially passes deep features through convolutional expansion channels, sub-pixel convolution to improve resolution, and convolutional compression channels to obtain upsampled features. The interpolation path performs bilinear interpolation upsampling on shallow features to obtain features consistent with the target resolution. The features from the two paths are input into a multi-level channel cross-attention module for alignment and fusion, and a high-resolution image is output.

[0015] Moreover, the multi-level channel cross-attention module generates a channel attention graph with the channel dimension as the primary dimension.

[0016] Moreover, the multi-level spatial cross-attention module and the multi-level channel cross-attention module take features from different levels as input and generate cross-attention maps of spatial dimension and channel dimension respectively, so as to realize the guidance and modulation of shallow features by deep features and complete the information interaction between features of different granularities and levels.

[0017] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image super-resolution method based on a frequency-space cooperative cross-attention network as described above.

[0018] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image super-resolution method based on a frequency-space cooperative cross-attention network as described above.

[0019] On the other hand, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network as described above.

[0020] In conclusion, this invention proposes a remote sensing image super-resolution method and apparatus based on a frequency-space collaborative cross-attention network. The method combines a dual-domain dense attention module and effectively simulates global dependencies through Fourier transform, thereby enhancing the sensing range while reducing computational complexity. Furthermore, the method employs a hierarchical interaction mechanism, including a hierarchical spatial cross-attention module and a hierarchical channel cross-attention module, effectively combining low-level high-frequency details with high-level features, thus promoting cross-layer feature modulation and adaptively aggregating key information. While maintaining high reconstruction performance, the network model has lower computational cost and faster inference speed, making it suitable for various remote sensing scenarios. Therefore, the efficient and accurate solution provided by this invention can better meet the needs of practical applications such as urban planning, resource and environmental monitoring, and disaster emergency response.

[0021] Specifically, the beneficial effects of the technical solution provided by this invention are as follows: (1) Excellent reconstruction performance: This invention innovatively designs a dual-domain dense attention module and a multi-level spatial / channel cross-attention module, which achieves efficient feature extraction through the transformation between the spatial domain and the frequency domain, and combines the interaction between multi-level features to achieve efficient modeling with global dependencies. It can improve accuracy and visual reconstruction effect in different remote sensing data scenarios, filling the need for high-precision interpretation of remote sensing images; (2) High computational efficiency: By adopting lightweight designs such as frequency domain attention computation and residual connections, the computational load of the model is significantly reduced compared with other deep learning super-resolution methods, and the inference speed is greatly improved. The method proposed in this invention can not only reduce the consumption of hardware resources, but also enable the model to achieve real-time processing in resource-constrained environments, better meeting the needs of real-time deployment; (3) Strong scene adaptability: The network model proposed in this invention has high adaptability and does not rely on assumptions such as self-similarity. Through global-local feature fusion and cross-scale interaction, it adapts to the diverse feature distribution of heterogeneous scenes (such as cities, mountains and water bodies) in remote sensing images. It maintains stable performance under different upsampling ratios (×2, ×4) and has great application potential and value in real-world scenarios. Attached Figure Description

[0022] Figure 1 is a schematic diagram of the overall network architecture according to an embodiment of the present invention.

[0023] Figure 2 is a schematic diagram of the dual-domain dense attention module structure according to an embodiment of the present invention.

[0024] Figure 3 is a schematic diagram of the spatial cross-attention module structure according to an embodiment of the present invention.

[0025] Figure 4This is a performance comparison chart of the method proposed in this invention with other comparative methods on the UCMerced dataset at ×2 and ×4 ratios. Detailed Implementation

[0026] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0027] To address the problems of over-reliance on shallow features, difficulty in balancing global dependency modeling and computational efficiency, and insufficient detail recovery in heterogeneous scenes in existing remote sensing image super-resolution reconstruction methods, this invention proposes a remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network. This method achieves efficient and high-fidelity reconstruction of remote sensing images through the collaborative work of the frequency and spatial domains and the cross-interaction of multi-level features.

[0028] Example 1 The remote sensing image super-resolution method proposed in this embodiment of the invention mainly includes the following steps: (1) Construct a training sample set of low-resolution-high-resolution paired remote sensing images; in practice, a series of preprocessing, division and enhancement of the acquired high-resolution remote sensing image data are carried out to generate the samples required for subsequent steps.

[0029] (2) Construct a frequency-space collaborative cross-attention super-resolution network model, and train and optimize the network model based on the sample set; the network model is an end-to-end deep learning model, and its overall architecture includes a shallow feature extraction stage, a deep feature extraction stage and a reconstruction stage connected in sequence; The shallow feature extraction stage is composed of cascaded convolutional layers; The deep feature extraction stage mainly comprises several stacked dual-domain dense attention modules and a multi-level spatial cross-attention module. The dual-domain dense attention modules utilize Fourier transform to perform attention calculations, achieving global modeling while reducing computational complexity. The multi-level spatial cross-attention module is position-sensitive, modeling local structures and fine textures. Located at the end of the deep feature extraction stage, it is used to fuse shallow and deep features. Specifically, shallow features are extracted from the remote sensing image through multiple dual-domain dense attention modules, interactively extracting complex texture and structural information in the frequency and spatial domains. After being stitched together and convolutionally adjusted for dimensionality, these features are input together with the initial shallow low-level features into the multi-level spatial cross-attention module to obtain deep high-level features.

[0030] The reconstruction stage is primarily guided by a multi-level channel cross-attention module. By guiding and modulating shallow, low-level features with deep, high-level features, it achieves cross-level information complementarity and enhancement, ultimately reconstructing a high-resolution result. The multi-level channel cross-attention module selectively aggregates key channel information from multi-level features, enriching the channel representation. The reconstruction stage employs a dual-path strategy, combined with the multi-level channel cross-attention module, to achieve cross-level information complementarity and enhancement, generating a super-resolution result.

[0031] (3) Input the low-resolution image into the trained frequency-space collaborative cross-attention super-resolution network model to reconstruct the high-resolution image.

[0032] By using the trained network model to perform super-resolution on the low-resolution remote sensing images to be processed, high-resolution remote sensing images that are close to reality can be generated.

[0033] Example 2 The frequency-space collaborative cross-attention super-resolution method provided in this embodiment of the invention further preferably adopts the following implementation method: I. Constructing a training sample set of low-resolution-high-resolution paired remote sensing images As a preferred embodiment, the sample library construction of the embodiment specifically includes the following sub-steps: (1) First, regarding the acquisition of remote sensing images, this invention emphasizes selecting datasets containing rich ground features during the acquisition process to ensure the diversity of the model in subsequent training.

[0034] In deep learning-driven image restoration tasks, high-quality datasets are the cornerstone of model performance. This invention employs three widely used benchmark datasets in remote sensing: UCMerced, RSSCN7, and AID, covering a variety of land cover types from urban to natural landscapes. The UCMerced dataset primarily targets land use scene classification tasks in urban areas, containing twenty-one typical land cover scene categories, such as farmland, baseball fields, beaches, buildings, and forests. Each image has a fixed pixel size and specific spatial resolution. For scientific training and evaluation, this embodiment divides the dataset into three subsets: 945 images for model training, 105 images for validation, and 1050 images for testing. The RSSCN7 dataset covers seven typical scene categories, characterized by its extremely high scene diversity. The sampled images span different seasons, varying weather conditions, and different sampling ratios, providing excellent material for examining the model's robustness. The AID dataset is a large-scale benchmark dataset for aviation scene classification, covering thirty scene categories. The image size is fixed at 600×600 pixels, but its spatial resolution varies greatly, ranging from 0.3 meters to 8 meters. This requires the model to have the ability to process multi-scale features.

[0035] (2) After acquiring these images, preprocessing and sample generation are performed. In this invention, the original high-resolution image is used as the ground truth. In order to simulate the degradation in the remote sensing imaging process and generate the corresponding low-resolution image, this embodiment uses bicubic interpolation to downsample the high-resolution image and generate the corresponding low-resolution image, thereby constructing a pair of training samples, and setting a variety of different downsampling ratios according to experimental requirements.

[0036] (3) After preprocessing and sample generation, data augmentation is performed on image patches during training to improve the robustness of the model and prevent overfitting. By applying affine transformations (including scaling, rotation, and translation), hue adjustments (such as brightness and contrast changes), random cropping, and flipping, the diversity of sample data can be significantly expanded. These augmentation methods effectively increase the model's robustness to different environments, lighting conditions, and viewpoint changes, and help reduce overfitting during training, thereby improving the overall model performance and generalization ability. Specific operations include random horizontal flipping, vertical flipping, and... Random rotation.

[0037] (4) Finally, to ensure the dataset can play a full role in the training process, it is divided into a training set, a validation set, and a test set according to actual training needs. In the subsequent network model training and optimization phase, the training set is used for model learning and parameter optimization, the validation set is used for parameter tuning and performance evaluation during model training, and the test set is used to evaluate the final performance of the model after training. This division method can ensure the model's stable performance on different data and effectively avoid overfitting and underfitting problems, providing solid data support for the training and evaluation of the network model. This invention first constructs a high-quality training dataset to ensure that the model learns rich texture and structural features. The dataset is divided into a training set, a validation set, and a test set according to common strategies (e.g., UCMerced is divided into 945 / 105 / 1050).

[0038] II. Constructing a Super-Resolution Network Model Based on Frequency-Space Collaborative Cross-Attention The network model constructed in this invention is as follows: Figure 1 As shown, the overall architecture is divided into three stages: shallow feature extraction stage, deep feature extraction stage, and image reconstruction stage.

[0039] The deep feature extraction stage includes a dual-domain dense attention module and a spatial interaction attention module. The dual-domain dense attention module uses Fourier transform to perform attention calculation, which reduces computational complexity while providing global modeling. The multi-level spatial cross attention module is position-sensitive and models local structures and fine textures. The multi-level channel cross attention module selectively aggregates key channel information from multi-level features to enrich channel representation. The image reconstruction stage employs a dual-path strategy and combines it with a multi-level channel cross-attention module to achieve cross-level information complementarity and enhancement, generating super-resolution results.

[0040] (1) Shallow feature extraction stage This stage, located at the beginning of the network, is primarily responsible for mapping the input low-resolution image from the image space to the feature space and extracting preliminary shallow features. This stage preferably uses two sequentially configured 3×3 convolutional layers as the mapping function. After processing, the input image outputs a shallow feature representation, with the shallow feature dimension being... (in For the set of real numbers, and Height and width, These shallow features (which are the number of feature channels) not only serve as inputs for the subsequent deep feature extraction stage, but are also directly passed to the final image reconstruction stage of the network through long skip connections, providing the original structural information and low-frequency components for the final image reconstruction.

[0041] (2) Deep feature extraction stage This stage is the core of the invention, aiming to extract and optimize deep features through the combination of frequency and spatial domains and multi-level interactions. To overcome the limitations of the receptive field in traditional convolutional neural networks and the excessive computational complexity of traditional self-attention mechanisms, this stage mainly includes... It consists of a stacked dual-domain dense attention module and a multi-level spatial cross attention module. The number of dual-domain dense attention modules significantly affects the model performance, from the set { Based on experiments and considering both model complexity and performance, it is recommended that the optimal number of dual-domain dense attention modules in the model be set to [value missing]. .

[0042] Each dual-domain dense attention module consists of a dense connection block and a Fourier attention block. The dense connection block contains several repeating building blocks consisting of 3×3 convolutions and ReLU activation functions (5-10 repeating building blocks are recommended). Each building block is concatenated along the channel dimension, and the output of the last concatenation layer is passed through a 1×1 pointwise convolution before entering the Fourier attention block. In the Fourier attention block, the key (K) and query (Q) undergo Fast Fourier Transform to calculate similarity in the frequency domain, and then return to the spatial domain through Inverse Fourier Transform and layer normalization. Finally, the sum and value (V) are multiplied to obtain the output of the dual-domain dense attention module.

[0043] The multi-level spatial cross-attention module takes the output of the dual-domain dense attention module (processed with 1×1 and 3×3 convolutions) and the output of the first 3×3 convolution in the shallow feature extraction stage as two inputs. These two inputs are independently normalized using LN to ensure stability, followed by channel adjustment through a 1×1 convolutional layer, and then spatial feature extraction through a 3×3 convolutional layer. The tensor is then converted into a token through a reshaping operation. Next, the ReLU activation function and the Softmax function are applied to the product of the previous two matrices to calculate the cross-attention map. The attention map is then multiplied by the values. The resulting size is reshaped. Finally, layer normalization, two 1×1 convolutional layers, and the SimpleGate nonlinear activation function are used to perform residual aggregation on the aforementioned output and the two inputs of this module to obtain the final output of the deep feature extraction stage.

[0044] In the deep feature extraction stage, on the one hand, a dual-domain dense attention module is used to capture global dependencies and to approximate the operation by transforming to the frequency domain, which effectively avoids a large number of matrix multiplications, reduces the computational cost of the model, and improves computational efficiency; on the other hand, a multi-level spatial cross attention module is used to achieve the fusion between features from shallow layers and features from deeper layers.

[0045] The suggested implementation method is as follows: A. Dual-domain dense attention module The dual-domain dense attention module primarily performs attention calculations through Fourier transform. This module not only reduces the computational complexity of spatial domain operations but also enhances global modeling capabilities through efficient cross-feature interactions. Traditional convolutional layers are limited by local receptive fields, making it difficult to capture long-distance dependencies. While attention mechanisms can capture global context, the computational complexity of calculating the correlation between pixel pairs in the spatial domain is typically proportional to the square of the image size, leading to excessive computational demands for high-resolution remote sensing images. Fourier transform provides an effective solution: by transforming the signal from the spatial domain to the frequency domain, global dependencies can be approximated through element-wise operations in the frequency domain, thus avoiding a large number of matrix multiplications.

[0046] The dual-domain dense attention module described in this embodiment of the invention is as follows: Figure 2 As shown, specifically, each dual-domain dense attention module consists of two basic building blocks: a dense connection block and a Fourier attention block.

[0047] 1) Dense Connection Blocks: A dense connection strategy is employed to promote feature reuse and enrich feature representations. As a preferred embodiment, the processing of dense blocks is as follows: Within this module, the input to each layer is a concatenation of the features output from all previous layers along the channel dimension. Specifically, the input to the first layer is the concatenation of the features output from all previous layers along the channel dimension. In the first dual-domain dense attention module, the first The convolutional layer (the first) The repeating building block includes the first Output feature map of (each convolutional layer and its corresponding ReLU activation function) have The first channel. Therefore, input to the first... The number of feature channels in each convolutional layer is ,in This is the initial number of channels in the module's input. This linearly increasing channel dimension design allows the network to progressively accumulate and refine feature representations. The output of the last layer of the repeated construction unit is concatenated along the channel dimension. The input to the Fourier attention block is generated after convolutional fusion, and it is recommended to set the number of densely connected blocks to 5-10.

[0048] 2) Fourier Attention Block: Features are mapped to the frequency domain using Fourier transform for processing. As a preferred embodiment, the processing procedure for the Fourier attention block is as follows: Step A1 (Feature Projection): Input Features After processing with densely connected blocks, the query is first generated through combined convolutional projection. ,key( Sum of values ​​( ) indicates that all dimensions are .

[0049] Step A2 (Frequency Domain Transformation): Use Fast Fourier Transform to transform... , Transform to the frequency domain to obtain , .

[0050] Step A3 (Frequency Domain Attention Calculation): Using the convolution theorem (spatial domain convolution equals frequency domain multiplication), in the frequency domain, by... , The frequency domain attention map is directly calculated by multiplying the vectors element by element. This step greatly reduces computational complexity.

[0051] Step A4 (Inverse Transform and Modulation): The frequency domain attention map is transformed back to the spatial domain using the inverse fast Fourier transform to obtain the spatial attention map. Then, layer normalization is applied to the obtained spatial attention map and compared with the value ( ) Perform matrix multiplication (vector cross product) on the features.

[0052] Step A5 (Residual Output): Finally, the modulated features are convolved and combined with the input features. The residuals from the outputs of the first repeated building unit are summed to obtain the output. .

[0053] B. Multi-level Spatial Cross-Attention Module The multi-level spatial cross-attention module is position-sensitive and adept at capturing spatial details, such as the specific location and shape of objects, thus effectively modeling local structures and fine-grained textures. It is used to aggregate key information from multi-level features to reconstruct spatial details. The multi-level spatial cross-attention module receives two inputs: "coarse-grained" shallow features from the shallow feature extraction stage and "fine-grained" deep features from the stacked output of the dual-domain dense attention module. This module first performs layer normalization and convolution on the input shallow and deep features, generating query tensors and key and value tensors, respectively. To capture spatial details, such as the location and shape of objects, the module is position-sensitive, reshaping the tensors into a sequence dominated by spatial dimensions. Next, it calculates the matrix product of the transpose of the key tensor and the query tensor, and then processes it through an activation function to generate a spatial cross-attention map, allowing deep features to guide the recovery of spatial details in shallow features. Finally, the attention map is applied to the value tensor, and residual aggregation is performed with the original input. Features are further refined through a non-linear activation mechanism before output.

[0054] The multi-level spatial cross-attention module described in this embodiment of the invention is as follows: Figure 3 As shown in the preferred embodiment, the processing procedure for multi-level spatial intersection attention blocks is as follows: Step B1 (Feature Embedding): First, the original input, including the "coarse-grained" shallow features from the shallow feature extraction stage and the "fine-grained" deep features from the stacked output of the dual-domain dense attention modules, is subjected to layer normalization. Then, channel adjustment and feature extraction are performed sequentially through two convolutions (1×1 convolution and 3×3 convolution). A query tensor is generated from the "coarse-grained" shallow features. Generate bond tensors from "fine-grained" deep features Sum tensor .

[0055] Step B2 (Spatial Reshaping): To capture spatial details (such as the position and shape of objects), the multi-level spatial cross-attention module is position-sensitive. The tensor is reshaped (reconstructed) into a token sequence primarily based on the spatial dimension: .

[0056] Step B3 (Spatial Cross-Attention Calculation): Calculate transpose and The matrix product, divided by the temperature coefficient And after ReLU and Softmax activation, generate Spatial Cross Attention Graph of Size Att spa This allows deep features to guide the recovery of spatial details from shallow features.

[0057] Step B4 (Fusion and Output): The specific implementation is as follows, 1) Attention Weighting and Reshaping: Improving the Spatial Cross-Attention Map Att spa Acting on We obtain the weighted features, then reshape them back to the original dimensions, and combine them with the output of the dual-domain dense attention module (i.e. the original input of the deep feature branch) after 1×1 and 3×3 convolutions to obtain the attention-weighted features.

[0058] 2) First residual fusion: The attention-weighted features obtained in 1) and the output of the first 3×3 convolution in the shallow feature extraction stage (i.e. the original input of the shallow feature branch) are subjected to residual aggregation to realize the guidance and enhancement of shallow details by deep semantic information.

[0059] 3) Feature Extraction: Subsequently, the fused features are sequentially processed through layer normalization, 1×1 convolution for channel compression and fusion, and SimpleGate non-linear activation (which splits the features in the channel dimension and multiplies them, such as...). Figure 3 (abbreviated as SG) and 1×1 convolution recover the number of channels to further refine features.

[0060] 4) Second residual connection: The features output in step 3) are summed element-wise with the features fused in step 2) to obtain the final multi-level spatial cross-attention module output, which is the final output of the deep feature extraction stage. Output deep features.

[0061] C. Image Reconstruction Stage The reconstruction stage employs a dual-path strategy, combined with a multi-level channel cross-attention module to achieve final image synthesis. This module uses the channel dimension as the primary dimension for feature reshaping to enhance detail recovery capabilities. By recalibrating the feature channels, it adaptively enhances information-rich channels and suppresses redundant channels. Finally, the features from the two paths are aligned and fused within this module to output the super-resolution result. The multi-level channel cross-attention module performs channel-level recalibration, selectively emphasizing information channels to suppress redundancy, and aggregating necessary information from multi-level features to enrich the channel representation.

[0062] In this embodiment, this stage maps deep features back to the high-resolution space, employing a dual-path reconstruction strategy that combines upsampling, bilinear interpolation, and multi-level channel cross-attention modules to generate the final super-resolution image. As a preferred embodiment, the dual-path processing in the image reconstruction stage is as follows: Upsampling path (deep features): Deep features first expand the channel dimension through convolutional layers, and then rearrange the features through sub-pixel convolution (pixel shuffle), increasing the spatial resolution from... Upgraded to (in (as a multiple), while the number of channels is restored to Then, it is compressed again through a convolutional layer to obtain 3-channel upsampled features.

[0063] Interpolation path (shallow features): In order to preserve the basic geometric structure of the original image, the shallow features are directly upsampled by bilinear interpolation to generate features consistent with the target resolution.

[0064] The features of the two paths are input into a multi-level channel cross-attention module. This module treats the channel dimension as the main dimension and reshapes it to further enhance detail recovery. Therefore, the implementation of the multi-level channel cross-attention module is consistent with the implementation of the multi-level spatial cross-attention module (see [link]). Figure 3 The process is similar to steps B1-B4, except that the multi-level channel cross-attention module generates a channel attention map. This design allows the model to recalibrate the channel dimensions, selectively emphasizing information-rich channels and suppressing redundant channels. Finally, the two feature paths are aligned and fused in this module to output the final high-resolution image. The specific implementation is as follows: C. Multi-level channel cross-attention module This module receives two inputs from the dual paths: shallow features from the interpolation path (denoted as...). ) and deep features of the upsampling path (denoted as The specific processing procedure is as follows: Step C1 (Feature Embedding): First, for and Perform layer normalization separately. Then, through... Convolution performs channel adjustment. This is based on interpolation path features. Generate query tensor From the upsampling path features Generate key tensors Sum tensor .

[0065] Step C2 (Channel Reshaping): To capture the dependencies between channels, the tensor is reshaped into a token sequence dominated by the channel dimension. All spatial locations of each channel are flattened into a vector, resulting in... .

[0066] Step C3 (Channel Cross-Attention Calculation): Calculate transpose and The matrix product, divided by the temperature coefficient And after being activated by Softmax, it generates Channel cross attention map of size This attention map characterizes the correlation between different channels, enabling the deep features of the upsampling path to guide the shallow features of the interpolation path for channel recalibration.

[0067] Step C4 (Fusion and Output): 1) Attention Weighting and Reshaping: Transforming the Channel Cross-Attention Map Acting on value tensor (Perform matrix multiplication based on dimensionality compatibility) to obtain the weighted features, and then reshape them back to the original spatial dimensions. To obtain attention-weighted features .

[0068] 2) First residual fusion: Original input features of the interpolation path and the original input features of the upsampling path Perform element-wise summation to achieve cross-path information complementarity.

[0069] 3) Feature extraction: The fused features are sequentially processed through layer normalization, ... Convolution for channel compression and fusion, SimpleGate non-linear activation, Convolution restores the number of channels, further refining features.

[0070] 4) Second residual connection: The refined features are summed element-wise with the features after the first residual fusion to obtain the final output of the multi-level channel cross-attention module, which is the fused feature used for subsequent image reconstruction.

[0071] This module interacts with the multi-level spatial cross-attention module ( Figure 3 The implementation processes are completely identical, with the core difference being: the former reshapes and generates channel attention maps primarily based on the channel dimension, achieving channel dimension recalibration; the latter generates spatial attention maps primarily based on the spatial dimension, achieving spatial detail guidance and enhancement. Together, they constitute a frequency-space collaborative cross-level feature interaction mechanism.

[0072] III. Inputting Remote Sensing Imagery for Super-Resolution Reconstruction A frequency-space collaborative cross-attention super-resolution network model, trained to its full potential, can be used to process input low-resolution remote sensing images. The multi-level spatial cross-attention module and multi-level channel cross-attention module more effectively promote cross-level feature fusion. Unlike traditional self-attention mechanisms that only calculate correlations within a single feature map, this invention uses features from different depths (i.e., different levels) as input and generates cross-attention maps in both spatial and channel dimensions. This allows deep features to guide and modulate shallow features containing details, thereby achieving more comprehensive and adaptive information interaction between features of different granularities and levels.

[0073] The preferred implementation method of this invention further includes the following model optimization process: In the network model training and optimization phase, this embodiment adopts an end-to-end learning approach. In deep learning networks, backpropagation of the loss function is used to optimize network weights. However, as the complexity of the network structure increases, the gradient vanishing problem may occur during parameter updates, leading to unstable optimization. To reduce the tendency of overly smooth reconstruction results and promote the restoration of sharp edges and rich textures, this invention selects the L1 norm loss function to optimize network parameters. The loss function calculates the mean absolute error between the super-resolution output generated by the model and the corresponding high-resolution ground truth image, and drives the update of network parameters by minimizing this error. In practice, an optimizer is used to optimize the network, and appropriate momentum parameters and numerical stability constants are set. An initial learning rate is also set and decayed in the later stages of training to ensure that the model converges to its optimal state. Finally, the trained network model is used to process the low-resolution remote sensing image to generate high-precision super-resolution reconstruction results.

[0074] As a preferred embodiment, the loss function of the embodiment is specifically constructed and implemented as follows:

[0075] in The total number of images in the training batch. The super-resolution output generated by the model based on the ηth LR input. This is the corresponding HR ground truth image.

[0076] To evaluate the effectiveness of the method of the present invention, the embodiments were conducted at two different magnifications ( The experiment was conducted under these conditions. For For this high-magnification super-resolution task, the example employs a two-stage cascaded upsampling strategy (i.e., two consecutive upsampling operations) during the image reconstruction stage. Upsampling is used to gradually restore details.

[0077] The specific configuration of the network model is as follows: The number of dual-domain dense attention modules is N = 16.

[0078] The growth rate D = 64 for each dual-domain dense attention module.

[0079] The number of input feature channels is set to 64.

[0080] The specific hyperparameter settings for the training process are as follows: Optimizer: Use the Adam optimizer, with parameters set to... = 0.9, the numerical stability constant is .

[0081] Learning rate: Initial learning rate .

[0082] The model was trained for a total of 500 epochs. After the 400th epoch, the learning rate decayed. .

[0083] In practice, the input is a low-resolution remote sensing image to be processed. The model first extracts basic features through a shallow feature extraction stage. Then, the features enter a deep feature extraction stage, where global context modeling is performed alternately in the frequency and spatial domains. Spatial details are corrected using deep semantic information through multi-level spatial cross-attention blocks. Finally, in the reconstruction stage, the deep high-level features are upsampled and convolutionally processed, and then input together with the shallow low-level features upsampled by bilinear interpolation into a multi-level channel cross-attention module to generate a high-resolution remote sensing image with clear details and accurate structure.

[0084] Experimental Results and Analysis To verify the effectiveness of the proposed model, three metrics were used for evaluation: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and Spectral Angle Mapper (SAM). PSNR serves as an objective measure of pixel-level fidelity between the ground truth image and the super-resolution network's output image; a higher PSNR value indicates better reconstructed image quality. SSIM provides a perceptual assessment of image quality; a value closer to 1 indicates better structure preservation. SAM is a key metric for evaluating spectral fidelity; a lower SAM value indicates less spectral distortion.

[0085] The four methods compared in this result are as follows: The FSRCNN method refers to the paper published by Dong et al., "C. Dong, CC Loy, and X. Tang, 'Accelerating the super-resolution convolutional neural network,'" Computer Vision – ECCV 2016 (B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), vol.9906, pp. 391–407, Cham: Springer International Publishing, 2016. "A classic accelerated super-resolution convolutional neural network for natural image processing; The DCM method refers to the paper published by Haut et al., "JM Haut, ME Paoletti, R. Fernandez-Beltran, J. Plaza, A. Plaza, and J. Li, “Remote sensing single-image superresolution based on a deep compendium model,” IEEE Geosci. Remote Sensing Lett. A network designed in , vol. 16, pp. 1432–1436, Sept. 2019, utilizes multiple strategies such as skip connections and residual connections to avoid degradation problems, specifically for the super-resolution problem of remote sensing images; The FeNet method refers to the paper published by Wang et al., "Z. Wang, L. Li, Y. Xue, C. Jiang, J. Wang, K. Sun, and H. Ma, “FeNet: Feature enhancement network for lightweight remote-sensing image super-resolution”. IEEE Trans. Geosci. Remote Sensing The method used in this paper, vol. 60, pp. 1–12, 2022, is a lightweight remote sensing image super-resolution reconstruction method with few parameters. The TransENet method refers to the paper published by Lei et al., "S. Lei, Z. Shi, and W. Mo, 'Transformer-based multistage enhancement for remote sensing image super-resolution'". IEEE Trans. Geosci. Remote Sensing A method for super-resolution reconstruction of remote sensing images using a multi-level enhancement architecture incorporating Transformer, vol. 60, pp. 1–11, 2022.

[0086] Table 1 shows the objective evaluation results of this invention and the four super-resolution methods mentioned above on three evaluation indicators: Table 1. Comparison of objective evaluations of the present invention and the comparative method on UCMerced, RSCCN7, and AID dataset ×2.

[0087] The comparison results of this invention with the above four super-resolution methods on the UCMerced dataset at two different magnifications, ×2 and ×4, are as follows: Figure 4 As shown. Furthermore, Figure 4 The visual comparison between the present invention and the comparison method is shown on the UCMerced dataset at ×2 and ×4 magnification. It can be clearly seen from the figure that the image reconstructed by the method of the present invention has clearer texture and more accurate structure.

[0088] From Table 1 and Figure 4As can be seen, the network model of this invention outperforms the other four comparative methods in PSNR, SSIM, and SAM metrics, and can recover detailed information from high-resolution images, providing more refined features of ground structure, texture, and spatial distribution, thus demonstrating broad and crucial application prospects in the field of remote sensing. Specifically, by enhancing the clarity and information richness of images, the method can directly improve the detection and recognition capabilities of small targets (such as vehicles and ships), and provides key discrimination criteria for fine classification of ground features (such as distinguishing crop types and forest tree species). Simultaneously, the recovered details can keenly reveal subtle changes on the land surface (such as minor building expansions and early vegetation degradation), significantly improving the accuracy and timeliness of change detection. Furthermore, clear texture information helps improve the accuracy of 3D reconstruction and terrain analysis, and enhances the automatic extraction capability of linear features such as roads and water systems. Therefore, the method of this invention not only empowers core tasks such as target detection, fine classification, and change detection, but also provides an indispensable data foundation and decision support for a series of fields highly dependent on spatial details, such as urban planning, precision agriculture, environmental monitoring, disaster assessment, and even national security.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0090] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0091] The apparatus provided by the present invention is described below. The apparatus described below can be referred to in correspondence with the remote sensing image super-resolution method based on frequency-space cooperative cross-attention network described above.

[0092] In another embodiment, the present invention provides an electronic device that may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network.

[0093] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] In another embodiment, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the software processing part of the remote sensing image super-resolution method based on frequency-space cooperative cross-attention network provided by the above methods.

[0095] In another embodiment, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the software processing portion of the remote sensing image super-resolution method based on frequency-space cooperative cross-attention network provided by the above methods.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network, comprising: Construct a training sample set of remote sensing images paired with low-resolution and high-resolution images; A frequency-space collaborative cross-attention super-resolution network model is constructed, and the network model is trained and optimized based on a sample set. The network model is an end-to-end deep learning model, and its overall architecture includes a shallow feature extraction stage, a deep feature extraction stage, and a reconstruction stage connected in sequence. The deep feature extraction stage includes a dual-domain dense attention module and a multi-level spatial interaction attention module. The dual-domain dense attention module uses Fourier transform to realize attention calculation. The multi-level spatial cross-attention module has position sensitivity and models local structure and fine texture. The reconstruction stage adopts a dual-path strategy and combines it with the multi-level channel cross-attention module to achieve cross-level information complementarity and enhancement, generating super-resolution results. Input low-resolution images into a trained frequency-space co-current attention super-resolution network model to reconstruct high-resolution images.

2. The remote sensing image super-resolution method based on frequency-space collaborative cross-attention network according to claim 1, characterized in that: The deep feature extraction stage includes multiple stacked dual-domain dense attention modules and a multi-level spatial cross-attention module. The shallow features are extracted from the remote sensing image by multiple dual-domain dense attention modules in the frequency domain and spatial domain. After being stitched and convolved to adjust the dimensions, they are input into the multi-level spatial cross-attention module together with the initial shallow features from the shallow feature extraction stage to obtain deep features.

3. The remote sensing image super-resolution method based on frequency-space collaborative cross-attention network according to claim 1, characterized in that: The dual-domain dense attention module includes a dense connection block and a Fourier attention block. The dense connection block adopts a dense connection strategy, where the input of each layer is a concatenation of the features output by all previous layers along the channel dimension. The result of the last concatenation is then input into the Fourier attention block after pointwise convolution. The Fourier attention block transforms the key and query to the frequency domain using Fast Fourier Transform for similarity calculation, and then multiplies the result by the value after inverse Fourier Transform and layer normalization to obtain the module output.

4. The remote sensing image super-resolution method based on frequency-space collaborative cross-attention network according to claim 1, characterized in that: The multi-level spatial cross-attention module receives shallow features from the shallow feature extraction stage and deep features from the stacked output of the dual-domain dense attention module as two inputs. First, layer normalization is performed on both inputs, followed by channel adjustment and spatial feature extraction via convolution. A query tensor is generated from the shallow features, and key and value tensors are generated from the deep features. The tensors are reshaped into sequences dominated by the spatial dimension. The matrix product of the transpose of the key tensor and the query tensor is calculated and processed by an activation function to generate a spatial cross-attention map. The attention map is applied to the value tensor, which is then reshaped back to its original dimensions and subjected to residual aggregation. Feature extraction is then performed sequentially through layer normalization, channel compression convolution, SimpleGate nonlinear activation, and channel restoration convolution, and residual concatenation is performed again with the features after residual aggregation to output deep features.

5. The remote sensing image super-resolution method based on frequency-space collaborative cross-attention network according to claim 1, characterized in that: The reconstruction stage employs a dual-path strategy, including an upsampling path and an interpolation path. The upsampling path sequentially expands the deep features through convolution, increases the resolution through sub-pixel convolution, and compresses the channels through convolution to obtain upsampled features. The interpolation path performs bilinear interpolation upsampling on the shallow features to obtain features consistent with the target resolution. The features from the two paths are input into a multi-level channel cross-attention module for alignment and fusion, outputting a high-resolution image.

6. The remote sensing image super-resolution method based on frequency-space collaborative cross-attention network according to claim 1, characterized in that: The multi-level channel cross-attention module generates a channel attention graph with the channel dimension as the primary dimension.

7. The remote sensing image super-resolution method based on a frequency-space collaborative cross-attention network according to claim 1, characterized in that: The multi-level spatial cross-attention module and the multi-level channel cross-attention module take features from different levels as input and generate cross-attention maps in spatial and channel dimensions respectively, so as to guide and modulate shallow features with deep features and complete information interaction between features of different granularities and levels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the remote sensing image super-resolution method based on frequency-space collaborative cross-attention network as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the remote sensing image super-resolution method based on frequency-space collaborative cross-attention network as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the remote sensing image super-resolution method based on frequency-space collaborative cross-attention network as described in any one of claims 1 to 7.

Images