A multi-scale frequency-aware lightweight remote sensing image super-resolution method

Abstract

This invention discloses a lightweight remote sensing image super-resolution method with multi-scale frequency awareness, belonging to the field of computer vision and remote sensing image processing technology. Addressing the problems of high-frequency detail attenuation and large model parameter count in existing CNN-Transformer serial hybrid architectures, which are difficult to deploy on edge devices, this invention constructs a three-branch parallel decoupled feature extraction architecture: the local spatial branch captures fine textures based on improved channel attention; the global context branch extracts macroscopic structural dependencies using a lightweight Swing Transformer; and the high-frequency branch specifically extracts and enhances high-frequency edge features based on the principle of unsharpened masks. The outputs of the three branches are dynamically weighted and fused through a zero-initialization adaptive spatial gating fusion mechanism to prevent early branch collapse. This invention also employs depthwise separable convolution to achieve a lightweight design.

Description

Technical Field

[0001] This invention relates to the field of computer vision and remote sensing image processing technology, specifically to a lightweight remote sensing image super-resolution method with multi-scale frequency sensing. Background Technology

[0002] Remote sensing image super-resolution reconstruction technology aims to restore high-resolution spatial information from low-resolution observation images. It is a key technological path to alleviate the physical limitations of sensors and enhance the application value of remote sensing data, and has important application needs in fields such as land resource surveys, refined urban management, dynamic environmental monitoring, and national defense security. In recent years, with the development of deep learning technology, image super-resolution methods based on convolutional neural networks and Transformers have made significant progress.

[0003] CNN-based super-resolution methods, such as EDSR and RCAN, learn end-to-end nonlinear mappings from low to high resolution by constructing deep convolutional networks and introduce channel attention mechanisms to adaptively recalibrate channel feature responses. However, the local receptive field characteristics of convolution operators make it difficult to effectively model the long-range spatial dependencies that are widespread in remote sensing scenes, resulting in poor global structural consistency of reconstructed images. Furthermore, deep CNN networks have a large number of parameters, making it difficult to meet the lightweight deployment requirements of edge devices.

[0004] To overcome the locality limitation of CNNs, Transformer-based super-resolution schemes employ self-attention mechanisms. These mechanisms model stronger contextual information by calculating the global correlations between image patches, and schemes such as SwinIR have demonstrated the potential of Transformers in super-resolution tasks. However, the computational complexity of self-attention mechanisms increases quadratically with image size. For high-resolution remote sensing images containing millions of pixels, the computational overhead and memory consumption increase significantly, severely limiting their deployment efficiency on edge computing platforms such as spaceborne and drone-borne systems.

[0005] To balance global modeling capabilities and computational efficiency, hybrid CNN-Transformer architectures have emerged in recent years, combining two types of operators through serial cascading. Representative works such as DCTA employ CNNs to extract local information, then input these features into Transformer modules to capture global dependencies. However, the inventors of this application have discovered inherent flaws in such serial hybrid architectures: the high-frequency detail features extracted by the CNN branches gradually decay as they pass through subsequent Transformer self-attention layers due to the self-attention mechanism's tendency to aggregate low-frequency smoothing components. This leads to overly smooth features and blurred edges in the final reconstructed image. Furthermore, in the serial architecture, the output of the preceding module directly determines the input of the following module; insufficient feature representation capability in any module will result in an error accumulation effect.

[0006] To address the aforementioned issues, some studies have explored parallel hybrid architecture schemes. For example, related technologies have disclosed lightweight remote sensing super-resolution methods that employ a dual-branch parallel structure to extract local and global features separately. The local branch extracts local context based on CNN cross-attention, while the global branch extracts global dependencies based on the Swin Transformer. Aggregate weights are assigned to the two feature paths through a dynamic weight generation branch. Another scheme uses a three-branch parallel residual block group to extract features at different spatial scales and performs multi-scale fusion through a channel attention mechanism. However, the branches in these parallel schemes are all divided from the spatial or scale dimensions, without addressing feature decoupling in the frequency dimension. The dual-branch architecture's approach of allocating CNN to high-frequency details and Transformer to low-frequency information, while involving frequency processing, is not a dedicated high-frequency protection mechanism. Furthermore, in existing three-branch schemes, the three branches are homogeneous structures, failing to functionally decouple local space, global context, and high-frequency edges, thus failing to fundamentally solve the root cause problem of high-frequency attenuation in serial architectures.

[0007] Furthermore, high-frequency information such as the edges of ground features and the boundaries of tree canopies in remote sensing images is crucial for reconstruction quality. However, existing lightweight super-resolution models are mostly designed for natural images and do not fully consider the characteristics of high spatial density and rich high-frequency boundaries in remote sensing images, resulting in poor texture reconstruction when transferred to remote sensing scenes. In terms of multi-branch feature fusion, existing techniques usually use simple addition or attention weighting after channel stitching for fusion. This is prone to early branch collapse in the early stages of training, where some branches are at a disadvantage due to gradient competition and cannot play a normal role in feature extraction.

[0008] In summary, how to fundamentally solve the problem of high-frequency detail attenuation in the CNN-Transformer hybrid architecture while taking into account the requirements of lightweight deployment, and at the same time achieve the decoupling extraction and adaptive fusion of local spatial, global context and high-frequency edge features, is a technical challenge that urgently needs to be solved in the field of remote sensing image super-resolution reconstruction. Summary of the Invention

[0009] The purpose of this invention is to provide a lightweight remote sensing image super-resolution method with multi-scale frequency sensing to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a lightweight remote sensing image super-resolution method based on multi-scale frequency sensing, comprising: acquiring a low-resolution remote sensing image; extracting shallow features from the low-resolution remote sensing image to obtain shallow features; inputting the shallow features into multiple stacked multi-scale frequency sensing modules, wherein in each multi-scale frequency sensing module, local spatial features, global context features, and high-frequency features are extracted through three functionally independent parallel branches; the local spatial features, global context features, and high-frequency features are weighted and fused using an adaptive gating fusion mechanism to obtain the output features of the multi-scale frequency sensing module; the output features of all multi-scale frequency sensing modules are globally fused and then residually connected with the shallow features to obtain deep fused features; and the deep fused features are upsampled and reconstructed using a pixel shuffling operation to obtain a high-resolution remote sensing image.

[0011] Preferably, the local spatial branch uses an attention mechanism based on spatial residuals and channel residuals to extract the local spatial features.

[0012] Preferably, the global context branch uses a self-attention mechanism or a large kernel convolutional attention mechanism to extract the global context features.

[0013] Preferably, the high-frequency branch extracts the high-frequency features based on the principle of unsharpened masking.

[0014] Preferably, the step of extracting the local spatial features using an attention mechanism based on spatial residuals and channel residuals specifically includes: processing the input features sequentially through a first 1×1 convolution, a modified linear unit activation function, and a second 1×1 convolution to generate channel attention weights; multiplying the channel attention weights by the input features to obtain channel recalibration features; and applying spatial attention weights to the channel recalibration features to obtain the local spatial features.

[0015] Preferably, the global context branch uses a lightweight Swing Transformer block to extract the global context features, and the Swing Transformer block uses a shift window multi-head self-attention mechanism.

[0016] Preferably, the extraction of the high-frequency features based on the unsharpened mask principle specifically includes: performing average pooling on the input features to obtain low-frequency background information; subtracting the low-frequency background information from the input features to obtain high-frequency residuals; performing parallel convolution processing on the high-frequency residuals using 3×3 depthwise separable convolution and 5×5 depthwise separable convolution to extract multi-scale edge features; and fusing the multi-scale edge features through 1×1 convolution and restoring the channel dimension to obtain the high-frequency features.

[0017] Preferably, the weighted fusion of the local spatial features, the global context features, and the high-frequency features through an adaptive gating fusion mechanism specifically includes: adding the output features of the three parallel branches to obtain a unified context descriptor; generating channel-level attention weights corresponding to the three parallel branches through global average pooling and bottleneck convolution operations; performing flexible maximum value normalization on the channel-level attention weights so that the sum of the weights of each branch is 1; initializing the weights of the last layer of extended convolution to 0 so that the weights of the three parallel branches are all one-third in the early stage of training; performing weighted fusion of the output features of the three parallel branches based on the channel-level attention weights automatically adjusted by gradient descent during training, and performing residual connection between the fused features and the input features of the multi-scale frequency perception module.

[0018] Preferably, the step of globally fusing the output features of all the multi-scale frequency sensing modules and then performing a residual connection with the shallow features to obtain deep fused features specifically includes: splicing the output features of all the multi-scale frequency sensing modules along the channel dimension to obtain spliced ​​features; performing dimensionality reduction and global residual capture on the spliced ​​features through a bottleneck structure composed of 1×1 convolution and 3×3 convolution to obtain global fused features; and performing a residual connection between the global fused features and the shallow features to obtain the deep fused features.

[0019] Preferably, the number of stacked multi-scale frequency sensing modules is four, and each multi-scale frequency sensing module progressively refines the shallow features; the four multi-scale frequency sensing modules share the same set of trainable parameters.

[0020] A lightweight remote sensing image super-resolution system with multi-scale frequency sensing includes: a shallow feature extraction module for extracting shallow features from an input low-resolution remote sensing image; multiple cascaded multi-scale frequency sensing modules, each of which includes a local spatial extraction unit, a global context extraction unit, a high-frequency feature extraction unit, and an adaptive gated fusion unit; the adaptive gated fusion unit includes: a weight generation subunit for generating channel-level attention weights for each branch through global average pooling and bottleneck convolution; a normalization subunit for normalizing the channel-level attention weights using flexible maximum values; and a zero-level normalization subunit. The system comprises the following modules: an initialization subunit, which initializes the weights of the last layer of extended convolution to 0; a weighted fusion subunit, which performs weighted fusion of the output features of the three parallel branches based on the channel-level attention weights of each branch; a residual connection subunit, which performs residual connection between the fused features and the input features of the multi-scale frequency sensing module; a global feature fusion module, which performs global fusion of the output features of all the multi-scale frequency sensing modules and then performs residual connection with the shallow features to obtain deep fused features; and an upsampling reconstruction module, which performs upsampling reconstruction of the deep fused features using a pixel shuffling operation to obtain a high-resolution remote sensing image.

[0021] Preferably, the local spatial extraction unit uses an attention mechanism based on spatial residuals and channel residuals to extract the local spatial features.

[0022] Preferably, the global context extraction unit uses a self-attention mechanism or a large kernel convolutional attention mechanism to extract the global context features.

[0023] Preferably, the high-frequency feature extraction unit extracts the high-frequency features based on the principle of unsharpened masking.

[0024] Preferably, the high-frequency feature extraction unit includes: an average pooling subunit for performing average pooling on the input features to obtain low-frequency background information; a difference calculation subunit for subtracting the low-frequency background information from the input features to obtain high-frequency residuals; a multi-scale depthwise separable convolution subunit for performing parallel convolution processing on the high-frequency residuals using 3×3 depthwise separable convolution and 5×5 depthwise separable convolution to extract multi-scale edge features; and a channel restoration subunit for fusing the multi-scale edge features and restoring the channel dimensions through 1×1 convolution to obtain the high-frequency features.

[0025] Preferably, the upsampling reconstruction module uses a pixel shuffling operation to upsample the spatial resolution of the deep fusion features by 4 times to obtain the high-resolution remote sensing image.

[0026] An edge computing device includes a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the method described above.

[0027] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention fundamentally avoids the attenuation of high-frequency detail information during propagation by constructing a three-branch parallel decoupled deep feature extraction architecture, and solves the problem of excessive smoothing of high-frequency information after passing through the self-attention layer in the traditional CNN-Transformer serial hybrid architecture.

[0028] 2. This invention achieves effective protection and enhancement of high-frequency edges while maintaining a lightweight design, and can simultaneously meet the stringent requirements of edge computing devices for the number of model parameters and computational load.

[0029] The high-frequency feature extractor designed in this invention is based on the principle of unsharpened masks. It first obtains the low-frequency background through average pooling, then differs it from the input features to obtain pure high-frequency residuals. Finally, it uses 3×3 and 5×5 depthwise separable convolutions in parallel to extract multi-scale edge features. This design achieves dedicated protection and multi-scale enhancement of high-frequency features with extremely low computational cost, avoiding the problem of high-frequency information being gradually lost as network depth increases in conventional methods.

[0030] 3. The zero-initialization adaptive spatial gating fusion mechanism proposed in this invention effectively prevents the early branch collapse problem in multi-branch training and realizes dynamic adaptive feature fusion based on image content.

[0031] This invention sums the output features of three parallel branches to obtain a unified context descriptor, then generates channel-level attention weights through global average pooling and bottleneck convolution, which are then distributed to each branch after Softmax normalization. The key is to initialize the weights of the last layer of extended convolution to 0, ensuring that the weights of all three branches are 1 / 3 in the early stages of training. This avoids the early collapse problem caused by some branches being disadvantaged in gradient competition and unable to perform their feature extraction function properly under conventional random initialization. As training progresses, the weights of each branch are adaptively adjusted based on the image content—the weights of high-frequency branches are automatically increased in urban scenes, and the weights of global branches are automatically enhanced in mountainous scenes—achieving true content-adaptive fusion and improving the model's generalization ability to different types of remote sensing images.

[0032] 4. This invention improves the traditional channel attention module by inserting a ReLU activation function between two 1×1 convolutions, which alleviates the linear degradation problem in channel attention, enhances the ability to capture fine-grained textures and channel contrast, strengthens the nonlinear interaction between channels, and enables the attention weights to respond more accurately to channel differences, thereby improving the ability to capture fine-grained textures such as tree canopy boundaries and forest edges in remote sensing images.

[0033] 5. This invention adopts a three-stage architecture: "shallow feature extraction → progressive refinement through stacking four MSFA modules → global feature fusion and residual connection → PixelShuffle upsampling". Shallow extraction preserves the original spatial structure; the four MSFA modules achieve progressive feature refinement through multi-stage parallel extraction and fusion; global fusion concatenates the features from each stage, reduces the dimensionality through a bottleneck structure, and connects them with the shallow feature residuals, preserving feature information at each level, enhancing gradient flow, and facilitating network training convergence.

[0034] 6. The present invention achieved state-of-the-art results on both the UC Merced and NWPU-RESISC45 standard remote sensing datasets, verifying the effectiveness and generalization ability of the method. Attached Figure Description

[0035] Figure 1 A schematic diagram of the overall network architecture of the lightweight remote sensing image super-resolution method with multi-scale frequency sensing provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multi-scale frequency sensing parallel extraction core module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the local spatial feature extraction module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the lightweight window self-attention extraction module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the multi-scale high-frequency feature extraction module provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figure 1 As shown, this embodiment provides a lightweight remote sensing image super-resolution method based on multi-scale frequency sensing. This method achieves super-resolution reconstruction of remote sensing images based on a three-branch parallel decoupling architecture.

[0038] The method in this embodiment includes the following steps: Step 1: Shallow feature extraction.

[0039] Low-resolution remote sensing images are acquired. During the model training phase, the low-resolution remote sensing images are obtained by bicubic downsampling degradation of high-resolution remote sensing images, while corresponding real high-resolution remote sensing images are acquired as training supervision signals. During the model inference phase, the low-resolution remote sensing images are actual low-resolution remote sensing images, and the model directly outputs the reconstruction results without the need for real high-resolution images.

[0040] In this embodiment, the input low-resolution remote sensing image is set as... ,in and These represent the height and width of the low-resolution remote sensing image, respectively. This indicates the number of channels in the input image.

[0041] Use one The convolutional layer performs shallow feature extraction on the low-resolution remote sensing image, mapping the low-resolution remote sensing image from the input space to a high-dimensional feature space. The kernel size of this convolutional layer is [size missing]. The step size is 1, and the padding is 1. Specifically, shallow feature extraction is achieved through the following formula:

[0042] in, This represents shallow features, with dimensions of . , This represents the number of channels for shallow features. In this embodiment, The value is 64.

[0043] Step 2: Deep feature extraction.

[0044] shallow features The input is fed into multiple stacked multi-scale frequency sensing modules, each progressively refining the shallow features. In this embodiment, there are four stacked multi-scale frequency sensing modules, labeled MSFA-1, MSFA-2, MSFA-3, and MSFA-4. Each module maintains the same spatial resolution as the input features; that is, the input and output features of each module remain consistent in both height and width dimensions. Therefore, the output features of the four multi-scale frequency sensing modules are... , , , The spatial resolution is Shallow features The signal is passed through four multi-scale frequency sensing modules in sequence to obtain the output features of each module. , , , .

[0045] In this embodiment, the four multi-scale frequency sensing modules adopt a parameter-sharing strategy, meaning that the four modules share the same set of trainable parameters to progressively refine the shallow features. The total number of parameters is comparable to that of a single MSFA module.

[0046] like Figure 2 As shown, each multi-scale frequency sensing module comprises three functionally independent parallel branches: a local spatial extractor, a global context extractor, and a high-frequency feature extractor. Given input features... The three branches respectively address Perform parallel processing.

[0047] Local spatial extractors are used to extract local spatial features from remotely sensed images. Input features First, it is processed by a local spatial extractor. For example... Figure 3 As shown, this local spatial extractor is designed based on an attention mechanism of spatial residuals and channel residuals. Specifically, the input features... In sequence through the first Convolution, modified linear unit activation function and second Convolution is used to generate channel attention weights. This process can be represented as:

[0048] in, and They represent the first Convolution and Second Convolution. First. Convolution reduces the number of channels in the input features from... Compress to ,second Convolution restores the number of channels to ,in This represents the channel compression ratio. In this embodiment, The value of is 4.

[0049] Then, the channel attention weights Input features Multiplying these yields the channel recalibration characteristics:

[0050] in, This indicates element-wise multiplication.

[0051] Next, spatial attention weighting is applied to the channel recalibration features to obtain local spatial features. Specifically, through a... The convolutional layer generates a spatial attention weight map with one channel. Each spatial location of this weight map takes a value between 0 and 1. Spatial feature recalibration is achieved by multiplying it element-wise with the channel recalibration features (broadcast along the channel dimension), resulting in local spatial features.

[0052] in, This represents the local spatial features output by the local spatial extractor.

[0053] A global context extractor is used to extract global contextual features from remotely sensed images. Input features Simultaneously input into the global context extractor. For example... Figure 4 As shown, in this embodiment, the global context extractor is implemented using a lightweight SwinTransformer block. The SwinTransformer block contains two consecutive SwinTransformer layers, each containing a window-based multi-head self-attention module and a two-layer multilayer perceptron, with residual connections and layer normalization between layers. This SwinTransformer block employs a shift-window multi-head self-attention mechanism, alternating between regular windows and shift windows for self-attention computation across different layers, capturing the macroscopic terrain structure dependency of remote sensing images while maintaining linear computational complexity. Input Features After processing by the Swin Transformer block, the global context features are obtained:

[0054] in, This represents the global context features output by the global context extractor. In this embodiment, the window size in the SwingTransformer block is set to 8, and the number of heads for multi-head attention is set to 6.

[0055] High-frequency feature extractors are used to extract high-frequency edge features from remotely sensed images. For example... Figure 5 As shown, input features Simultaneously, the input is fed into a high-frequency feature extractor. This high-frequency feature extractor is designed based on the principle of unsharpened masks. Specifically, the input features are... Average pooling is performed to obtain low-frequency background information. The pooling window size for average pooling is... The step size is 1, and the fill size is 1. This process can be represented as:

[0056] in, This indicates low-frequency background information.

[0057] Input features With the low-frequency background information Subtraction yields the high-frequency residual:

[0058] in, This represents the high-frequency residual.

[0059] use Depthwise separable convolution and Depthwise separable convolution pairs the high-frequency residuals Parallel convolution processing is performed to extract multi-scale edge features. This process can be represented as:

[0060] in, and These represent the convolution kernel as follows: The depth can be separated into convolution and convolution kernel as Depth-separable convolution, and They represent Depthwise separable convolution and Multi-scale edge features extracted by depthwise separable convolution.

[0061] pass Convolutional fusion of the multi-scale edge features and restoration of channel dimensions yields high-frequency features:

[0062] in, This indicates splicing along the channel dimension. This represents the high-frequency features output by the high-frequency feature extractor.

[0063] In obtaining local spatial features Global context features and high frequency characteristics Then, the three are weighted and fused using an adaptive gating fusion mechanism (see...). Figure 2 The dimensionality reduction and zero-initialized weight generation network and feature fusion dimensionality reduction layer in the text. Specifically, the output features of the three parallel branches are first summed to obtain a unified context descriptor:

[0064] Then, channel-level attention weights corresponding to the three parallel branches are generated through global average pooling and bottleneck convolution operations. Specifically, for the unified context descriptor... Perform global average pooling to compress the spatial dimension to The channel descriptor is obtained. Then, it is processed through a bottleneck convolutional structure (dimensionality reduction ratio of 1). The initial attention weights for the three branches are generated by processing the compressed linear unit activation function and then restoring the channel dimension through expanded convolution. The "expanded convolution" refers to the process used to restore the number of channels from the compressed dimension to the original dimension. of The convolutional layer, whose weights are the same as "the weights of the last extended convolutional layer" in the claim. The kernel weights of the extended convolutional layer are initialized to 0, so that the initial attention weights of the three branches are all 0. After flexible maximum normalization, the weights of each branch are all 1 / 3. This process can be expressed as:

[0065] in, Indicates global average pooling. Indicates dimensionality reduction convolution, Representing extended dimensions Convolution, the weights of this convolutional layer are initialized to 0.

[0066] The channel-level attention weights are then subjected to flexible maximum normalization so that the sum of the weights of each branch is 1.

[0067] in, , which correspond to the weights of local spatial features, global context features, and high-frequency features, respectively.

[0068] During training, the weights of each branch are automatically adjusted as gradient descent occurs, thereby achieving adaptive feature fusion based on image content.

[0069] Based on the channel-level attention weights that are automatically adjusted by gradient descent during training, the output features of the three parallel branches are weighted and fused to obtain the fused features:

[0070] Finally, the fusion features Input features of the multi-scale frequency sensing module Perform residual connections to obtain the output features of the multi-scale frequency sensing module:

[0071] Step 3: Global feature fusion and residual connection.

[0072] like Figure 1 As shown, the four multi-scale frequency sensing modules output features respectively. , , , Since each module maintains a constant spatial resolution, to All space dimensions are Therefore, they can be directly concatenated along the channel dimension. By concatenating the output features of all multi-scale frequency sensing modules along the channel dimension, the concatenated features are obtained:

[0073] pass Convolution and The bottleneck structure formed by convolutions performs dimensionality reduction and global residual capture on the concatenated features to obtain globally fused features. Specifically, firstly, through... Convolution performs channel dimensionality reduction on the concatenated features, and then... Convolution performs feature transformation to restore the target number of channels:

[0074] Then, the global fusion features Compared with the shallow features extracted in step one Perform residual connections to obtain deep fusion features:

[0075] In this embodiment, the Convolution will The number of channels is set to This allows subsequent pixel shuffling operations to rearrange information from the channel dimension to the spatial dimension, achieving a 4x upsampling.

[0076] Step 4: Upsampling and reconstruction.

[0077] For the deep fusion features Upsampling reconstruction is performed to obtain a high-resolution remote sensing image. In this embodiment, a pixel shuffling operation is used to upsample the deep fusion features. The pixel shuffling operation upsamples the spatial resolution of the feature map by a factor of 4. Specifically, firstly, through... Convolution reduces the number of channels in deeply fused features from Expand to Then, by shuffling pixels, the channel dimensions are rearranged to the spatial dimensions to achieve upsampling.

[0078] in, This represents the final reconstructed high-resolution remote sensing image. In this embodiment, the upsampling factor... , The value of is the same as the number of channels in the input image.

[0079] Step 5: Training optimization.

[0080] The L1 loss function is used to train and optimize the network. For a batch of training samples, the L1 loss function is calculated as follows:

[0081] in, Indicates batch size, Indicates the first Reconstructed high-resolution images of each sample, Indicates the first A true high-resolution image of a sample, This represents the L1 norm. All trainable parameters in the network are updated using backpropagation and gradient descent algorithms until the loss function converges. In this embodiment, the batch size... The typical value is 16, and the initial learning rate is... A cosine annealing learning rate scheduling strategy is adopted.

[0082] Table 1 presents a quantitative comparison between the proposed method (MSFA-Net) and existing mainstream super-resolution models on the UC Merced remote sensing dataset at a 4x magnification. As can be seen from Table 1, the proposed method achieves the best performance across all seven evaluation metrics: PSNR, SSIM, FSIM, VIF, LPIPS, ERGAS, and SAM.

[0083] Table 1. Quantitative comparison results of 4x super-resolution over the UC Merced dataset.

[0084] Note: Bold values ​​represent the optimal results.

[0085] Table 2 presents the quantitative evaluation results of the proposed method on the NWPU-RESISC45 remote sensing dataset, which features more complex scenes and greater terrain diversity, at a 4x magnification. As can be seen from Table 2, the proposed method comprehensively outperforms all baseline models on this dataset, achieving the highest PSNR of 29.48 dB, the highest SSIM of 0.7776, the highest FSIM of 0.8449, and the highest VIF of 0.3790. Simultaneously, it achieves the lowest LPIPS (0.3321), the lowest ERGAS (2.1371), and the lowest SAM (0.8744), demonstrating optimal performance across all seven evaluation metrics.

[0086] Table 2. Quantitative comparison results of 4x super-resolution NWPU-RESISC45 dataset.

[0087] Note: Bold values ​​represent the optimal results.

[0088] Table 3 presents the ablation experiment results of each parallel branch within the multi-scale frequency sensing module in this embodiment. All experimental schemes in this table assume the use of a zero-initialization adaptive spatial gating fusion mechanism. As can be seen from Table 3, none of the metrics reached their optimal levels when operating with a single branch; adding any functionally decoupled parallel branch consistently resulted in a steady improvement in PSNR and SSIM; when all three branches were fully combined, the model achieved a maximum PSNR of 29.26 dB and an SSIM of 0.7911. Furthermore, compared to the two-branch baseline (LSE+GCE), the addition of the complete three-branch architecture only slightly increased the number of parameters by approximately 32.4K (from 424.896K to 457.296K), demonstrating that the introduction of the HFE branch achieved a significant improvement in reconstruction quality with minimal computational cost.

[0089] Table 3 Ablation experimental results of each parallel branch within the MSFA module

[0090] illustrate: 1. The bolded values ​​represent the optimal results. This indicates that the branch is enabled.

[0091] 2. In the table, Params and FLOPs represent the total number of parameters and the total computational cost after stacking the four MSFA modules.

[0092] 3. All experimental schemes in this table use the zero-initialization adaptive spatial gating fusion mechanism by default.

[0093] Example 2 The difference between this embodiment and Embodiment 1 is that the specific implementation of the global context extractor can be replaced by other lightweight general-purpose visual Transformer modules or large kernel convolutional attention modules.

[0094] Similar to Example 1, this example also employs a three-branch parallel decoupled architecture, including a local spatial extractor, a global context extractor, and a high-frequency feature extractor. The specific structures of the local spatial extractor and the high-frequency feature extractor are exactly the same as in Example 1.

[0095] In the global context extractor, Example 1 uses a lightweight Swing Transformer block to implement global context modeling. In this example, the Swing Transformer block is replaced with a large-kernel convolutional attention module. This large-kernel convolutional attention module uses a convolutional kernel size of... or Depth-separable convolutions capture large-scale spatial dependencies in images, enabling the extraction of macroscopic terrain structure features from remote sensing images. The specific calculation process is as follows:

[0096] in, This represents the large kernel convolutional attention operation, including... Depthwise separable convolution, modified linear unit activation function and A concatenated structure of convolutions.

[0097] The other steps and parameters in this embodiment are the same as in Embodiment 1, and will not be repeated here.

[0098] Example 3 The difference between this embodiment and Embodiment 1 is that the method of separating high-frequency textures in the high-frequency feature extractor can be equivalently replaced by a Gaussian blur filter.

[0099] Similar to Example 1, the high-frequency feature extractor in this example is designed based on the principle of unsharpened masks. The difference from Example 1 is that in this example, a Gaussian blur filter is used instead of average pooling when acquiring low-frequency background information. Specifically, for the input features... Gaussian blurring was performed to obtain low-frequency background information:

[0100] in, The standard deviation is expressed as Gaussian blur filtering operation.

[0101] Then, input features With the low-frequency background information Subtraction yields the high-frequency residual:

[0102] After that, adopt Depthwise separable convolution and Depthwise separable convolutions are used to perform parallel convolution processing on the high-frequency residuals to extract multi-scale edge features, and then... Convolutional fusion restores the channel dimension, yielding high-frequency features. The specific implementation of the above steps is the same as in Example 1.

[0103] The other steps and parameters in this embodiment are the same as in Embodiment 1, and will not be repeated here.

[0104] Example 4 The difference between this embodiment and Embodiment 1 is that the Sobel operator can be used to replace the high-frequency residual enhancement stage.

[0105] Similar to Example 1, the high-frequency feature extractor in this example first performs average pooling on the input features to obtain low-frequency background information. Then, it subtracts the low-frequency background information from the input features to obtain the high-frequency residual. Unlike Example 1, in this example, after extracting the high-frequency residual, the Sobel operator is used to perform edge enhancement processing on the high-frequency residual, instead of using... and Depthwise separable convolutions are used for parallel extraction. Specifically:

[0106] in, This represents the Sobel edge detection operator, which includes Sobel convolution kernels in the horizontal and vertical directions:

[0107] The high-frequency residuals are convolved along both the horizontal and vertical directions, and then the modulus is taken to obtain the edge enhancement features. Alternatively, edge detection high-pass filtering algorithms such as the Laplacian operator can be used as replacements; these variations are all within the scope of this invention.

[0108] The other steps and parameters in this embodiment are the same as in Embodiment 1, and will not be repeated here.

[0109] Example 5 The difference between this embodiment and Embodiment 1 is that the PixelShuffle upsampling operation in the upsampling reconstruction module can be equivalently replaced by transposed convolution.

[0110] Similar to Example 1, this example requires upsampling and reconstruction of the deep fusion features after global feature fusion and residual connection. Unlike Example 1, this example uses transposed convolution instead of pixel shuffling for upsampling. Specifically, a convolution kernel size of [size missing] is used. Step size is The transposed convolution upsamples the deep fusion features:

[0111] in, The upsampling factor (in this embodiment) ), Indicates the kernel size as Step size is The transposed convolution operation. During training, the kernel parameters of the transposed convolution are automatically updated during backpropagation.

[0112] The other steps and parameters in this embodiment are the same as in Embodiment 1, and will not be repeated here.

[0113] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, equipment, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0116] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program instructions, such as USB flash drives, portable hard drives, read-only storage servers, random access storage servers, magnetic disks, or optical disks.

[0117] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0118] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0119] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight remote sensing image super-resolution method based on multi-scale frequency sensing, characterized in that, include: Acquire low-resolution remote sensing images; Shallow features are extracted from low-resolution remote sensing images to obtain shallow features; Shallow features are input into multiple stacked multi-scale frequency sensing modules. In each multi-scale frequency sensing module, local spatial features, global context features, and high-frequency features are extracted through three functionally independent parallel branches. The local spatial branch extracts local spatial features using an attention mechanism based on spatial residuals and channel residuals. The global context branch extracts global context features using a self-attention mechanism or large-kernel convolutional attention. The high-frequency branch extracts high-frequency features based on the principle of unsharpened masks. By using an adaptive gating fusion mechanism, local spatial features, global context features, and high-frequency features are weighted and fused to obtain the output features of the multi-scale frequency sensing module. After globally fusing the output features of all multi-scale frequency sensing modules, residual connections are made with the shallow features to obtain deep fused features; The deep fusion features are upsampled and reconstructed using a pixel shuffling operation to obtain a high-resolution remote sensing image.

2. The method according to claim 1, characterized in that, The step of weightedly fusing the local spatial features, the global context features, and the high-frequency features through an adaptive gating fusion mechanism specifically includes: The output features of the three parallel branches are added together to obtain a unified context descriptor; Channel-level attention weights corresponding to the three parallel branches are generated through global average pooling and bottleneck convolution operations. The channel-level attention weights are processed by flexible maximum value normalization so that the sum of the weights of each branch is 1; The weights of the last extended convolution layer are initialized to 0, so that the weights of the three parallel branches mentioned in the early stage of training are all one-third. Based on the channel-level attention weights that are automatically adjusted by gradient descent during training, the output features of the three parallel branches are weighted and fused, and the fused features are residually connected with the input features of the multi-scale frequency sensing module.

3. The method according to claim 1, characterized in that, The extraction of high-frequency features based on the principle of unsharpened masks specifically includes: The input features are subjected to average pooling to obtain low-frequency background information; Subtracting the input features from the low-frequency background information yields the high-frequency residual; The high-frequency residuals are processed by parallel convolution of 3×3 depthwise separable convolution and 5×5 depthwise separable convolution to extract multi-scale edge features; The high-frequency features are obtained by fusing the multi-scale edge features through 1×1 convolution and restoring the channel dimension.

4. The method according to claim 1, characterized in that, The extraction of local spatial features using an attention mechanism based on spatial residuals and channel residuals specifically includes: The input features are processed sequentially through a first 1×1 convolution, a modified linear unit activation function, and a second 1×1 convolution to generate channel attention weights. Multiply the channel attention weights by the input features to obtain the channel recalibration features; The local spatial features are obtained by applying spatial attention weighting to the channel recalibration features.

5. The method according to claim 1, characterized in that, The global context branch uses a lightweight SwinTransformer block to extract the global context features, and the SwinTransformer block uses a shifted window multi-head self-attention mechanism.

6. The method according to claim 1, characterized in that, The step of globally fusing the output features of all the multi-scale frequency sensing modules and then performing a residual connection with the shallow features to obtain deep fused features specifically includes: The output features of all the multi-scale frequency sensing modules are concatenated along the channel dimension to obtain the concatenated features; The concatenated features are reduced in dimensionality and global residuals are captured by a bottleneck structure consisting of 1×1 convolutions and 3×3 convolutions to obtain global fused features. The global fusion feature is residually connected with the shallow feature to obtain the deep fusion feature.

7. The method according to claim 1, characterized in that, The number of stacked multi-scale frequency sensing modules is four, and each multi-scale frequency sensing module progressively refines the shallow features; the four multi-scale frequency sensing modules share the same set of trainable parameters.

8. A lightweight remote sensing image super-resolution system with multi-scale frequency sensing, characterized in that, include: The shallow feature extraction module is used to extract shallow features from the input low-resolution remote sensing image to obtain shallow features; Multiple cascaded multi-scale frequency sensing modules, each of which includes: The local spatial extraction unit is used to extract local spatial features using an attention mechanism based on spatial residuals and channel residuals; The global context extraction unit is used to extract global context features using a self-attention mechanism or a large kernel convolutional attention mechanism. The high-frequency feature extraction unit is used to extract high-frequency features based on the principle of unsharpened mask. An adaptive gated fusion unit is used to weightedly fuse the local spatial features, the global context features, and the high-frequency features through an adaptive gated fusion mechanism to obtain the output features of the multi-scale frequency sensing module. The adaptive gated fusion unit includes: a weight generation subunit, used to generate channel-level attention weights for each branch through global average pooling and bottleneck convolution; a normalization subunit, used to normalize the channel-level attention weights using flexible maximum value normalization; a zero-initialization subunit, used to initialize the weights of the last layer of extended convolution to 0; a weighted fusion subunit, used to weightedly fuse the output features of the three parallel branches based on the channel-level attention weights of each branch; and a residual connection subunit, used to perform residual connections between the fused features and the input features of the multi-scale frequency sensing module. The global feature fusion module is used to globally fuse the output features of all the multi-scale frequency sensing modules and then perform residual connection with the shallow features to obtain deep fused features. The upsampling reconstruction module is used to upsample and reconstruct the deep fusion features to obtain a high-resolution remote sensing image.

9. The system according to claim 8, characterized in that, The high-frequency feature extraction unit includes: The average pooling subunit is used to perform average pooling on the input features to obtain low-frequency background information; The differential calculation subunit is used to subtract the input features from the low-frequency background information to obtain the high-frequency residual; Multi-scale depth-separable convolutional subunits are used to perform parallel convolution processing on the high-frequency residuals using 3×3 depth-separable convolution and 5×5 depth-separable convolution to extract multi-scale edge features; The channel recovery subunit is used to fuse the multi-scale edge features and recover the channel dimensions through 1×1 convolution to obtain the high-frequency features.

10. An edge computing device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

Images