A lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency enhancement Mamba

Abstract

The application discloses a kind of light weight remote sensing image super-resolution methods based on multi-scale convolution and frequency enhancement Mamba, it is related to remote sensing image processing and deep learning technical field.The purpose is to solve the problem such as insufficient feature extraction of existing light weight model, Mamba local information loss etc..Method includes: data preprocessing, low-resolution remote sensing image normalization;MFEM network is constructed and trained, which includes shallow feature extraction module, cascaded multi-scale separation distillation module, frequency Mamba enhancement module of grouping fusion and reconstruction module;The image after pre-processing is input into the MFEM network trained, multi-scale features are extracted by MSDB, long-range dependence and local details are balanced by FMEB using parallel branch, and finally high-resolution image is output by the reconstruction module up-sampling.The application has low calculation amount, fast inference speed, can be deployed on edge device, and is suitable for geological exploration, disaster emergency and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and deep learning technology, specifically to a lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba. Background Technology

[0002] Remote sensing images (RSI), as a core data carrier for Earth surface observation, provide rich information on surface spatial characteristics, target structural features, and dynamic changes. They hold irreplaceable application value in several key areas, including geological exploration, land resource management, environmental monitoring, agricultural production planning, and disaster emergency response. In geological exploration, high-resolution remote sensing images can accurately identify strata lithology and fault traces, providing reliable data support for mineral resource exploration and geological hazard risk assessment. In land resource management, clear image details help accurately classify land use types such as arable land, construction land, and ecological land, supporting national land spatial planning and land use control. In environmental monitoring, high-resolution images can track real-time changes in vegetation cover, water pollution levels, and atmospheric particulate matter distribution, providing a scientific basis for ecological protection and pollution control decisions. Furthermore, in agricultural production, they can assist in identifying crop growth and pest distribution, enabling precise irrigation and fertilization. In disaster emergency response, they can quickly present the scope and extent of damage, providing crucial reference for rescue plan development and resource allocation. Therefore, acquiring high-resolution remote sensing images is a core prerequisite for improving the scientific nature of decision-making, execution efficiency, and response timeliness in various fields.

[0003] However, current remote sensing image super-resolution technology still faces many prominent challenges in practical applications, specifically in the following aspects:

[0004] The limited hardware of edge devices makes it difficult to guarantee the quality of image acquisition. Remote sensing image acquisition largely relies on edge devices such as portable observation equipment, drone-mounted terminals, and remote sensing monitoring base stations. These devices are generally limited in terms of hardware computing power and storage resources due to factors such as size, power consumption, and cost. Under the combined influence of adverse factors such as long-distance transmission, complex weather conditions (e.g., cloud cover, lighting variations), and wide-angle imaging, acquired remote sensing images often suffer from low resolution, blurred details, and noise interference. Upgrading hardware to improve image quality not only requires high procurement costs but also increases equipment energy consumption and maintenance complexity, making it difficult to promote in large-scale application scenarios.

[0005] Existing lightweight RSISR methods suffer from insufficient feature extraction and inadequate ability to reconstruct complex structures. To adapt to the deployment requirements of edge devices, existing lightweight remote sensing image super-resolution methods (such as BSRN, AMFFN, and FeNet) have achieved lightweighting by simplifying network structures and reducing parameter size, but they generally suffer from low feature extraction efficiency and limited receptive field. These methods struggle to fully capture the structural features of complex targets in remote sensing images (such as dense building clusters, irregular terrain boundaries, and scenes with interwoven multiple types of ground features), leading to distorted target contours and missing details in the reconstructed images, thus failing to meet the needs of high-precision applications.

[0006] Convolutional Neural Networks (CNNs) have inherent limitations, particularly in their ability to model multi-scale information and long-range dependencies. Traditional lightweight Regression-Symptom Reconstruction (RSISR) methods are largely based on CNN architectures, but the convolutional operations of CNNs have significant limitations: firstly, most models use single-scale convolutional kernels (such as 3×3 convolutions) for feature extraction, which cannot effectively adapt to the coexistence of multi-scale targets in remote sensing images (such as large mountains and small infrastructure, vast farmland and scattered buildings), resulting in incomplete multi-scale feature capture; secondly, the local receptive field characteristic of convolutional operations makes it difficult to model the correlation information between long-distance targets in images (such as the relationship between road extension trajectories, river flow directions, and surrounding terrain), thus affecting the overall reconstruction accuracy of complex scenes.

[0007] Self-attention mechanisms are too complex, contradicting the requirements of lightweight design. While self-attention mechanisms (such as the attention module in Transformer-based models) can effectively capture global contextual information and improve long-range dependency modeling capabilities, their computational complexity increases quadratically with the input image size. For example, the TTST model in the field of remote sensing image super-resolution has a parameter size of 18367K, a computational cost (FLOPs) of 32.7G, and an inference time of 278.61ms. This excessive complexity makes it unsuitable for deployment on resource-constrained edge devices, completely failing to meet the core requirements of lightweight design.

[0008] The Mamba model has inherent limitations, with the loss of local information affecting reconstruction results. In recent years, the Mamba model, with its global awareness capabilities and linear complexity, has gained attention in image reconstruction (methods such as ConvMambaSR and FMSR). However, it suffers from inherent local information loss when modeling long-range dependencies. Although some methods (such as FMSR) supplement high-frequency information by introducing external Fast Fourier Transform (FFT), the optimization effect of this external linking method is limited and still cannot fully compensate for the loss of local details (such as the texture of ground features and the contours of small targets), resulting in insufficient detail clarity and realism in the reconstructed image.

[0009] High-performance models incur excessive computational overhead, making them unsuitable for lightweight deployment scenarios. While some RSISR methods, such as EDiffSR, can achieve high reconstruction accuracy, they rely on complex probability generation processes, requiring numerous iterative calculations, resulting in enormous computational costs and slow inference speeds. Their large parameter scale and computational requirements limit their operation to high-performance servers, making them completely unsuitable for lightweight deployment scenarios on edge devices and failing to meet the real-time and convenience requirements of practical applications.

[0010] In summary, existing remote sensing image super-resolution technologies fail to effectively balance the core requirements of "lightweight deployment" and "high-precision reconstruction": traditional high-precision models (such as Transformer-based models and diffusion models) are highly complex and cannot be adapted to edge devices; lightweight models (such as CNN-based and Mamba-based lightweight solutions) suffer from insufficient multi-scale feature extraction, weak long-range dependency modeling capabilities, and loss of local information, resulting in reconstruction accuracy that fails to meet practical application requirements. Therefore, an innovative RSISR method is urgently needed to achieve sufficient multi-scale feature extraction, efficient long-range dependency modeling, and accurate preservation of local details while strictly controlling model complexity and meeting the deployment requirements of edge devices, thus resolving the core contradiction in existing technologies where lightweightness and high precision are difficult to achieve simultaneously. Summary of the Invention

[0011] This invention addresses the problems of existing technologies by providing a lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba. It addresses core issues in existing remote sensing image super-resolution (RSISR) techniques, such as insufficient multi-scale feature extraction, limited receptive field, loss of inherent local information in the Mamba model, and the difficulty in balancing complexity and reconstruction accuracy. Through an innovative design that integrates a multi-scale feature extraction module with a frequency-enhanced Mamba fusion mechanism, it achieves an optimal balance between lightweight deployment and high-precision reconstruction. This method aims to overcome the hardware resource limitations of edge devices, fully capturing multi-scale target features of remote sensing images, efficiently modeling long-range dependencies, and accurately preserving local details while strictly controlling model parameter scale and computational overhead. This meets the high-precision resolution improvement needs of resource-constrained edge devices in practical scenarios such as geological exploration, land resource management, environmental monitoring, and disaster emergency response, promoting the engineering implementation and widespread application of lightweight RSISR technology.

[0012] To address the above technical problems, this invention provides the following technical solution: a lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba, comprising the following steps:

[0013] Step 1: Preprocess the input low-resolution LR remote sensing image to generate a high-resolution HR remote sensing image, and construct a training sample dataset including pairs of low-resolution LR remote sensing images and high-resolution HR remote sensing images.

[0014] Step 2: Construct a multi-scale frequency-enhanced Mamba network (MFEM), including:

[0015] The shallow feature extraction module is used to perform channel stacking and convolution operations on the input image to extract the shallow feature map F0;

[0016] Several cascaded multiscale separation distillation modules (MSDB) are used to receive the output features of the previous module and progressively extract multiscale feature maps F1~F_N through the internal multiscale blueprint residual blocks (MBRB) and feature distillation mechanism. Each multiscale separation distillation module (MSDB) includes three sizes of multiscale blueprint residual blocks (MBRB-a, MBRB-b, MBRB-c), as well as an enhanced spatial attention (ESA) module and a contrastive perception channel attention (CCA) module.

[0017] The group fusion consists of several frequency Mamba enhancement modules (FMEBs). Each frequency Mamba enhancement module (FMEB) receives at least two input features and performs feature fusion and optimization through internal parallel long-range dependency modeling branches, modulation gating branches, and frequency enhancement branches, outputting enhanced features.

[0018] The reconstruction module is used to fuse the high-level features output by the last FMEB with the shallow feature map F0, and perform pixel shuffling upsampling to output a high-resolution remote sensing image.

[0019] Step 3: Use the dataset from Step 1 to train a multi-scale frequency augmentation Mamba network (MFEM) to obtain a lightweight remote sensing image super-resolution model. Then, use this model to process low-resolution LR remote sensing images and output high-resolution HR remote sensing images.

[0020] Furthermore, in step 1 above, the input LR remote sensing image is randomly horizontally flipped (flipping probability 50%) and randomly rotated 90° (including four angles randomly selected: 0°, 90°, 180°, and 270°) to expand the diversity of training samples, improve the model's generalization ability, and adapt to remote sensing image scenarios with different acquisition angles and postures.

[0021] Pixel value normalization: Normalize the pixel values ​​of all images to the range of [0,1] to eliminate training fluctuations caused by differences in pixel value scale, ensure numerical stability, and optimize gradient descent efficiency.

[0022] Training sample construction: Based on high-resolution (HR) remote sensing images, bicubic interpolation is used to perform downsampling by ×2, ×3, and ×4 times respectively to generate LR images that correspond one-to-one with HR images. "LR-HR" paired training samples are constructed to simulate the scenario of image resolution reduction due to equipment limitations and transmission loss in real-world applications.

[0023] Furthermore, the aforementioned step 2, which constructs the multi-scale frequency-enhanced Mamba network (MFEM), adopts a progressive structure of "shallow feature extraction → multi-scale feature refinement → group fusion enhancement → upsampling reconstruction." The functions and connections of each module are as follows:

[0024] Shallow Feature Extraction Module: First, the input LR image is stacked n times along the channel dimension (n is the number of channel stackings, preferably n=4) to enrich the texture details of the initial features. Then, a 3×3 Blueprint Separable Convolution (BSConv) is used to convolve the stacked features to extract a shallow feature map F0 with uniform dimensions. In this module, the stride of the 3×3 BSConv is set to 1 and the padding method is "same" to ensure that the input and output feature map sizes are consistent. At the same time, the lightweight nature of BSConv is utilized to reduce computational overhead while preserving cross-channel information and spatial features.

[0025] Multiscale Separation Distillation Module (MSDB): As the core of multiscale feature extraction, there are 4 MSDBs deployed in series in sequence. Each MSDB receives the output features from the previous module, and through the internal multiscale blueprint residual block (MBRB) and feature distillation mechanism, it progressively refines the features and extracts key information, outputting multiscale feature maps F1~F4 that combine local details and structural features.

[0026] Frequency Mamba Enhancement Module (FMEB): Three modules are configured, operating in a grouped fusion mode. The first two FMEBs receive the output features from two adjacent MSDBs (F1 and F2, F3 and F4), respectively, achieving cross-level feature fusion through frequency enhancement and long-range dependency modeling. The third FMEB receives the output features from the first two FMEBs (M1 and M2), further enhancing global feature interaction and outputting the integrated high-level feature M3. This module combines internal frequency analysis with the Mamba architecture to compensate for local information loss while modeling long-range dependencies.

[0027] Reconstruction module: First, a 3×3 BSConv layer is used to deeply integrate the high-level feature M3, and then element-wise addition is performed with the shallow feature F0 to preserve low-level texture details. Next, a 3×3 convolutional layer is used to map the number of feature channels to 3 (to adapt to RGB three-channel image output). Finally, pixel shuffle operation is used to achieve upsampling, and the upsampling factor is consistent with the target scaling factor (×2, ×3 or ×4). The final output is an SR remote sensing image.

[0028] Furthermore, the structure and working mechanism of the aforementioned Multiscale Blueprint Residual Block (MBRB), Multiscale Separation Distillation Module (MSDB), and Frequency Mamba Enhancement Module (FMEB) are as follows:

[0029] Multi-scale Blueprint Residual Blocks (MBRBs): Three specifications (MBRB-a, MBRB-b, MBRB-c) are designed, employing a multi-scale BSConv and channel separation strategy to balance lightweight design with multi-scale feature extraction capabilities.

[0030] Channel separation: The input features are uniformly divided into two sub-features P1(xi) and Q1(xi) along the channel dimension, each accounting for 1 / 2 of the original number of channels, to achieve parallel processing and reduce computational complexity;

[0031] Different specifications and structures:

[0032] MBRB-a: Sub-feature P1(xi) is extracted using 3×3 BSConv, and sub-feature Q1(xi) is adjusted for channel dimensions using 1×1 convolution. The two are concatenated and then residually connected to the input features to output refined features.

[0033] MBRB-b: Sub-feature P1(xi) is processed by 5×5 BSConv to extract large-scale features, and sub-feature Q1(xi) is processed by 1×1 convolution to adjust the channel dimension. After concatenation, the non-linear expression is enhanced by the GELU activation function, and then residual connection is performed with the input features.

[0034] MBRB-c: Sub-feature P1(xi) is processed by 5×5 BSConv to extract large-scale features, and sub-feature Q1(xi) is processed by 3×3 BSConv to extract medium-scale features. After concatenation, the features are enhanced by the GELU activation function and then residually connected with the input features.

[0035] Core advantages: By combining BSConv at different scales, it adapts to the characteristics of multi-scale target coexistence in remote sensing images. At the same time, the kernel correlation design of BSConv can reduce parameter redundancy while preserving feature representation ability.

[0036] Multiscale Separation Distillation Module (MSDB): Designed based on the principle of information distillation, it achieves efficient refinement of local features and extraction of key information.

[0037] Feature progressive refinement: The input feature Fin is progressively processed by MBRB-a, MBRB-b, and MBRB-c. After each processing, the feature is distilled by a 1×1 convolution (the number of channels is halved), generating distilled features F1^distilled~F4^distilled and intermediate coarse features F1^coarse~F3^coarse. The last distilled feature F4^distilled is extracted from F3^coarse by a 3×3 BSConv.

[0038] Feature fusion enhancement: F1^distilled~F4^distilled are concatenated along the channel dimension and dimensionality reduced by 1×1 convolution to obtain the fused feature F_fuse; then the enhanced spatial attention (ESA) module and the contrastive perceptual channel attention (CCA) module are passed in sequence to enhance the representation ability of key spatial locations and important feature channels, and the enhanced feature F_enhance is output.

[0039] Residual connection adjustment: A learnable scaling factor α1 (α1∈[0,1]) is introduced to adjust the weights of the input feature Fin and then perform element-wise addition with F_enhance to obtain the final output F_out of MSDB. This ensures the integrity of the feature information and enhances the gradient propagation ability of the network.

[0040] Frequency Enhancement Module (FMEB): Optimizes the Mamba architecture through internal frequency analysis, achieving a balance between long-range dependency modeling and local information compensation.

[0041] Input feature processing: The two input features are concatenated along the channel dimension and dimensionality reduced by 1×1 convolution to obtain feature P0, thereby reducing subsequent computational overhead;

[0042] Three-parallel-branch design: P0 is normalized using LayerNorm to obtain P_LN, and then processed through three parallel branches:

[0043] Long-range dependency modeling branch: P_LN is processed through a linear layer φ1, a 3×3 depthwise convolution (DWConv), a SiLU activation function, a 2-D selective scan (SS2D) module, and LayerNorm normalization to generate features P1 for modeling long-range spatial dependencies;

[0044] Modulation-gated branch: P_LN is passed through the linear layer φ2 and the SiLU activation function to generate an adaptive gating signal P2, which is used to adjust the feature contribution weights of the other two branches;

[0045] Frequency enhancement branch: After passing through the linear layer φ3, P_LN is transformed to the frequency domain by 2-D real fast Fourier transform (FFT). The frequency features are extracted and refined by two 1×1 convolutions. Then, the GELU activation function is used to dynamically suppress low redundant frequency components and enhance key high frequency components (corresponding to image details). Finally, it is transformed back to the spatial domain by inverse 2-D real FFT to obtain the frequency enhancement feature P3.

[0046] Feature fusion and optimization: Element-wise multiplication is performed between the gated signal P2 and P1 and P3 respectively to adaptively adjust the contribution of long-range dependent features and local detail features. Then, the two are added element-wise and integrated through the linear layer φ4 to obtain the enhanced feature P_enhance.

[0047] Residual connection output: P_enhance is further refined by 3×3 BSConv, and after weighting P0 with a learnable scaling factor α2 (α2∈[0,1]), element-wise addition is performed to obtain the final output P_out of FMEB, realizing the deep integration of feature information.

[0048] Furthermore, the aforementioned model training dataset configuration includes:

[0049] Training sets: Two publicly available remote sensing datasets, UCMerced and RSCCN7, are used. The UCMerced dataset contains 21 remote sensing scene categories, with 100 images of 256×256 pixels per category; the RSCCN7 dataset contains 7 remote sensing scene categories, with a total of 2800 images of 400×400 pixels. Both datasets cover diverse remote sensing scenes such as dense buildings, farmland, roads, mountains, and rivers, which are suitable for the model's feature learning needs.

[0050] Test sets are divided into remote sensing-specific test sets and general benchmark test sets. The remote sensing-specific test sets are subsets of UCMerced and RSCCN7, used to verify the model's specific performance in remote sensing image tasks; the general benchmark test sets include Set5, Set14, BSD100, and Urban100, used to verify the model's generalization ability on non-remote sensing natural images and comprehensively evaluate the model's super-resolution performance.

[0051] Furthermore, the training parameter settings include:

[0052] Loss function: The L1 loss function is used to measure the pixel-level difference between the SR and HR images. The optimization objective is to minimize the mean absolute error, and the formula is:

[0053]

[0054] in Here, N is the model parameter set, and N is the total number of training samples. For the reconstructed SR image of the i-th LR image, For the corresponding HR image;

[0055] Optimizer: The Adam optimizer was selected, with β1=0.9 and β2=0.999, and an initial learning rate of 5×10⁻⁶. -4 The training batch size is 16;

[0056] Training epochs and learning rate strategy: The total number of training epochs is 2000, and the learning rate is halved at 1500 epochs to improve the stability and convergence accuracy of later training.

[0057] Training sample processing: Randomly crop 48×48 image patches from the HR image as training input to simulate feature learning scenarios in different local regions.

[0058] Training environment: Implemented based on the PyTorch deep learning framework, with NVIDIA GeForce RTX 3090 GPU selected for the hardware environment, supporting parallel computing to improve training efficiency.

[0059] Early stopping mechanism: When the validation set loss does not decrease for 8 consecutive epochs, training is automatically terminated and the current optimal model parameters are saved, effectively avoiding model overfitting and ensuring the model's generalization ability.

[0060] Furthermore, the reasoning steps include:

[0061] Preprocessing: The LR remote sensing images to be processed are first normalized to pixel values ​​(normalized to the [0,1] interval), without the need for additional data augmentation operations, ensuring the consistency of the inference process;

[0062] Model inference: The preprocessed LR image is input into the trained MFEM model, and the model performs a series of operations such as feature extraction, multi-scale thinning, fusion enhancement, and upsampling by passing through the shallow feature extraction module, 4 MSDB modules, 3 FMEB modules, and reconstruction module in sequence.

[0063] Post-processing and output: The pixel values ​​of the SR feature map output by the model are denormalized (restored to the 0~255 range) to obtain the final high-resolution remote sensing image. The inference process requires no manual intervention, supports batch processing, and is suitable for large-scale remote sensing image resolution upgrades.

[0064] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0065] Its lightweight design offers significant advantages and provides extremely high deployment flexibility. For example... Figure 5 As shown, the MFEM model proposed in this invention has a parameter size of only 304K and a computational cost (FLOPs) of only 1.39G, which has significant advantages over existing mainstream RSISR models: compared with the Transformer-based model TTST (18367K parameters, 32.7G FLOPs), the number of parameters is reduced by 98% and the computational cost by 96%; compared with OmniSR (793K parameters, 3.15G FLOPs), the number of parameters is reduced by 62% and the computational cost by 56%; compared with HAUNet (596K parameters, 1.90G FLOPs), the number of parameters is reduced by 49% and the computational cost by 27%. The extremely small parameter size and computational cost allow the model to be directly deployed on resource-constrained edge devices such as remote sensing monitoring terminals, portable observation equipment, and UAV-mounted terminals, without relying on high-performance servers, greatly expanding the scope of application scenarios.

[0066] The model exhibits excellent reconstruction accuracy and high feature restoration. Through multi-scale feature extraction using the MBRB module and frequency enhancement fusion mechanism using the FMEB module, the model can fully capture the complex target structure and subtle texture details of remote sensing images: In the UCMerced dataset ×2 scaling scenario, the PSNR reaches 34.48dB and SSIM reaches 0.9344, only 0.02dB lower than the current best model TTST, with a complexity reduction of orders of magnitude; In the RSCCN7 dataset ×2 and ×3 scaling scenarios, the PSNR reaches 30.29dB and 28.17dB respectively, and the SSIM reaches 0.8095 and 0.7102 respectively, both ranking first; In the general dataset Set5 ×2 scaling scenario, the PSNR reaches 38.02dB and SSIM reaches 0.9615, matching or surpassing the existing state-of-the-art lightweight models. The reconstructed images have clear target outlines and rich details, and can accurately restore the structural features of complex remote sensing scenes such as dense building clusters, irregular terrain boundaries, and multi-scale interweaving of ground features.

[0067] A balance between long-range and local information ensures strong reconstruction integrity. The FMEB module effectively solves the problem of local information loss when modeling long-range dependencies in traditional Mamba models through internal frequency analysis and dynamic interaction with the Mamba architecture: the SS2D module efficiently captures long-distance target associations (such as the correspondence between road extension trajectories, river flow directions and surrounding terrain), the FFT frequency enhancement branch accurately preserves local details (such as the edge texture of ground features and the outline of small facilities), and the gating mechanism adaptively adjusts the contribution weights of both, achieving an optimal balance between long-range dependencies and local details, avoiding the inherent defects of existing models that "emphasize the global over the local" or "emphasize details over associations".

[0068] The model exhibits strong generalization ability and broad scene adaptability. Through multi-scale feature extraction and attention mechanism optimization, it can simultaneously adapt to super-resolution tasks involving both remote sensing and natural images. For complex scenes commonly found in remote sensing images, such as dense buildings, irregular terrain, coexistence of multi-scale targets, and cloud occlusion, the model can effectively avoid feature loss and distortion, ensuring the structural integrity of the reconstructed image. For scenes in natural images, such as urban buildings, ecological environments, and vegetation, the model also achieves excellent texture restoration and edge smoothing effects. No additional scene-specific optimization is required, significantly reducing the adaptation cost in practical applications.

[0069] The model boasts fast inference speed and high real-time performance. Based on lightweight convolution and an optimized Mamba architecture, it avoids complex self-attention calculations, significantly improving inference efficiency. On an NVIDIA GeForce RTX 3090 GPU, when scaling a 64×64 LR image by 4, the inference time is only 18.27ms, more than 15 times faster than TTST (278.61ms) and 46% faster than OmniSR (33.59ms). This meets the time-sensitive requirements of scenarios such as disaster early warning, real-time monitoring, and emergency rescue, enabling rapid high-definition reconstruction and instant analysis of low-resolution remote sensing images, providing timely data support for decision-making. Attached Figure Description

[0070] Figure 1 This is a diagram of the overall architecture of the MFEM network.

[0071] Figure 2 A schematic diagram of the structure of multi-scale blueprint residual blocks (MBRB);

[0072] Figure 3 This is a schematic diagram of the Multiscale Separation Distillation Module (MSDB).

[0073] Figure 4 This is a schematic diagram of the Frequency Mamba Enhancement Module (FMEB).

[0074] Figure 5 This is a performance comparison chart for the models. Detailed Implementation

[0075] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0076] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0077] like Figure 1-4 As shown, this embodiment is based on the PyTorch deep learning framework and constructs an end-to-end multi-scale frequency enhancement Mamba network (MFEM). Through five key steps—data preprocessing, network architecture construction, core module construction, model training, and inference—it achieves accurate reconstruction from low-resolution (LR) remote sensing images to high-resolution (SR) remote sensing images. The specific technical solution uses "data preprocessing → feature extraction → multi-scale refinement → frequency enhancement fusion → upsampling reconstruction" as its core process, achieving both lightweight design and high accuracy through innovative module design. The network adopts a modular architecture, including a shallow feature extraction module, a multi-scale separation distillation module (MSDB), a frequency Mamba enhancement module (FMEB), and a reconstruction module. These modules work collaboratively to ensure comprehensive feature extraction and efficient fusion while minimizing model complexity. The method includes the following steps:

[0078] 1. Data Preparation

[0079] (1) Training set configuration

[0080] Dataset Selection and Splitting: Two publicly available remote sensing datasets, UCMerced and RSSCN7, were selected as training data sources. The UCMerced dataset contains 21 typical remote sensing scene categories (such as farmland, dense buildings, roads, airports, and mountains), with 100 256×256 pixel RGB images per category, totaling 2100 images. These are divided into a training set (1680 images) and a validation set (420 images) at an 8:2 ratio. The RSSCN7 dataset contains 7 remote sensing scene categories (such as cities, forests, rivers, and deserts), totaling 2800 400×400 pixel RGB images. These are divided into a training set (1960 images) and a validation set (840 images) at a 7:3 ratio. Both datasets cover diverse land cover types and complex imaging environments, adapting to the model's multi-scale feature learning needs.

[0081] LR Image Generation: Using the HR image as a reference, bicubic interpolation is used to perform downsampling by ×2, ×3, and ×4 times respectively to generate LR images that correspond one-to-one with the HR images, constructing "LR-HR" paired training samples. For example, a 256×256 HR image is downsampled by ×4 to obtain a 64×64 LR image, simulating low-resolution scenes captured by edge devices in real-world applications.

[0082] Data augmentation operations: To improve the model's generalization ability and avoid overfitting, the following augmentation strategies are performed on the LR images in the training set: random horizontal flipping (flipping probability 50%), random 90° rotation (including four angles randomly selected: 0°, 90°, 180°, and 270°); at the same time, the pixel values ​​of all images are divided by 255 and normalized to the [0,1] interval to eliminate numerical scale differences and ensure the stability and efficiency of gradient descent during training.

[0083] Training sample processing: During training, 48×48 image blocks are randomly cropped from the HR image, and the corresponding LR image blocks are obtained by bicubic interpolation downsampling to ensure the size consistency and feature specificity of the training samples.

[0084] (2) Test set configuration

[0085] Remote sensing-specific test sets: The validation sets of UCMerced and RSCCN7 are used as remote sensing-specific test sets, with 420 images in the UCMerced test set and 840 images in the RSCCN7 test set. These are used to verify the core performance of the model in remote sensing image super-resolution tasks, focusing on evaluating the reconstruction effect of complex ground structures and multi-scale targets.

[0086] General benchmark sets: The commonly used datasets in the field of super-resolution are Set5, Set14, BSD100, and Urban100. Set5 contains 5 high-detail images of 48×48 pixels, Set14 contains 14 multi-scene images of 128×128 pixels, BSD100 contains 100 natural scene images of 321×481 pixels, and Urban100 contains 100 dense building scene images of 256×256 pixels. These are used to verify the model's generalization ability on non-remote sensing natural images and to comprehensively evaluate the model's cross-scene adaptability.

[0087] Test sample preprocessing: Before the test, only pixel value normalization (to the [0,1] interval) is performed on the LR image, without data augmentation, to ensure the objectivity of the test results; if the LR image size is not an integer multiple of the target scaling factor, edge padding (fill value is 0) is used to the nearest integer multiple of the size, and the image is cropped back to the original size after the test to avoid edge effects affecting the evaluation results.

[0088] 2. Model Parameter Settings

[0089] (1) Shallow feature extraction module

[0090] Channel stacking times: n is preferably 4, that is, the input 3-channel LR image is stacked 4 times along the channel dimension to obtain a 12-channel feature map, which enriches the representation of shallow texture details.

[0091] 3×3 BSConv configuration: 12 input channels, 48 ​​output channels, stride 1, padding "same" (ensuring consistent input and output feature map sizes); BSConv consists of "1×1 convolution + depthwise convolution (DWConv)", with 48 kernels in the 1×1 convolution, 3×3 kernel size in the depthwise convolution, stride 1, and padding "same". It integrates cross-channel information and spatial features through kernel correlation to achieve lightweight feature extraction.

[0092] No additional activation function: The output of this module is directly input to the subsequent MSDB module, and the activation function is provided by the internal MSDB module.

[0093] (2) Multiscale blueprint residual blocks (MBRB)

[0094] Channel separation rule: The input features are divided into two sub-features P1(xi) and Q1(xi) along the channel dimension, each accounting for 1 / 2 of the total number of channels, to achieve parallel processing of two branches.

[0095] MBRB parameters for various specifications:

[0096] MBRB-a: P1(xi) is processed by a 3×3 BSConv (stride 1, padding "same", output channels 24), and Q1(xi) is processed by a 1×1 convolution (stride 1, padding "same", output channels 24); the outputs of the two branches are concatenated along the channel dimension (resulting in 48 channels), and residual connections are performed with the input feature xᵢ (element-wise addition), with no additional activation function.

[0097] MBRB-b: P1(xi) is processed by a 5×5 BSConv (stride 1, padding "same", number of output channels 24), and Q1(xi) is processed by a 1×1 convolution (stride 1, padding "same", number of output channels 24). The outputs of the two branches are concatenated and then activated by the GELU activation function to enhance the nonlinear expression. Finally, a residual connection is performed with the input feature xᵢ.

[0098] MBRB-c: P1(xi) is processed by a 5×5 BSConv (step size 1, padding with "same", number of output channels 24), and Q1(xi) is processed by a 3×3 BSConv (step size 1, padding with "same", number of output channels 24); the outputs of the two branches are concatenated and then activated by the GELU function, and then residually connected with the input feature xᵢ.

[0099] Common parameters: The number of depthwise convolution kernels for all BSConv is the same as the number of input channels, and the number of 1×1 convolution kernels is the same as the number of output channels for the corresponding branch of BSConv.

[0100] (3) Multiscale Separation Distillation Module (MSDB)

[0101] Feature distillation parameters: 1×1 convolution stride 1, padding with "same", halving the number of input feature channels (48→24), to obtain the distilled feature Fᵢ^distilled; the coarse feature Fᵢ^coarse is the MBRB output, with the number of channels remaining at 48.

[0102] ESA module parameters: 3×3 convolution (stride 1, padding "same", output channel number 48), dimensionality reduction ratio 4 (the number of channels is reduced from 48 to 12 through 1×1 convolution, and then restored to 48 through 3×3 convolution), activation function is ReLU, and key region features are enhanced through spatial attention weights.

[0103] CCA module parameters: The feature map is compressed into a 1×1×C vector through global average pooling, then passed through two layers of 1×1 convolution (the first layer reduces the dimensionality by 8, 48→6; the second layer restores the original number of channels by 6→48), and then activated by ReLU in between. Finally, channel attention weights are generated through sigmoid to highlight important feature channels.

[0104] Learnable scaling factor α1: Initial value is set to 0.1, with a range of [0,1]. The contribution weight of the residual connection is adaptively adjusted through backpropagation.

[0105] (4) Frequency Mamba Enhancement Module (FMEB)

[0106] Input processing parameters: The number of channels after concatenating the two input features is 96. After 1×1 convolution (stride 1, padding "same"), the dimensionality is reduced to 48, resulting in feature P0.

[0107] Three-branch parameters:

[0108] Long-range dependency modeling branch: Linear layer φ1 maps P_LN (48 channels) to 256 channels, 3×3DWConv (step size 1, padding "same"), SiLU activation function, SS2D module (hidden layer dimension 256, scan direction row priority, number of heads 8), and outputs P1 (48 channels) after LayerNorm normalization.

[0109] Modulation gating branch: The linear layer φ2 maps P_LN to 48 channels, and the SiLU activation function generates the gating signal P2 (48 channels).

[0110] Frequency enhancement branch: Linear layer φ3 maps P_LN to 48 channels, 2-D real FFT transforms it to the frequency domain, two 1×1 convolutions (both 48 channels, stride 1, padding "same"), GELU activation in the middle, and inverse 2-D real FFT transforms it back to the spatial domain to obtain P3 (48 channels).

[0111] Learnable scaling factor α2: Initial value is set to 0.1, range is [0,1], adaptively adjusts the contribution weight of residual connections; 3×3 BSConv (step size 1, padding "same", number of output channels 48) is used to refine P_enhance.

[0112] (5) Reconstruction Module

[0113] 3×3 Convolution Configuration: The first 3×3 convolution (48 input channels, 48 ​​output channels, stride 1, padding "same") is used to integrate high-level features; the second 3×3 convolution (48 input channels, 3×s² output channels, where s is a scaling factor, ×2 outputs 12 channels, ×3 outputs 27 channels, and ×4 outputs 48 channels) is used to map to the target number of channels.

[0114] PixelShuffle upsampling: The upsampling factor is consistent with the target scaling factor s (×2, ×3, ×4), which enlarges the feature map size by a factor of s and restores the number of channels to 3 (RGB).

[0115] Detail compensation: No additional bicubic interpolation fusion (because shallow features have already been added to deep features to compensate for details), directly output SR image.

[0116] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba, characterized in that, Includes the following steps: Step 1: Preprocess the input low-resolution LR remote sensing image to generate a high-resolution HR remote sensing image, and construct a training sample dataset including pairs of low-resolution LR remote sensing images and high-resolution HR remote sensing images. Step 2: Construct a multi-scale frequency-enhanced Mamba network (MFEM), including: The shallow feature extraction module is used to perform channel stacking and convolution operations on the input image to extract the shallow feature map F0; Several cascaded multiscale separation distillation modules (MSDB) are used to receive the output features of the previous module and progressively extract multiscale feature maps F1~F_N through the internal multiscale blueprint residual blocks (MBRB) and feature distillation mechanism. Each multiscale separation distillation module (MSDB) includes three sizes of multiscale blueprint residual blocks (MBRB-a, MBRB-b, MBRB-c), as well as an enhanced spatial attention (ESA) module and a contrastive perception channel attention (CCA) module. The group fusion consists of several frequency Mamba enhancement modules (FMEBs). Each frequency Mamba enhancement module (FMEB) receives at least two input features and performs feature fusion and optimization through internal parallel long-range dependency modeling branches, modulation gating branches, and frequency enhancement branches, outputting enhanced features. The reconstruction module is used to fuse the high-level features output by the last FMEB with the shallow feature map F0, and perform pixel shuffling upsampling to output a high-resolution remote sensing image. Step 3: Use the dataset from Step 1 to train a multi-scale frequency augmentation Mamba network (MFEM) to obtain a lightweight remote sensing image super-resolution model. Then, use this model to process low-resolution LR remote sensing images and output high-resolution HR remote sensing images.

2. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 1, characterized in that, In the multi-scale blueprint residual block (MBRB), the input features are progressively processed through MBRB-a, MBRB-b, and MBRB-c. After each processing, the features are distilled by halving the number of channels through a 1×1 convolution, generating multiple distilled features and intermediate coarse features. After concatenating all distillation features along the channel dimension, they are sequentially passed through the Enhanced Spatial Attention (ESA) module and the Contrastive Perception (CCA) module to obtain the enhanced feature F_enhance. After introducing a learnable scaling factor α1 to adjust the input feature, element-wise addition is performed with F_enhance to obtain the final output F_out of MSDB.

3. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 2, characterized in that, MBRB-a, MBRB-b, and MBRB-c all employ a channel separation strategy, uniformly dividing the input feature into two sub-features along the channel dimension, and using different sizes of Blueprint Separable Convolution BSConv for parallel processing: MBRB-a: the first sub-feature is processed by 3×3 BSConv, the second sub-feature is processed by 1×1 convolution, and the two are concatenated and then residually connected to the input feature. MBRB-b: The first sub-feature is processed by 5×5 BSConv, the second sub-feature is processed by 1×1 convolution, the two are concatenated and activated by GELU, and then residual connection is performed with the input features; MBRB-c: The first sub-feature is processed by 5×5 BSConv, the second sub-feature by 3×3 BSConv, the two are concatenated and activated by GELU, and then residual connection is performed with the input features.

4. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 1, characterized in that, The internal structure of the frequency Mamba enhancement module FMEB is as follows: Two input features are concatenated along the channel dimension to reduce dimensionality, resulting in feature P0. P0 is then normalized and fed into three parallel branches. Long-range dependency modeling branch: sequentially through linear layers, depthwise convolution, SiLU activation, 2-D selective scanning SS2D module, and layer normalization to generate feature P1; Modulation gated branch: sequentially activated by the linear layer and SiLU to generate gated signal P2; Frequency enhancement branch: The frequency enhancement feature P3 is generated by sequentially passing through a linear layer, 2-D real FFT, two 1×1 convolutions, GELU activation, and inverse 2-D real FFT. The gated signal P2 is multiplied element-wise by P1 and P3 respectively, and then added together. The result is then integrated by a linear layer to obtain P_enhance. After convolution thinning, P_enhance is added element-wise by P0 adjusted by the learnable scaling factor α2 to obtain the final output P_out of FMEB.

5. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 1, characterized in that, The shallow feature extraction module extracts shallow feature maps as follows: the input low-resolution image is stacked repeatedly along the channel dimension n times, n≥2, and then a 3×3 blueprint separable convolution BSConv is used to extract the shallow feature map F0.

6. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 5, characterized in that, The reconstruction module is configured to perform the following steps: first, integrate high-level features through a 3×3 BSConv, then perform element-wise additive fusion with F0, then map the number of feature channels to 3·s² through a 3×3 convolutional layer, where s is the upsampling factor, and finally achieve s-fold upsampling through a pixel shuffle operation.

7. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 1, characterized in that, In step 3, the multi-scale frequency augmentation Mamba network MFEM is trained, including: using high-resolution remote sensing images to construct low-resolution-high-resolution image pairs as training samples through bicubic interpolation downsampling; using the Adam optimizer to train the network end-to-end with the goal of minimizing the L1 loss function; and using random horizontal flipping and random 90° rotation for data augmentation during the training process.

8. The lightweight remote sensing image super-resolution method based on multi-scale convolution and frequency-enhanced Mamba as described in claim 7, characterized in that, When training the multi-scale frequency-enhanced Mamba network MFEM, the total training epochs are 2000, and the initial learning rate is 5×10⁻. 4 The learning rate is halved after 1500 epochs; the training batch size is 16, and 48×48 image patches are randomly cropped from high-resolution images as training input.

9. A lightweight remote sensing image super-resolution system based on multi-scale convolution and frequency-enhanced Mamba, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images