Mamba-based optical guided synthetic aperture radar image super-resolution reconstruction method and system
By employing the optical guidance method of Mamba networks and utilizing the visual state space equation for feature fusion, the problems of insufficient texture information recovery and speckle noise in synthetic aperture radar images are solved, achieving high-quality image reconstruction results.
Patent Information
- Application Number
- CN202510030759.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-08
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2045-01-08
AI Technical Summary
Existing single-image super-resolution methods struggle to fully utilize complementary information in optical images, especially in synthetic aperture radar images. They cannot effectively recover true texture information and structural details and suffer from significant speckle noise problems.
We employ an optical guidance method based on Mamba, which utilizes visual state space equations for long-distance modeling through shallow feature extraction, multimodal feature fusion, and high-resolution image reconstruction. This adaptively integrates cross-modal features, reduces noise interference, and restores texture details.
It achieves higher-quality super-resolution reconstruction of synthetic aperture radar images, recovers more realistic details, suppresses artifacts, and improves evaluation metrics and perception performance.
Smart Images
Figure CN119850422B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and image processing, specifically to a method and system for super-resolution reconstruction of synthetic aperture radar images based on Mamba optical guidance. Background Technology
[0002] Synthetic Aperture Radar (SAR) can image in all weather conditions and at all times, and has strong penetration capabilities, making it widely used in military reconnaissance, geological exploration, environmental monitoring, and disaster assessment. However, due to the inherent limitations of its imaging system and unavoidable interference, obtaining high-quality SAR images is relatively challenging. SAR images often exhibit significant speckle noise and insufficient resolution, severely affecting information extraction and interpretation, and making it difficult to meet the needs of practical applications.
[0003] To obtain high-quality SAR images, super-resolution (SR) technology has become a cost-effective solution compared to hardware improvements, as enhancing system configurations incurs significant costs. Recently, deep learning-based single-image super-resolution (SISR) methods have achieved good results by utilizing convolutional neural networks (CNNs) to learn the mapping from low-resolution (LR) to high-resolution (HR) images. For example, FSRCNN, based on SRCNN, introduces the structural similarity index (SSIM) into the loss function to improve reconstruction capabilities. Shen et al. proposed a residual CNN for polarimetric SAR image SR, utilizing deconvolution to reduce accuracy loss caused by interpolation and incorporating PReLU to preserve structural information. Guo et al. developed a residual attention-enhanced U-shaped network (RAAUNet) for SAR image restoration, which achieves good performance with an encoder-decoder architecture. LSRGAN is a lightweight GAN-based network that uses depthwise separable convolution (DSConv) to compress the GAN and applies the SeLU activation function and attention module to improve feature learning for SAR images. However, these methods face two main challenges when applied to SAR images: (1) limited information in low-quality SAR images leads to blurred details, and (2) significant speckle noise inherent in SAR imaging further reduces reconstruction quality. These problems become more pronounced at higher noise levels or with greater magnification.
[0004] Existing single-image super-resolution (SR) methods, such as RCAN, Ufomer, MambaIR, and SwinIR, struggle to recover true texture information, particularly structural details and edge information, without the support of optical images. The fundamental challenge in acquiring high-quality SAR images stems from the inherent limitations of SAR imaging mechanisms, such as coherent echo superposition, insufficient imaging resources, and radar system errors. In contrast, obtaining high-quality optical images is relatively easier. Furthermore, optical and SAR images of the same scene share content information and are complementary; utilizing the rich details in optical images to guide SAR super-resolution is a promising direction. This approach, by introducing additional spatial and structural information from the optical domain, has the potential to overcome the limitations of traditional single-image methods.
[0005] Recent research has explored Guided Image SR (GISR) methods, which incorporate additional information to supplement missing texture details in low-resolution images. While these methods have shown success in other areas such as thermally visible SR, GISR methods specifically designed for SAR images remain limited. Existing methods, such as OGSRN, attempt to bridge this gap by converting SAR images to optical images and then optimizing the network using high-quality optical images. Despite these advancements, current methods struggle to fully utilize the complementary information in optical images or lack specific mechanisms to handle the unique features of SAR images, often resulting in blurring and artifacts. Summary of the Invention
[0006] The technical problem to be solved by this invention is how to make full use of complementary information in optical images for super-resolution reconstruction of synthetic aperture radar images.
[0007] The present invention solves the above-mentioned technical problems through the following technical means:
[0008] A method for optically guided super-resolution reconstruction of synthetic aperture radar images based on Mamba, characterized by the following steps:
[0009] The steps of shallow feature extraction are as follows: SAR images and optical images are encoded into a unified feature space using convolution.
[0010] The steps of multimodal feature fusion are as follows: the extracted shallow features are used as inputs to the CFSM and FSAM modules, relevant and significant optical features are dynamically selected, and multimodal feature fusion is adaptively performed in the spatial and frequency domains according to the state space parameters to generate high-quality fused features.
[0011] The steps of high-resolution image reconstruction are: aggregating shallow features and fused features, and then upsampling the aggregated features to reconstruct the high-resolution image.
[0012] This invention leverages the advantages of long-distance modeling using the visual state space equation, progressively selects useful information from optical information and adaptively integrates cross-modal features, effectively utilizing spatial and frequency domain features, reducing noise interference while recovering more realistic texture details, and achieving superior performance.
[0013] As a preferred embodiment of the above scheme, the multimodal feature fusion step includes multiple multimodal feature fusions.
[0014] As a preferred embodiment of the above scheme, the specific process of the multimodal feature fusion step is as follows:
[0015] The feature extraction process of the CFSM module is as follows: First, the optical and SAR input features are linearly projected to expand the channel size to c and 2c, respectively; both sets of features are processed by depthwise convolution and SiLU activation; the process can be described as follows:
[0016]
[0017] SAR features are divided in half along the channel size, and are represented as follows: and Then they are sent to the two branches respectively;
[0018] In the first branch, The inputs are fed into the cross-selection scanning mechanism, where the linear projection layer processes them to generate the basic matrix: The model is given context awareness and can be modeled for remote spatial information; From hidden state Decoding optical features effectively filters optical information that is beneficial for SAR image reconstruction; this process can be represented as:
[0019]
[0020] in, and It is a discrete matrix; The optical input at time step t serves as the input feature for the cross-selection scanning mechanism. Indicates output features;
[0021] In the second branch, using The provided gating information refines the gating mechanism in the original SS2D to better guide the information flow; subsequently, features from both branches are aggregated with the Hadamard product. This process is summarized below:
[0022]
[0023] The feature extraction process of the FSAM module is as follows: The FSAM module integrates multimodal information in both the spatial and frequency domains, and finally combines the fused features from these two domains to enhance key high-frequency information; the feature processing procedures in the spatial and frequency domains are the same. Taking the spatial domain as an example: the FSAM module includes two branches. and First, the channel size is expanded to 2c using linear projection, followed by depthwise convolution and SiLU activation. Then, the channel size is divided into two halves, each input to one of the two branches. The specific process is as follows:
[0024]
[0025] In the first branch, to further facilitate the comprehensive exchange and integration of useful information, a parameter matrix is generated using the input features of SAR. ;use Iterative update of hidden state The feature representation within the hidden state is gradually enhanced through multiple iterations; subsequently, through... The updated hidden state is processed to extract more refined features; this mechanism ensures accurate reconstruction of SAR image details; this process can be represented as:
[0026]
[0027] In the second branch, a joint gating mechanism is implemented to enhance feature flow control, with gating information provided collaboratively by optical and SAR features. Subsequently, features from both branches are aggregated and linearly transformed to obtain the output. The process is summarized below:
[0028] .
[0029] As a preferred embodiment of the above scheme, in order to effectively capture local spatial features and reduce channel redundancy, while also considering the relatively sparse texture information in SAR images, a convolution method is used to replace the MLPs in the two Vmamba layers of the VSS block; the input feature x after global exploration and local enhancement can be expressed as:
[0030]
[0031] Here, x represents the input feature of RSSL, which can be a shallow feature. or fusion features ; and It is a learnable scaling parameter used for dynamic feature aggregation.
[0032] As a preferred option of the above scheme, the method for calculating the loss in the spatial domain and frequency domain is as follows:
[0033] For the loss of spatial domain, the traditional method is used. The loss is used as a constraint, and the loss is calculated as follows:
[0034]
[0035] in, The prediction results for the SR task, For GT SAR images, For spatial domain loss;
[0036] For the loss in the frequency domain, a two-dimensional FFT is first applied to the SR and HR images along the two-dimensional space;
[0037]
[0038] The obtained complex number is decomposed into real and imaginary components; in this embodiment, these components are represented as... and These components are then stacked along the last dimension to form a multichannel tensor.
[0039]
[0040] in, and The SR and HR images are represented in the frequency domain, respectively; subsequently, the frequency domain representations of the SR and HR images are calculated. loss:
[0041]
[0042] Finally, the total training loss is defined as:
[0043]
[0044] in and They represent and Hyperparameters of weight loss.
[0045] This invention also provides an optically guided synthetic aperture radar image super-resolution reconstruction system based on Mamba, comprising:
[0046] Shallow feature extraction module: Uses convolution to encode SAR images and optical images into a unified feature space;
[0047] Multimodal feature fusion module: The extracted shallow features are used as inputs to the CFSM and FSAM modules, relevant and significant optical features are dynamically selected, and multimodal feature fusion is adaptively performed in the spatial and frequency domains according to the state space parameters to generate high-quality fused features;
[0048] High-resolution image reconstruction module: It aggregates shallow features and fused features, and then upsamples the aggregated features to reconstruct a high-resolution image.
[0049] As a preferred embodiment of the above scheme, the multimodal feature fusion module includes multiple multimodal feature fusions.
[0050] As a preferred embodiment of the above scheme, the multimodal feature fusion module is specifically executed as follows:
[0051] The feature extraction process of the CFSM module is as follows: First, the optical and SAR input features are linearly projected to expand the channel size to c and 2c, respectively; both sets of features are processed by depthwise convolution and SiLU activation; the process can be described as follows:
[0052]
[0053] SAR features are divided in half along the channel size, and are represented as follows: and Then they are sent to the two branches respectively;
[0054] In the first branch, The inputs are fed into the cross-selection scanning mechanism, where the linear projection layer processes them to generate the basic matrix: The model is given context awareness and can be modeled for remote spatial information; From hidden state Decoding optical features effectively filters optical information that is beneficial for SAR image reconstruction; this process can be represented as:
[0055]
[0056] in, and It is a discrete matrix; The optical input at time step t serves as the input feature for the cross-selection scanning mechanism. Indicates output features;
[0057] In the second branch, using The provided gating information refines the gating mechanism in the original SS2D to better guide the information flow; subsequently, features from both branches are aggregated with the Hadamard product. This process is summarized below:
[0058]
[0059] The feature extraction process of the FSAM module is as follows: The FSAM module integrates multimodal information in both the spatial and frequency domains, and finally combines the fused features from these two domains to enhance key high-frequency information; the feature processing procedures in the spatial and frequency domains are the same. Taking the spatial domain as an example: the FSAM module includes two branches. and First, the channel size is expanded to 2c using linear projection, followed by depthwise convolution and SiLU activation. Then, the channel size is divided into two halves, each input to one of the two branches. The specific process is as follows:
[0060]
[0061] In the first branch, to further facilitate the comprehensive exchange and integration of useful information, a parameter matrix is generated using the input features of SAR. ;use Iterative update of hidden state The feature representation within the hidden state is gradually enhanced through multiple iterations; subsequently, through... The updated hidden state is processed to extract more refined features; this mechanism ensures accurate reconstruction of SAR image details; this process can be represented as:
[0062]
[0063] In the second branch, a joint gating mechanism is implemented to enhance feature flow control, with gating information provided collaboratively by optical and SAR features. Subsequently, features from both branches are aggregated and linearly transformed to obtain the output. The process is summarized below:
[0064] .
[0065] As a preferred embodiment of the above scheme, in order to effectively capture local spatial features and reduce channel redundancy, while also considering the relatively sparse texture information in SAR images, a convolution method is used to replace the MLPs in the two Vmamba layers of the VSS block; the input feature x after global exploration and local enhancement can be expressed as:
[0066]
[0067] Here, x represents the input feature of RSSL, which can be a shallow feature. or fusion features ; and It is a learnable scaling parameter used for dynamic feature aggregation.
[0068] As a preferred option of the above scheme, the method for calculating the loss in the spatial domain and frequency domain is as follows:
[0069] For the loss of spatial domain, the traditional method is used. The loss is used as a constraint, and the loss is calculated as follows:
[0070]
[0071] in, The prediction results for the SR task, For GT SAR images, For spatial domain loss;
[0072] For the loss in the frequency domain, a two-dimensional FFT is first applied to the SR and HR images along the two-dimensional space;
[0073]
[0074] The obtained complex number is decomposed into real and imaginary components; in this embodiment, these components are represented as... and These components are then stacked along the last dimension to form a multichannel tensor.
[0075]
[0076] in, and The SR and HR images are represented in the frequency domain, respectively; subsequently, the frequency domain representations of the SR and HR images are calculated. loss:
[0077]
[0078] Finally, the total training loss is defined as:
[0079]
[0080] in and They represent and Hyperparameters of weight loss.
[0081] The advantages of this invention are:
[0082] 1. This invention proposes a novel hierarchical selection-fusion Mamba network for super-resolution of optically guided SAR images. This framework effectively integrates the advantages of state-space models in handling long-term dependencies with linear complexity, achieving an optimal balance between noise suppression, resolution enhancement, and computational efficiency. Specifically, this invention designs two modules: CFSM dynamically selects salient optical features through input-dependent state-space parameters, while FSAM promotes adaptive feature fusion in the spatial and frequency domains, effectively enhancing structural details while suppressing artifacts.
[0083] 2. Compared with existing state-of-the-art guided image super-resolution reconstruction methods and single image super-resolution reconstruction methods, the method of the present invention can recover more realistic details and has better performance in terms of evaluation metrics and perception. Attached Figure Description
[0084] Figure 1 This is a general framework diagram of the hierarchical selective fusion of Mamba networks in an embodiment of the present invention;
[0085] Figure 2 for Figure 1 Framework diagram of the mid-span modal feature selection module (CFSM);
[0086] Figure 3 for Figure 1 Mid-Frequency Spatial Adaptive Aggregation Module (FSAM) Framework Diagram
[0087] Figure 4 for Figure 1 Residual State Space Layer (RSSL)
[0088] Figure 5 for Figure 4 Framework diagram of the Visual State Space Module (VSSM). Detailed Implementation
[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0090] This embodiment proposes a hierarchical selective fusion Mamba network (HSFMamba) for optically guided super-resolution reconstruction of synthetic aperture radar (SAR) images. Specifically, leveraging the advantages of long-range modeling using visual state-space equations, it progressively selects useful information from optical data and adaptively integrates cross-modal features, effectively utilizing spatial and frequency domain features, reducing noise interference while recovering more realistic texture details, and achieving superior performance. To effectively extract key information from optical images and integrate cross-modal features, this embodiment designs two modules in a hierarchical manner: (1) a cross-modal feature selection module (CFSM), which dynamically adapts to SAR image features and a specially designed gating mechanism by inputting dependent state-space parameters. This module aims to filter useful information from optical images. Using a specially designed visual state-space model, features highly correlated with SAR images can be decoded, thereby eliminating redundancy while retaining key information.
[0091] (2) A Frequency-Spatial Adaptive Aggregation Module (FSAM) employs a dedicated selective scanning mechanism to adaptively aggregate cross-modal features from both the frequency and spatial domains in order to fully utilize key information in optical images, thereby achieving better structural detail recovery and enhancement of key high-frequency information. This method achieves comprehensive integration of multimodal features, enhancing structural information while reducing noise interference.
[0092] Based on the aforementioned CFSM and FSAM modules, the optically guided super-resolution reconstruction method for synthetic aperture radar images in this embodiment is as follows:
[0093] Step 1: Shallow feature extraction stage, which involves performing two types of shallow feature extraction for images from different patterns.
[0094] Given a low-quality (LQ) SAR image This embodiment uses 3×3 convolution to increase the number of channels in the function map. H and W represent the height and width of the input SAR image, and C is the number of channels.
[0095] For optical images Where n is 2 or 4, this embodiment uses the first four layers of a pre-trained VGG-19 network, followed by 1 or 2 layers of 3×3 convolutions to extract features.
[0096] Step 2: Perform multimodal feature fusion.
[0097] The Multimodal Feature Fusion (SIRM) stage, based on a state-space model, is a core component of HSFMamba. SIRM consists of two main modules: CFSM and FSAM, which are two distinct but progressive cross-modal visual state-space modules.
[0098] like Figure 2 As shown, CFSM employs a dedicated cross-selection scanning mechanism. First, the optical and SAR input features are linearly projected, expanding the channel sizes to c and 2c, respectively. Both sets of features are then processed using depthwise convolution and SiLU activation. This process can be described as follows:
[0099]
[0100] SAR features are divided in half along the channel size, and are represented as follows: and Then, they are sent to two branches respectively.
[0101] In the first branch, The inputs are fed into a dedicated cross-selection scanning mechanism, where linear projection layers process them to generate the fundamental matrix: The model is given context awareness and can model remote spatial information. From hidden state The optical feature representation in the middle decodes and effectively filters out optical information that is beneficial to SAR image reconstruction. This process can be represented as:
[0102]
[0103] here, and It is a discrete matrix. The optical input at time step t serves as the input feature for a dedicated cross-selection scanning mechanism. This indicates the output features.
[0104] In the second branch, using The provided gating information refines the gating mechanism in the original SS2D to better guide the information flow. Following this, features from both branches are aggregated with the Hadamard product. The process is summarized below:
[0105]
[0106] like Figure 3As shown, it integrates multimodal information in both the spatial and frequency domains, and finally combines the fused features from these two domains to enhance key high-frequency information and suppress speckle noise in SAR images.
[0107] For the frequency components, this embodiment first uses a Fourier transform to convert them to the frequency domain, and then performs the same operations as in the spatial domain. This embodiment uses the spatial domain as an example for detailed explanation. Similar to VSSM and CFSM, FSAM also consists of two branches. and First, the channel size is expanded to 2c using linear projection, followed by depthwise convolution and SiLU activation. Then, the channel size is split in half, with each half input to one of the two branches. The specific process is as follows:
[0108]
[0109] In the first branch, to further facilitate the comprehensive exchange and integration of useful information, a parameter matrix is generated using the input features of SAR. Specifically, such as Figure 3 As shown, this embodiment uses Iterative update of hidden state The feature representation within the hidden state is gradually enhanced through multiple iterations. Subsequently, through... The updated hidden state is then processed to extract finer features. This mechanism ensures accurate reconstruction of SAR image details. This process can be represented as:
[0110]
[0111] In the second branch, a joint gating mechanism is implemented to enhance feature flow control, with gating information provided collaboratively by optical and SAR features. Subsequently, features from both branches are aggregated and linearly transformed to obtain the output. The process is summarized below:
[0112]
[0113] To effectively capture local spatial features and reduce channel redundancy, while also considering the relatively sparse texture information in SAR images, this embodiment replaces the two MLPs in the Vmamba within the VSS block with convolution, as follows: Figure 4 As shown. This method not only reduces the computational cost of the model but also better adapts to the characteristics of SAR images. The input feature x after global exploration and local enhancement can be expressed as:
[0114]
[0115] Here, x represents the input feature of RSSL, which can be a shallow feature. It can also be a fusion feature. . and It is a learnable scaling parameter used for dynamic feature aggregation.
[0116] like Figure 4 , 5 As shown, the input features of the VSSM (vision state space model) are activated by linear projection, depthwise convolution (DWConv), and SiLU. This process can be represented as follows:
[0117] x = SiLU(DWConv(Linear( x )).
[0118] It should be noted that, in order to achieve a good balance between performance and efficiency, this embodiment, through multiple experiments, ultimately designed four SIRM modules within the overall framework of the hierarchical selective fusion Mamba network, such as... Figure 1 As shown.
[0119] Step 3, High-Quality Reconstruction Stage: Reconstruct high-quality images to obtain HQ optical images and LR SAR images.
[0120] like Figure 1 As shown, this embodiment combines shallow features and fused features to reconstruct a high-quality image. Through long skip connections, HSFMamba can directly transmit low-frequency information to the reconstruction module, allowing SIRMs to focus on high-frequency details. To implement the reconstruction module, this embodiment uses sub-pixel convolutional layers to upsample features and reconstruct a high-resolution (HQ) SAR image. .
[0121] The working principle of super-resolution guided by HQ optical images and LR SAR images in this embodiment is as follows:
[0122] The input to HSFMamba is an HQ optical image and an LR SAR image, where the LR SAR image is obtained by downsampling the GT SAR image and adding noise, which is simulated as speckle noise using a Gaussian or gamma distribution.
[0123] The overall process is divided into three stages. The first stage is the shallow feature extraction stage, where convolutional methods are used to encode SAR and optical images into a unified feature space. This unified representation provides strong support for cross-modal information exchange and integration in subsequent stages. These shallow features are then input into CFSM and FSAM, dynamically selecting relevant and salient optical features, and adaptively fusing multimodal features in the spatial and frequency domains based on state space parameters to generate high-quality fused features. The final stage is the high-quality reconstruction stage, where the shallow features are aggregated with the fused features. Sub-pixel convolutional layers are then used to upsample the aggregated features, thereby reconstructing a high-resolution image.
[0124] This embodiment calculates the spatial and frequency domain losses, providing a more comprehensive optimization strategy for SR tasks. For the spatial domain loss, this embodiment uses the traditional... The loss is used as a constraint, and the loss is calculated as follows:
[0125]
[0126] in, This is the prediction result of the SR task in this embodiment. For GT SAR images, This represents the spatial domain loss.
[0127] For the loss in the frequency domain, this embodiment utilizes Fast Fourier Transform (FFT) to transform the SR and HR images from the spatial domain to the frequency domain. To achieve this, this embodiment first applies a two-dimensional FFT to the SR and HR images along the two-dimensional space.
[0128]
[0129] The obtained complex number is decomposed into real and imaginary components. In this embodiment, these components are represented as... and These components are then stacked along the last dimension to form a multichannel tensor.
[0130]
[0131] in, and The SR and HR images are represented in the frequency domain, respectively. Subsequently, this embodiment calculates the frequency domain representations of the SR and HR images. loss:
[0132]
[0133] Finally, the total training loss is defined as:
[0134]
[0135] in and They represent and Hyperparameters of weight loss.
[0136] To validate the effectiveness of the proposed FSAM, we designed several comparative experiments, including non-aggregation, spatial domain-only aggregation, and spatial and frequency domain aggregation (FSAM). To quantitatively evaluate the quality of SAR images, we used two commonly used metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). Higher PSNR and SSIM values are better. The experimental results are shown in Table 1. FSAM significantly improved performance compared to the baseline, with PSNR increasing by 0.62 (from 26.16 to 26.78) and SSIM increasing by 0.0310 (from 0.7548 to 0.7858). Notably, while spatial domain aggregation alone brought significant improvements, this method struggled to achieve complete fusion. When frequency domain information was further introduced, PSNR increased again by 0.06 (from 26.72 to 26.78), and SSIM improved by 0.001 (from 0.7848 to 0.7858). These results highlight the crucial role of frequency domain information in achieving high-quality super-resolution results.
[0137] To demonstrate the effectiveness of the proposed CFSM, we conducted two sets of experiments. As shown in Table 1, the results show that, based on FSAM, applying the proposed CFSM improves PSNR by 0.03 (from 26.78 to 26.81) and SSIM by 0.001 (from 0.7858 to 0.7868). Even when FSAM is fused only in the spatial domain, the introduction of CFSM further enhances performance. This indicates that CFSM can effectively select useful information from optical features while filtering out unimportant components.
[0138] Table 1
[0139]
[0140] To verify the effectiveness of the second stage, we investigated the impact of the number of SIRMs on the method's performance. As shown in Table 1, with the increase of the number of SIRMs, the number of model parameters and computational cost gradually increased, while PSNR and SSIM also improved. However, after setting SIRM to 4, the improvement in PSNR and SSIM became very limited. Although SSIM improved slightly when increasing to settings 5 and 6, the performance improvement was negligible compared to the corresponding increase in computational complexity and number of parameters. This indicates that further increasing model complexity has reached its limit in improving performance. Therefore, we ultimately chose to set the number of SIRMs to 4 to achieve a better balance between performance and computational resource consumption. At this point, PSNR reached 26.81 and SSIM was 0.7868, demonstrating good performance.
[0141] Table 2
[0142]
[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for super-resolution reconstruction of synthetic aperture radar images based on Mamba optical guidance, characterized in that, Includes the following steps: The steps of shallow feature extraction are as follows: SAR images and optical images are encoded into a unified feature space using convolution. The steps of multimodal feature fusion are as follows: the extracted shallow features are used as inputs to the CFSM and FSAM modules, relevant and significant optical features are dynamically selected, and multimodal feature fusion is adaptively performed in the spatial and frequency domains according to the state space parameters to generate high-quality fused features. The CFSM module dynamically adapts to SAR image features and a gating mechanism by input-dependent state-space parameters, filtering useful information from optical images and decoding features related to SAR images, thereby eliminating redundancy while retaining key information; the FSAM module adaptively aggregates cross-modal features from the frequency and spatial domains to achieve structural detail recovery and enhancement of key high-frequency information. The steps of high-resolution image reconstruction are: aggregating shallow features and fused features, and then upsampling the aggregated features to reconstruct the high-resolution image.
2. The Mamba-based optically guided super-resolution reconstruction method for synthetic aperture radar images according to claim 1, characterized in that, The multimodal feature fusion step includes multiple multimodal feature fusions.
3. The Mamba-based optically guided super-resolution reconstruction method for synthetic aperture radar images according to claim 1 or 2, characterized in that, The specific process of the multimodal feature fusion steps is as follows: The CFSM module feature extraction process is as follows: First, the optical and SAR input features are linearly projected to expand the channel size to c and 2c, respectively; both sets of features are processed using depthwise convolution and SiLU activation; the process is described as follows: SAR features are divided in half along the channel size, and are represented as follows: and Then they are sent to the two branches respectively; In the first branch, The inputs are fed into the cross-selection scanning mechanism, where the linear projection layer processes them to generate the basic matrix: The model is given context awareness and can be modeled for remote spatial information; From hidden state The optical feature representation in the middle decodes effectively filters optical information that is beneficial for SAR image reconstruction; this process is represented as: in, and It is a discrete matrix; The optical input at time step t serves as the input feature for the cross-selection scanning mechanism. This represents the output features after CFSM; In the second branch, utilizing The provided gating information refines the gating mechanism in the original SS2D to better guide the information flow; subsequently, features from both branches are aggregated with the Hadamard product; this process is summarized below: The feature extraction process of the FSAM module is as follows: The FSAM module integrates multimodal information in both the spatial and frequency domains, and finally combines the fused features from these two domains to enhance key high-frequency information; the feature processing procedures in the spatial and frequency domains are the same. Taking the spatial domain as an example: the FSAM module includes two branches. and First, the channel size is expanded to 2c using linear projection, followed by depthwise convolution and SiLU activation. Then, the channel size is divided into two halves, each input to one of the two branches. The specific process is as follows: In the first branch, to further facilitate the comprehensive exchange and integration of useful information, a parameter matrix is generated using the input features of SAR. ;use Iterative update of hidden state The feature representation within the hidden state is gradually enhanced through multiple iterations; subsequently, through... The updated hidden state is then processed to extract finer features; this mechanism ensures accurate reconstruction of SAR image details; this process is represented as: In the second branch, a joint gating mechanism is implemented to enhance feature flow control, with gating information provided collaboratively by optical and SAR features. Subsequently, features from both branches are aggregated and linearly transformed to obtain the output. The process is summarized below: 。 4. The Mamba-based optically guided super-resolution reconstruction method for synthetic aperture radar images according to claim 3, characterized in that, To effectively capture local spatial features and reduce channel redundancy, while also considering the relatively sparse texture information in SAR images, a convolutional method was used to replace the two MLPs in the Vmamba within the VSS block. The input feature x after global exploration and local enhancement is expressed as: Here, x represents the input features of RSSL, which are shallow features. or fusion features ; and It is a learnable scaling parameter used for dynamic feature aggregation.
5. The Mamba-based optically guided super-resolution reconstruction method for synthetic aperture radar images according to claim 3, characterized in that, The methods for calculating the loss in the spatial and frequency domains are as follows: For the loss of spatial domain, the traditional method is used. The loss is used as a constraint, and the loss is calculated as follows: in, The prediction results for the SR task, For GT SAR images, For spatial domain loss; For the loss in the frequency domain, a two-dimensional FFT is first applied to the SR and HR images along the two-dimensional space; The resulting complex number is decomposed into real and imaginary components; these components are represented as... and These components are then stacked along the last dimension to form a multichannel tensor. in, and The SR and HR images are represented in the frequency domain, respectively; subsequently, the frequency domain representations of the SR and HR images are calculated. loss: Finally, the total training loss is defined as: in and They represent and Hyperparameters of weight loss.
6. A super-resolution reconstruction system for optically guided synthetic aperture radar images based on Mamba, characterized in that, include: Shallow feature extraction module: Uses convolution to encode SAR images and optical images into a unified feature space; Multimodal feature fusion module: The extracted shallow features are used as inputs to the CFSM and FSAM modules, relevant and significant optical features are dynamically selected, and multimodal feature fusion is adaptively performed in the spatial and frequency domains according to the state space parameters to generate high-quality fused features; The CFSM module dynamically adapts to SAR image features and a gating mechanism by input-dependent state-space parameters, filtering useful information from optical images and decoding features related to SAR images, thereby eliminating redundancy while retaining key information; the FSAM module adaptively aggregates cross-modal features from the frequency and spatial domains to achieve structural detail recovery and enhancement of key high-frequency information. High-resolution image reconstruction module: It aggregates shallow features and fused features, and then upsamples the aggregated features to reconstruct a high-resolution image.
7. The Mamba-based optically guided synthetic aperture radar image super-resolution reconstruction system according to claim 6, characterized in that, The multimodal feature fusion module includes multiple multimodal feature fusions.
8. The Mamba-based optically guided synthetic aperture radar image super-resolution reconstruction system according to claim 6 or 7, characterized in that, The specific execution process of the multimodal feature fusion module is as follows: The CFSM module feature extraction process is as follows: First, the optical and SAR input features are linearly projected to expand the channel size to c and 2c, respectively; both sets of features are processed using depthwise convolution and SiLU activation; the process is described as follows: SAR features are divided in half along the channel size, and are represented as follows: and Then they are sent to the two branches respectively; In the first branch, The inputs are fed into the cross-selection scanning mechanism, where the linear projection layer processes them to generate the basic matrix: The model is given context awareness and can be modeled for remote spatial information; From hidden state The optical feature representation in the middle decodes effectively filters optical information that is beneficial for SAR image reconstruction; this process is represented as: in, and It is a discrete matrix; The optical input at time step t serves as the input feature for the cross-selection scanning mechanism. This represents the output features after CFSM; In the second branch, utilizing The provided gating information refines the gating mechanism in the original SS2D to better guide the information flow; subsequently, features from both branches are aggregated with the Hadamard product; this process is summarized below: The feature extraction process of the FSAM module is as follows: The FSAM module integrates multimodal information in both the spatial and frequency domains, and finally combines the fused features from these two domains to enhance key high-frequency information; the feature processing procedures in the spatial and frequency domains are the same. Taking the spatial domain as an example: the FSAM module includes two branches. and First, the channel size is expanded to 2c using linear projection, followed by depthwise convolution and SiLU activation. Then, the channel size is divided into two halves, each input to one of the two branches. The specific process is as follows: In the first branch, to further facilitate the comprehensive exchange and integration of useful information, a parameter matrix is generated using the input features of SAR. ;use Iterative update of hidden state The feature representation within the hidden state is gradually enhanced through multiple iterations; subsequently, through... The updated hidden state is then processed to extract finer features; this mechanism ensures accurate reconstruction of SAR image details; this process is represented as: In the second branch, a joint gating mechanism is implemented to enhance feature flow control, with gating information provided collaboratively by optical and SAR features. Subsequently, features from both branches are aggregated and linearly transformed to obtain the output. The process is summarized below: 。 9. The Mamba-based optically guided synthetic aperture radar image super-resolution reconstruction system according to claim 8, characterized in that, To effectively capture local spatial features and reduce channel redundancy, while also considering the relatively sparse texture information in SAR images, a convolutional method was used to replace the two MLPs in the Vmamba within the VSS block. The input feature x after global exploration and local enhancement is expressed as: Here, x represents the input features of RSSL, which are shallow features. or fusion features ; and It is a learnable scaling parameter used for dynamic feature aggregation.
10. The Mamba-based optically guided synthetic aperture radar image super-resolution reconstruction system according to claim 8, characterized in that, The methods for calculating the loss in the spatial and frequency domains are as follows: For the loss of spatial domain, the traditional method is used. As a constraint, the loss is calculated as follows: in, The prediction results for the SR task, For GT SAR images, For spatial domain loss; For the loss in the frequency domain, a two-dimensional FFT is first applied to the SR and HR images along the two-dimensional space; The resulting complex number is decomposed into real and imaginary components; these components are represented as... and These components are then stacked along the last dimension to form a multichannel tensor. in, and The SR and HR images are represented in the frequency domain, respectively; subsequently, the frequency domain representations of the SR and HR images are calculated. loss: Finally, the total training loss is defined as: in and They represent and Hyperparameters of weight loss.