SAR-assisted remote sensing image cloud removal method based on collaborative space-frequency fusion

Abstract

The invention discloses an SAR-assisted remote sensing image cloud removal method based on collaborative space-frequency fusion, belongs to the technical field of remote sensing image processing, and is used for image cloud removal. The method comprises the following steps: firstly, processing an input clouded optical image and an SAR image through pixel rearrangement and a shallow feature extraction network; then, a residual space-frequency fusion module (RSFF) is used for carrying out parallel processing on features in a space domain and a frequency domain, and global context information is captured through a two-way cross attention fusion unit so as to keep structural consistency; then, a boundary adaptive difference enhancement module (BDEM) calculates by using a structure texture enhancement unit and a SimAM attention mechanism, and generates a difference attention graph by using the structure difference between modalities, so as to sharpen the boundary of the object; the method comprises the following steps of: generating a dynamic cloud attention map by an SAR (Synthetic Aperture Radar)-Guided Attention Module (SAR-Guided Attention Module, SGAM), and performing accurate feature enhancement and guidance on a cloud layer shielding region; and finally, reconstructing a high-fidelity cloudless optical image through a backbone network formed by the residual dense blocks and an up-sampling network. According to the method, the defects in the aspects of global context modeling, cross-modal feature fusion and accurate guidance in the prior art are systematically solved, and the structural fidelity and spectrum consistency of the cloud-removed image are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically relating to a deep learning network model that uses synthetic aperture radar (SAR) data to assist in removing cloud cover from optical remote sensing images. Background Technology

[0002] Optical remote sensing imagery, with its high spatial resolution, rich multispectral information, and intuitive interpretability, has become an indispensable core data source in the field of Earth observation. It plays a fundamental role in key applications such as land use and land cover mapping, global environmental change monitoring, precision agricultural resource management, urban planning, and disaster emergency response. However, the effectiveness of this powerful tool has long been constrained by a common and severe natural factor—cloud cover.

[0003] The obstruction of optical signals by clouds is not an occasional interference, but a pervasive physical limitation. Statistics show that globally, especially in lush, moisture-rich tropical and subtropical regions, cloud cover often exceeds 50% annually, resulting in incomplete or even completely invalid remote sensing imagery. This data gap not only undermines the usability of individual images, but more seriously, it disrupts the continuity of time-series analysis, making continuous observation of dynamic surface processes (such as monitoring crop growth cycles and tracking deforestation dynamics) extremely difficult, thus introducing significant uncertainty. Therefore, cloud cover has become one of the major bottlenecks restricting the depth and breadth of applications of optical remote sensing technology.

[0004] Against this backdrop, research on cloud removal from remote sensing imagery has emerged, with its core objective being to recover clear and accurate surface information from cloud-contaminated images. This research has significant academic value and urgent practical needs.

[0005] From an academic perspective, cloud removal from remote sensing imagery is essentially a highly challenging and ill-posed image restoration problem. This is especially true for thick optical clouds, where the reflected signals from objects beneath are completely blocked, resulting in absolute information loss rather than simple signal attenuation. This transforms cloud removal from traditional image inpainting into complex image generation based on auxiliary information. Designing advanced algorithms to effectively fuse heterogeneous multi-source data (such as optical and radar data) to reconstruct missing geospatial information with high fidelity has not only driven technological advancements in computer vision, deep learning, and generative models but also deepened our understanding of the intrinsic correlations within multimodal remote sensing data.

[0006] From an application perspective, developing efficient and accurate cloud removal technologies has immeasurable practical value. It can revitalize massive amounts of historical remote sensing data archives that have been rendered idle due to cloud contamination, significantly improving the data utilization rate of existing and future Earth observation satellite systems. A stable and reliable cloud removal model can provide continuous, seamless, and high-quality data input for downstream applications, thereby ensuring the accuracy and reliability of various analytical models and providing stronger data support for addressing global challenges such as climate change, ensuring food security, and achieving sustainable development. To address this challenge, Synthetic Aperture Radar (SAR), with its all-weather, all-day operation and cloud-penetrating capabilities, has become a key auxiliary data source for compensating for the lack of optical data.

[0007] In recent years, although various SAR-assisted cloud removal methods have been proposed, a series of challenges remain in achieving high-fidelity reconstruction. First, existing models typically rely on stacked convolutional layers, which, limited by local receptive fields, struggle to effectively model the global context. This makes it difficult to guarantee the global consistency of ground structure when dealing with large areas of thick clouds. Second, due to the fundamental differences in imaging mechanisms between SAR and optical data, simple feature stitching or addition strategies cannot effectively fuse heterogeneous information, often leading to blurred cloud edges and artifacts. Finally, many methods fail to fully utilize the reliable structural information in SAR data to explicitly guide the reconstruction process, lacking a precise mechanism to focus model resources on the areas requiring restoration.

[0008] In existing technologies, single-image-based methods often lead to texture blurring and structural distortion; multi-temporal methods rely on high-precision temporal registration and are sensitive to changes in ground features. While existing SAR-assisted deep learning methods, such as DSe2-CR or GLF-CR, have made some progress, they still fall short in balancing global structural consistency and local boundary fidelity. Therefore, there is an urgent need for a cloud-free network model that can systematically address the challenges of global context modeling, cross-modal fusion, and accurate guidance. Summary of the Invention

[0009] The purpose of this invention is to provide a high-fidelity SAR-assisted image declouding network model (S2F2-Net) based on collaborative space-frequency fusion, in order to solve the problems of difficulty in structure recovery under large-area cloud layers, boundary blurring caused by heterogeneous feature fusion, and lack of accurate reconstruction guidance in the prior art.

[0010] A high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion includes:

[0011] S1 takes cloud-covered optical images and SAR images as input to the channel expansion and shallow feature extraction module. First, pixel reshuffle is performed, halving the spatial resolution and quadrupling the number of channels to expand the receptive field and reduce computational complexity. Then, shallow features are extracted through two cascaded 5x5 convolutional layers.

[0012] S2 inputs shallow optical features into the residual space-frequency fusion module (RSFF). This module processes features through parallel spatial and frequency branches. The spatial branch captures local details, while the frequency branch captures global dependencies in the frequency domain using Fast Fourier Transform (FFT). Both are dynamically modulated and aggregated with complementary information through a bidirectional cross-attention fusion unit (BCAF), effectively avoiding structural distortion.

[0013] S3 inputs the RSFF-enhanced optical features and SAR shallow layer features into the Boundary Adaptive Differential Enhancement Module (BDEM). This module first refines the features using a Structured Texture Enhancement Unit (STE-Unit), then calculates the absolute difference between the two to generate a modal difference map. The difference map is then optimized using the SimAM parametric attention mechanism to generate a "differential attention map," which adaptively sharpens object boundaries, achieving fine fusion of heterogeneous features.

[0014] S4 inputs the BDEM-enhanced optical features and the original SAR shallow features into the SAR Guided Attention Module (SGAM). This module generates a dynamic cloud attention map (SoftCloudAttentionMap) through a lightweight convolutional network and uses an additive attention enhancement mechanism to amplify the feature response of the cloud-covered area, providing accurate spatial guidance for reconstruction.

[0015] S5 inputs the features processed by the core modules described above into a backbone network containing multiple residual dense blocks (RDBs), leveraging dense connections to fully utilize hierarchical features. Deep and shallow features are fused through long skip connections to stabilize the training process;

[0016] S6 inputs the fused features into the upsampling network, restores the original resolution through convolutional layers and pixel reorganization layers (PixelShuffle), and performs a global residual connection with the original cloudy image to output the final predicted cloudless image.

[0017] According to claim 1, a high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion is characterized in that the specific formula for the pixel rearrangement operation is:

[0018]

[0019] in, This is the scaling factor.

[0020] In the residual space-frequency fusion module RSFF, the global feature extraction process of the frequency branch is as follows:

[0021] $$

[0022] By processing the real and imaginary parts in the frequency domain, the model is able to capture long-range spatial dependencies.

[0023] In the boundary adaptive difference enhancement module BDEM, SimAM is used to calculate the neuron energy function:

[0024]

[0025] The minimum energy obtained from this is used to calculate attention weights to enhance the differential features between modalities:

[0026]

[0027] The attention map and feature modulation process generated by the SAR-guided attention module SGAM are as follows:

[0028] $$

[0029] This mechanism does not require an external cloud mask truth value and achieves end-to-end self-supervised guidance.

[0030] Compared with existing technologies, this invention has the following advantages: The RSFF module proposed in this invention captures global context information in the frequency domain, ensuring the fidelity of ground object structure under large-area thick cloud cover, and solving the problem of limited receptive field in traditional convolutional networks; The BDEM module constructed in this invention explicitly models the structural differences between SAR and optical data, and adaptively sharpens object contours using difference attention maps, effectively eliminating boundary blurring and artifacts in cross-modal fusion; The SGAM module introduced in this invention utilizes the penetration characteristics of SAR to generate dynamic cloud attention maps, accurately amplifying the feature responses of cloud-contaminated areas, enabling the model to adaptively focus on key reconstruction areas, significantly improving the accuracy and robustness of cloud removal; Experimental results show that on the SEN12MS-CR benchmark dataset, this invention outperforms the state-of-the-art methods in quantitative indicators such as PSNR and SSIM, as well as visual quality. Attached Figure Description

[0031] Figure 1 This is the overall network architecture diagram (S2F2-Net) of the method proposed in this invention.

[0032] Figure 2 This is a schematic diagram of the residual space-frequency fusion module (RSFF) in this invention.

[0033] Figure 3 This is a schematic diagram of the boundary adaptive difference enhancement module (BDEM) in this invention.

[0034] Figure 4 This is a schematic diagram of the SAR-guided attention module (SGAM) in this invention.

[0035] Figure 5 This is a comparison chart showing the cloud removal effects of the present invention and existing technologies in different scenarios. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] A high-fidelity SAR-assisted image cloud removal network model (S2F2-Net) based on collaborative space-frequency fusion, the overall process of which includes:

[0038] S1 data preprocessing and shallow feature extraction input is a cloud-covered optical image. and the corresponding SAR image First, a pixel reshuffle operation is performed on both modal images, halving the spatial resolution and quadrupling the number of channels, as shown in the following formula:

[0039]

[0040] Subsequently, the rearranged feature maps are input into a shallow feature extraction network (SFE-Net) consisting of two cascaded 5x5 convolutional layers to obtain optical shallow features. and SAR shallow features .

[0041] S2 Residual Spatial-Frequency Fusion (RSFF) addresses the global context modeling problem by... Input the RSFF module. This module contains spatial and frequency branches. The spatial branch uses residual blocks to extract local features. :

[0042]

[0043] Frequency branching uses Fast Fourier Transform to map features to the frequency domain, and then performs an inverse transform back to the spatial domain to obtain the desired result. :

[0044]

[0045] The two interact through a bidirectional cross-attention fusion unit (BCAF):

[0046] $$

[0047] Finally, a unified spatiotemporal attention graph is generated through a multi-dimensional attention fusion unit. Weight the features and output .

[0048] S3 Boundary Adaptive Difference Enhancement (BDEM) aims to address boundary ambiguity in cross-modal fusion. and Input BDEM. First, the features are refined using parallel Structured Texture Enhancement Units (STE-Units) to obtain... and STE-Unit internally utilizes the SimAM module and differential edge extraction to generate weights:

[0049]

[0050] Next, the absolute difference plot between modes is calculated. And then SimAM was used again to optimize and obtain the differential attention map. :

[0051] $$

[0052] This difference map sharpens the boundaries by compensating for the optical feature flow through residual connections.

[0053] S4SAR Guided Attention (SGAM) aims to provide accurate spatial reconstruction guidance by concatenating BDEM output features with SAR features and inputting them into a lightweight convolutional network to generate a cloud attention map. :

[0054]

[0055] This attention map is used to perform weighted enhancement of optical features:

[0056]

[0057] In cloud-covered areas, the characteristic response is significantly amplified, forcing the network to focus on reconstructing that area.

[0058] The enhanced feature input after S5 deep feature extraction and reconstruction consists of a backbone network with four residual dense blocks (RDBs). Each RDB employs dense connections. Deep features shallow features Perform long-skip connection fusion:

[0059]

[0060] Finally, the resolution is restored through an upsampling network (convolution + pixelShuffle), and then added to the input clouded image to obtain the final predicted image. :

[0061]

[0062] S6 Model Training and Loss Function: This invention employs a composite loss function for training, including L1 loss, SSIM loss, and perceptual loss. and mask-guided reconstruction loss .in, Generated using SGAM As an adaptive weight, it is defined as:

[0063]

[0064] The total loss function is:

[0065]

[0066] The dataset used in this invention is SEN12MS-CR, a large-scale multimodal dataset for cloud removal. It provides paired synthetic aperture radar (SAR) Sentinel-1 measurements and cloudy and cloudless optical multispectral Sentinel-2 observations. For data preprocessing, for the Sentinel-2 data, the 13 band values ​​were truncated to [0, 10, 000] and then linearly normalized to [0, 1]. For the Sentinel-1 data, the VV and VH polarization channels were truncated to [-25, 0] and [-32.5, 0] respectively, followed by channel-level normalization. The dataset was divided into 20,468 pairs for training, 2,036 pairs for validation, and 2,036 pairs for testing.

[0067] This invention is implemented using the PyTorch deep learning framework, and all experiments were conducted on an NVIDIA RTX 4090 GPU. The training strategy uses the Adam optimizer, with parameters set to... and The initial learning rate is set to... The StepLR strategy is used to dynamically adjust the learning rate, which is halved every 5 epochs. The training batch size is 8, and the total iteration period is 30 epochs.

[0068] To verify the effectiveness of this invention, we conducted quantitative comparative experiments with other mainstream cloud removal algorithms, including single-source methods SpAGAN and SAR2OPT, and multi-modal fusion methods DSen2-CR, GLF-CR, and HPN-CR. Table 1 shows the quantitative comparison results of each method on the SEN12MS-CR dataset:

[0069] Table 1 Comparative Experimental Results

[0070]

[0071] As shown in Table 1, the method of this invention achieves optimal performance across all evaluation metrics. Specifically, the PSNR of this invention reaches 29.5105 dB, an improvement of approximately 1.5 dB compared to the suboptimal method, indicating higher pixel-level reconstruction accuracy. The SSIM reaches 0.8965, indicating better preservation of structural similarity. The MAE decreases to 0.0250, further confirming the reduction in reconstruction error. The SAM is only 0.1460, demonstrating a significant advantage in spectral fidelity.

[0072] In addition to quantitative analysis, we also conducted a visual comparison. Figure 5 This paper compares the cloud removal performance of our invention with other advanced methods in four representative complex scenarios. In scenario (a), facing a farmland area covered by thick clouds, single-source methods such as SpAGAN and SAR2OPT produce large-area blurring and spectral distortion. Although DSen2-CR and HSSP recover some green tones, the texture is too smooth. In contrast, our invention successfully recovers the dense texture of the dark farmland with clear edges and intact structure. In scenario (b), for mountainous areas where optical information is completely lost, other multimodal methods (such as HPN-CR and USSRN-CR) show obvious color deviations (such as pink or blue artifacts) and blurring. Our invention can better utilize the structural information of SAR to recover the contours of the ridges without obvious artifacts. In the semi-arid hilly scene of scenario (c), USSRN-CR still retains white haze and has low contrast. This invention completely removes smog and clearly restores multi-scale surface textures in the detailed areas marked by the red box, with spectral information highly consistent with the real image (Ground Truth). In the high-contrast urban scene of scene (d), although GLF-CR restores building outlines, the spectral characteristics are altered. This invention exhibits stronger spectral and structural fidelity in this scene.

[0073] In summary, this invention, through the spatial-frequency fusion mechanism of the RSFF module, the differential enhancement mechanism of the BDEM module, and the guidance mechanism of the SGAM module, outperforms the state-of-the-art methods in both quantitative indicators and visual quality, demonstrating its effectiveness and advancement in solving the task of SAR-assisted image declouding.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion, characterized in that, include: S1 takes cloud-covered optical images and SAR images as input, performs pixel rearrangement operations on them respectively, converts spatial information into channel information, and then inputs them into the shallow feature extraction network SFE-Net to obtain optical shallow features and SAR shallow features respectively. S2 inputs the shallow optical features into the residual space-frequency fusion module RSFF, which contains parallel spatial and frequency branches. The spatial branch uses residual units to extract local features, while the frequency branch uses fast Fourier transform to extract global context information. The two interact through a bidirectional cross-attention fusion unit BCAF to generate enhanced optical features to solve the structural consistency problem under large cloud areas. S3 inputs the RSFF-enhanced optical features and SAR shallow features into the boundary adaptive difference enhancement module BDEM. This module first refines the features through the parallel structure texture enhancement unit STE-Unit, then calculates the absolute difference between the two to generate an explicit modal difference map, and uses the SimAM parameterless attention mechanism to optimize the difference map, thereby calibrating the optical feature flow and sharpening the object boundary. S4 inputs the enhanced optical features output by BDEM and the original shallow SAR features into the SAR Guided Attention Module (SGAM). This module generates a dynamic cloud attention map through a lightweight convolutional network, and uses this attention map to weight and enhance optical features, so that the network focuses on the reconstruction of cloud-occupied areas. S5 extracts deep features from the backbone network composed of multiple residual dense blocks RDB after the features are processed by the core collaborative module, and fuses shallow features through long skip connections. S6 inputs the deep fusion features into the upsampling network to restore the original resolution, and adds it to the original clouded optical image through global residual connections to output the final cloud-free optical image. During training, a composite loss function including pixel-level L1 loss, structural similarity SSIM loss, perceptual loss, and mask-guided reconstruction loss is used for optimization.

2. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The pixel rearrangement operation will input a tensor Convert to output tensor Its mathematical expression is: ； In the formula, denoted as scaling factor, H, W, and C represent the height, width, and number of channels of the image, respectively, y and x are the spatial coordinates of the output tensor, and i and j are the offsets within the spatial block.

3. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The frequency branching process in the residual space-frequency fusion module RSFF includes: ； ； In the formula, and These represent the two-dimensional real-number fast Fourier transform and its inverse transform, respectively. and Representing the real and imaginary parts, Concat and Split represent channel concatenation and splitting, and Complex represents recombination into a complex tensor. This is a frequency domain convolution operation.

4. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The interaction process of the bidirectional cross-attention fusion unit (BCAF) includes: ； ； ； In the formula, and These are the characteristics of spatial branching and frequency branching, respectively. and For the generated cross-attention map, This represents element-wise multiplication, a mechanism that allows the two branches to adaptively aggregate complementary information.

5. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The energy function of the SimAM attention mechanism in the boundary adaptive difference enhancement module BDEM is defined as: ； The module's final output is an optimized difference attention map. for: ； In the formula, For the target neuron, For other neurons, This represents the total number of neurons within the channel. The regularization coefficient is . For energy diagrams, For the Sigmoid function, This indicates the operation of absolute difference.

6. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The cloud attention map generated by the SAR-guided attention module SGAM The feature enhancement process is as follows: $$ ； In the formula, and These represent the weights and biases of a lightweight convolutional network. Use the Sigmoid activation function; in cloud regions As the value approaches 1, the characteristic response is amplified; in cloudless areas... Approaching 0, retaining the original characteristics.

7. The high-fidelity SAR-assisted image cloud removal network model based on collaborative space-frequency fusion according to claim 1, characterized in that, The composite loss function include: ； in, To guide the reconstruction loss using a mask.

8. The dynamic cloud attention map generated by SGAM is used as a self-supervised mask, which is defined as: ； In the formula, , To predict the image, This is a true cloudless image. To prevent division by zero of constants.

Images