Cloud removal method and device based on light SAR remote sensing image fusion

By combining dual sparse Transformer and multi-sensory loss function, the problems of spectral differences and blurring in cloud removal in optical remote sensing images are solved, and high-quality cloudless image reconstruction is achieved, especially in preserving the details and structural features of the image under complex cloud interference.

CN120823119APending Publication Date: 2025-10-21WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510891517.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-30
Publication Date
2025-10-21

AI Technical Summary

Technical Problem

Existing cloud removal techniques in optical remote sensing images suffer from spectral differences, blurring, and artifacts between the reconstructed and real images. In particular, they are difficult to effectively remove clouds and preserve ground details, especially under complex cloud interference.

Method used

A cloud removal method based on dual sparse Transformers is adopted, which combines spatial domain and channel domain Transformers. The method suppresses the influence of cloud and speckle noise through sparse attention and gated local mechanisms, and optimizes the reconstruction effect through multi-sensory loss function.

Benefits of technology

It improves the performance of spatial texture restoration and spectral consistency reconstruction in cloud-covered areas, generating cloudless images with rich details and clear structure, and is suitable for high-quality remote sensing image reconstruction under complex cloud interference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120823119A_ABST
    Figure CN120823119A_ABST
Patent Text Reader

Abstract

The invention provides an image cloud removal method and device based on optical SAR remote sensing image fusion, and the method comprises the steps: constructing an image cloud removal network, and sequentially setting an image fusion shallow feature extraction module, a two-dimensional deep feature extraction module and a cloudless image reconstruction module; the image fusion shallow feature extraction module performs preliminary feature interaction and fusion on the cloud optical image and the SAR image; the two-dimensional deep feature extraction module is used for extracting depth features by using a stacked double sparse Transform residual group, and the residual group comprises a sparse space Transform module and a sparse channel Transform module which are alternately connected; the cloudless image reconstruction module reconstructs a cloudless optical image through a channel attention mechanism and a deconvolution operation; and adopting a multi-perception loss function to supervise and train the image cloud removal network, and extracting a cloud-free optical image reconstruction result based on the trained image cloud removal network. According to the method, the cloud removal visualization effect and the quantitative index result of the cloud-containing optical remote sensing image are improved, and the spectral characteristics and the structure detail characteristics of the reconstructed image are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to an image cloud removal technical solution based on optical SAR remote sensing image fusion. Background Art

[0002] Optical imagery plays a vital role in the continuous monitoring of the Earth's surface. However, optical imagery is inevitably contaminated by clouds, which limits its use and impacts many continuous detection tasks. Therefore, cloud removal from optical imagery has become an important and valuable topic in remote sensing imagery. Depending on the degree of cloud contamination, cloud removal from optical imagery can be categorized into two types: thin cloud removal and thick cloud removal. Thin cloud removal methods typically utilize uncontaminated areas within the same image for auxiliary removal, using methods such as physical modeling and neural networks. In the case of thick cloud removal, due to the severe cloud contamination, auxiliary information is often required to aid in the reconstruction of cloud-free optical images. Currently, the most commonly used auxiliary information is optical images from different time periods or image data of different modalities.

[0003] Multi-temporal methods typically utilize cloud-free optical imagery from adjacent periods within the same geographic region to assist in cloud removal. However, unpredictable natural events and weather conditions can cause significant changes in surface features or prevent the acquisition of optical imagery with low cloud cover for extended periods, complicating long-term observation missions. Multimodal methods typically utilize data from different sensors to assist in cloud removal. Images from some sensors are unaffected by cloud contamination and can fully reflect ground information within a specific area and at a specific time, thus serving as supplementary information for image reconstruction. Synthetic aperture radar (SAR) is the most popular auxiliary information in multimodal cloud removal methods. SAR is an active imaging method that can acquire ground information around the clock, regardless of weather conditions. While its imagery is less intuitive than optical imagery, it better preserves ground detail, facilitating cloud removal from optical imagery. However, due to the varying degrees of contamination in different bands within cloud-covered images and the random distribution of stratified clouds, both spatial and channel correlations within the images vary. Considering only one of these two methods can lead to spectral discrepancies between the reconstructed image and the true image. At the same time, the inherent speckle noise in SAR images will have an adverse effect on the cloud-free area of ​​the optical image during the reconstruction process. If it is not processed, the reconstructed image will be blurred and have artifacts, thus affecting its subsequent use.

[0004] Due to the differences between optical and SAR imagery, some researchers have used deep learning networks to directly convert SAR images into optical images, establishing a mapping relationship between the two through deep learning to achieve cloud-free image generation. The SAR2OPT method utilizes a cGAN network to train SAR images into optical images, demonstrating the feasibility of this method. Hwang et al. used a CycleGAN with structural similarity loss (SSIM) and perceptual loss to train paired SAR and optical images to obtain a mapping relationship between the two, thereby converting SAR images into optical images, demonstrating the potential of SAR-to-optical image conversion. Darbaghshahi et al. used two GAN networks for cloud removal: one to convert SAR images into optical images, and the other to generate cloud-free optical images from a cloud-filled optical image and the converted SAR image. Most SAR-to-optical image conversion methods are trained using GAN networks and their variants. While detailed features present in SAR images can be transferred to optical images, the lack of real optical images results in insufficient spectral fidelity. Furthermore, noise in the SAR imagery can also affect the reconstructed image.

[0005] Another approach is to directly fuse optical and SAR images and remove clouds through neural network learning. Optical and SAR images are stacked directly, and cloud-contaminated areas are repaired using the DSen-CR network. This network is based on a convolutional residual structure, and uses long skip connections to add cloud-free areas to the result. Li et al. proposed a hierarchical spectral and structure-preserving fusion network (HS2P). This network extracts features from optical and SAR images and integrates these features through a channel-wise residual attention module (RBCA). This network reconstructs images hierarchically, both spectrally and structurally, achieving excellent results.

[0006] These methods rely heavily on traditional convolutional modules, whose inherent flaws can lead to blurry and distorted reconstruction results. Due to the size of the convolution kernel, traditional convolution can only capture a receptive field within a specific area. To expand the receptive field, multiple layers of convolutional modules must be stacked, significantly reducing training effectiveness and efficiency and limiting global feature extraction capabilities. Furthermore, the fixed weights of each component lead to variations in reconstruction results for different images. Summary of the Invention

[0007] To meet the need for cloud-free optical images for continuous surface detection and to provide effective auxiliary information for optical images, the present invention provides an image declouding method based on dual sparse Transformers, combining the strong penetrating properties of SAR images. By combining spatial and channel domain Transformers, spatial and channel features are extracted from optical and SAR data, and feature interaction is performed to fully utilize the information from both image data. At the same time, sparse attention and gated local mechanisms are added to the self-attention structures of the two Transformers to suppress the influence of undesirable information such as speckle noise and cloud cover, while retaining useful features such as details.

[0008] The technical solution provided by the present invention is an image declouding method based on optical SAR remote sensing image fusion, which constructs an image declouding network, wherein an image fusion shallow feature extraction module, a two-dimensional deep feature extraction module and a cloud-free image reconstruction module are sequentially arranged. The image fusion shallow feature extraction module performs preliminary feature interaction and fusion on the cloud optical image and the SAR image; The dual-dimensional deep feature extraction module extracts deep features using a stacked dual sparse Transformer residual group. The residual group contains alternating sparse spatial Transformer modules and sparse channel Transformer modules to extract spatial domain features and channel domain features, respectively. A gated local feature extraction module is introduced into the sparse spatial Transformer module and the sparse channel Transformer module to dynamically enhance effective local features and suppress cloud and speckle noise interference. The cloud-free image reconstruction module reconstructs the cloud-free optical image through a channel attention mechanism and a deconvolution operation; The multi-perceptual loss function is used to supervise the training of the image declouding network, and the cloud-free optical image reconstruction results are extracted based on the trained image declouding network.

[0009] Moreover, the image fusion shallow feature extraction module connects the optical image and the SAR image in series, and then performs downsampling convolution and feature dimension-raising convolution to achieve preliminary fusion.

[0010] Moreover, the sparse spatial Transformer module adopts a window partitioning mechanism to calculate spatial attention, and the sparse channel Transformer module calculates the through-hole global attention based on the number of channels.

[0011] Moreover, the gated local feature extraction module generates main features and gated features through dual-branch convolution, and outputs dynamic local features through weighted activation function.

[0012] Moreover, the sparse spatial Transformer module and the sparse channel Transformer module further include a dual-gated feedforward network module, which fuses dual-dimensional features through a spatial gating branch and a channel gating branch.

[0013] Moreover, the cloud-free image reconstruction module optimizes spectral dependencies through channel attention.

[0014] Moreover, the loss function is composed of a weighted combination of L1 loss, LPIPS loss and DISTS loss.

[0015] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image declouding method based on optical SAR remote sensing image fusion as described above is implemented.

[0016] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image declouding method based on optical SAR remote sensing image fusion as described above.

[0017] On the other hand, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image declouding method based on optical SAR remote sensing image fusion as described above.

[0018] The present invention proposes an image declouding technology solution based on optical SAR remote sensing image fusion. First, a shallow feature extraction and fusion module is constructed to combine the spatial texture and channel representation information of optical images and SAR images to strengthen the initial feature interaction and improve the anti-interference ability; further, a deep feature fusion unit of sparse spatial Transformer and sparse channel Transformer is designed, and a gated local feature extraction module and a dual-gated feedforward network module are introduced to effectively suppress the interference caused by cloud and SAR speckle noise; finally, through the cloud-free image reconstruction module, the spatial texture restoration and spectral consistency reconstruction performance of cloud-covered areas are improved. By introducing multi-perceptual loss constraints, the spectral fidelity and structural detail integrity of the reconstructed image are further optimized. This solution not only improves the declouding visualization effect and quantitative index results of optical remote sensing images containing clouds, but also ensures the spectral characteristics and structural detail characteristics of the reconstructed image. It is especially suitable for high-quality remote sensing image reconstruction scenarios under complex cloud interference.

[0019] Compared with the prior art, the present invention has the following advantages: 1) This invention combines spatial-domain Transformers with channel-domain Transformers to extract spatial and channel features from optical and SAR images, respectively. Through feature interaction, it achieves deep fusion, leveraging the information advantages of both types of imagery. Compared to traditional cloud removal methods, this invention more accurately captures both texture details in optical images and structural information in SAR images. Consequently, it can reconstruct cloud-free images with rich detail and clear structure, even in complex cloud-interference scenarios.

[0020] 2) This paper proposes a local gating feature extraction module to address the common adverse information in optical and SAR images, such as cloud occlusion and speckle noise. Combined with a global feature sparsification mechanism, this module effectively suppresses adverse features during the fusion of global and local features to prevent them from affecting the image, while retaining useful information to improve the cloud removal effect.

[0021] 3) This paper proposes a multi-perceptual loss function, incorporating structural features and visual perceptual properties as multiple constraints for network training. This improves the spectral fidelity and structural information of reconstructed cloud-free images. This multi-perceptual loss design effectively overcomes the specific optimization deficiencies of traditional single loss functions, making the training process more efficient and ultimately generating cloud-free images that exhibit superior performance across all metrics. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is an overall framework diagram of the image declouding method based on optical SAR remote sensing image fusion provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a sparse spatial attention module provided by an embodiment of the present invention; Figure 3 2 is a schematic diagram of a gated local feature extraction module provided by an embodiment of the present invention; Figure 4 is a schematic diagram of a sparse channel attention module provided by an embodiment of the present invention; Figure 5 Schematic diagram of a dual-gated feedforward network module provided by an embodiment of the present invention; Figure 6 This is a sample image after reconstructing the SEN12MS-CR cloud-containing optical image provided by an embodiment of the present invention; Figure 7This is a sample image after reconstructing the LuojiaSET-OSFCR cloud-containing optical image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] like Figure 1 As shown, an embodiment of the present invention provides an image declouding method based on optical SAR remote sensing image fusion. The method constructs an image declouding network, in which an image fusion shallow feature extraction module, a two-dimensional deep feature extraction module, and a cloud-free image reconstruction module are sequentially arranged. By designing a dual sparse Transformer structure in the spatial domain and the channel domain, two-dimensional features are extracted and fused, and useless features are suppressed through the sparse attention and gated local feature extraction modules, thereby improving the declouding effect. In addition, a multi-perceptual loss function is used to supervise the training of the image declouding network. The cloud-free optical image reconstruction result is extracted based on the trained image declouding network to ensure the spectral fidelity and detail structure of the reconstruction result. While retaining the features of the cloud-free area in the original image, this method significantly improves the reconstruction effect of the clouded area.

[0026] The embodiment provides an image cloud removal method based on optical SAR remote sensing image fusion, which specifically includes the following steps: Step 1. Construct an image shallow fusion (ISF) module to perform preliminary information exchange between the cloud-covered optical image and the SAR image through image fusion and shallow feature extraction. The preferred solution adopted in the embodiment is to include a series layer, a 3×3 convolution layer with a stride of 2, and a 3×3 convolution layer with a stride of 1. First, the optical image and the SAR image are connected through the series layer, and then down-sampled through a 3×3 convolution layer with a stride of 2 to convert spatial information into channel information. Then, through a 3×3 convolution layer with a stride of 1, the features are converted to a higher dimension for better fusion expression.

[0027] Step 2. Build a two-dimensional deep feature extraction module, which consists of n1 stacked double sparse transformer residual groups (RDSTGs) and a 3×3 convolutional layer. The recommended value of n1 is 4 to 8.

[0028] The present invention proposes to construct a two-dimensional deep feature extraction module and extract deep features using a stacked dual sparse Transformer residual group. The residual group contains an alternately connected sparse spatial Transformer module and a sparse channel Transformer module to extract spatial domain features and channel domain features respectively.

[0029] In the embodiment, the following preferred implementation scheme of the dual-dimensional deep feature extraction module is further proposed: To fully utilize the available information in both the spatial and channel domains, the network undergoes dual sparse Transformer residual processing. This process consists of n2 sparse spatial Transformers and n2 sparse channel Transformers, connected in series and overlapping fashion. For example, a sparse spatial Transformer is followed by a sparse channel Transformer, which is then followed by another sparse spatial Transformer. Recommended values ​​for n2 are 4, 6, or 8. A 3×3 convolutional block is also included, and the result of this convolution is added to the residual input. The entire process can be expressed as:

[0030] in, represents the input of each double sparse Transformer residual group, Represents the output of each dual sparse Transformer residual group (RDSTG) module, represents the number of feature maps, and Represent the height and width of the feature map respectively, and denote the nth sparse spatial Transformer module and the nth sparse channel Transformer module, respectively, where n ranges from 1 to n2, Represents a 3×3 convolutional module.

[0031] Given the input of an nth double sparse Transformer module , n ranges from 1 to n2, the process of the sparse spatial Transformer module can be expressed as:

[0032]

[0033] Among them, SSA and DSFN represent the sparse spatial attention module and the dual-gated feedforward network module respectively, and LN represents the layer normalization operation. represents the output of the spatial sparse attention in the nth sparse channel Transformer module, Represents the output features of the nth sparse channel Transformer module.

[0034] Assume that the input feature of the nth double sparse Transformer module is , the process of the sparse channel Transformer module can be expressed as:

[0035]

[0036] Among them, SCA represents the sparse channel attention module, and the sparse channel Transformer module also uses the dual-gated feedforward network module DSFN and layer normalization operation LN. represents the output of the sparse channel attention in the nth sparse channel Transformer module, Represents the output features of the nth sparse channel Transformer.

[0037] The output of the first sparse spatial Transformer module serves as the input of the first sparse channel Transformer module, and the features processed by the first sparse channel Transformer module enter the second sparse spatial Transformer module. The two are alternately connected in series, enhancing the information transmission and communication of spatial features and channel features.

[0038] The sparse spatial attention module aims to extract spatial features while suppressing the influence of useless features and retaining useful features. Assume that the input feature of the sparse spatial attention module is First, the shape transformation reshape operation is used to divide it into windows, and then the query is generated through linear transformation ,key Sum , expressed as:

[0039]

[0040]

[0041] in, , , Represents the weight of the linear transformation.

[0042] See also Figure 2 , the embodiment further proposes that the sparse spatial attention module is specifically implemented as follows: A1) will get The local features of useful pixels are dynamically acquired through the gated local feature extraction module, which generates main features and gated features through dual-branch convolution and outputs dynamic local features weighted by an activation function.

[0043] like Figure 3 In the embodiment, the gated local feature extraction module is specifically implemented as follows: After two 3×3 convolution operations, the number of channels doubles. At this time, the features are split into the main branch and the gated branch along the channel dimension. The main branch and the gated branch processed by the activation function are multiplied to obtain the gated local features. The process of the gated local feature extraction module can be expressed as:

[0044]

[0045]

[0046] in, Indicates the intermediate feature with twice the number of channels, Indicates that the features are split along the channel dimension. , Represent the main features and gated features respectively, is the GELU activation function, Indicates the multiplication of elements at the same position in the matrix, is the final gated local feature.

[0047] A2) , Combined with the RELU activation function, the spatial attention is filtered. In the calculation of spatial global features, in order to reduce the amount of calculation, a window division mechanism similar to SwinTransformer is used to divide the entire image into non-overlapping windows for calculation. Spatial Self-Attention The calculation formula is:

[0048] in, is the dimension of the attention head.

[0049] The above formula uses the same method as other attention calculations, , Divide into multiple heads along the channel dimension, calculate them separately, and then sum them up. The number of channels in each head after division is obtained. The RELU activation function is used to replace the softmax activation function in the traditional self-attention calculation, so that it can better retain more important global information in the generation of spatial global features, while deleting useless information in the cloud layer to prevent it from having an adverse effect on the subsequent aggregation. Then the obtained sparse spatial attention is combined with After calculation, the global sparse spatial features are obtained.

[0050] A3) reshapes the global sparse spatial features obtained in step A2) and performs a matrix addition aggregation operation on them together with the gated local features obtained in step A1). The resulting aggregation is then subjected to a 3×3 convolution operation to obtain the output.

[0051] The sparse channel attention module aims to dynamically select features and choose more important information. Unlike the sparse spatial attention module, the sparse channel attention module pays more attention to the channel dimension. Assume that the input feature of the sparse channel attention module is After reshape transformation, linear transformation is used to generate query ,key Sum , expressed as:

[0052]

[0053]

[0054] in, , , Represents the weight of the linear transformation.

[0055] See also Figure 4 , the embodiment further proposes that the sparse channel attention module is specifically implemented as follows: B1) corresponds to the sparse spatial attention module, generating The gated local features are also obtained through the gated local feature extraction module, which is consistent with step A1. This gated local feature extraction module shares the same structure with the sparse spatial attention and sparse channel attention, suppressing local interference from clouds and SAR speckle noise.

[0056] B2) , Combine and filter the channel attention through the RELU activation function. In the sparse channel attention module, the channel global attention is calculated, and the channel number is used as the token in the Transformer to calculate the self-attention. Channel Attention The calculation formula is as follows:

[0057] in, Represents a learnable dynamic parameter. Here, RELU is used instead of the traditional softmax activation function to remove useless features in the channel dimension. Then the obtained sparse channel attention is combined with After calculation, the global sparse channel features are obtained.

[0058] B3) After performing a reshape operation on the global sparse channel features obtained in step B2), a matrix addition aggregation operation is performed on the gated local features obtained in step B1), and the resulting aggregation result is then subjected to a 3×3 convolution operation to obtain the output.

[0059] See also Figure 5 The dual-gated feedforward network module is designed to further fuse the features between the two dimensions of spatial channels, and fuse the two-dimensional features through the spatial gating branch and the channel gating branch. , through 1×1 convolution and GELU function, we get , Indicates the number of hidden channels. Then it will be sent to the channel gating branch below and the spatial gating branch above respectively.

[0060] The channel gate is divided into two parts along the channel. One part obtains the global features of each channel through global average pooling and 1×1 convolution, and the other part serves as the main branch.

[0061] The working process of the channel gating branch of the dual-gated feedforward network module can be expressed as:

[0062]

[0063] in .

[0064] The spatial gating branch of the dual-gated feedforward network module is also divided into the main branch and the depth convolution operation branch along the channel. Its formula is:

[0065]

[0066] in , DWConv represents the depthwise convolution operation.

[0067] Finally, the upper and lower results are added together and reconverted to the initial number of channels using a 1×1 convolution. The dual-gated feedforward network module (DSFN) dynamically acquires information from both spatial and channel dimensions for efficient information extraction and fusion.

[0068] Step 3. Construct a cloud-free image reconstruction module to further enhance the interactive fusion of two-dimensional features in optical and SAR images based on the features extracted by deep fusion. It includes a channel attention module, two 3×3 convolutional layers with a stride of 1, a deconvolution layer with a stride of 2, and two GELU activation functions. This module optimizes the dependency between different channels and improves the reconstruction effect of different bands in optical images.

[0069] Step 4. Construct a multi-sensory loss function to supervise model training. On the basis of and DISTS loss The learning-based perceptual image patch similarity (LPIPS) loss calculates the difference in high-dimensional features between the real cloud-free image and the image generated by the proposed model, while the deep image structure and texture similarity (DISTS) loss is used to measure the similarity in structure and texture between the two in high-dimensional features. Combining these three loss functions can better reconstruct the spectral and structural information of the optical image. Multi-perceptual loss function The formula is:

[0070] in, , LPIPS loss and DISTS loss The weight of .

[0071]

[0072] in, Represents the position of the model output in channel k The pixel value at Represents the position of the real image in channel k The pixel value at , H, W, and C represent the height, width, and number of channels of the image slice, respectively.

[0073]

[0074] Among them, l represents multiple perception layers, , Represent the features of the model output and the real image extracted at the lth layer, represents the channel weight, , Represents the height and width of the feature, Represents channel-by-channel multiplication.

[0075]

[0076] Among them, x, y represent the model output and the real image respectively, l represents the multiple convolutional layers in the VGG16 network, represents the structural similarity of the lth layer, and represents the texture similarity of the lth layer, , The weights representing the structural and texture similarities of the corresponding layers.

[0077] The datasets used in this embodiment include the SEN12MS-CR dataset released by the Technical University of Munich in Germany and the LuojiaSET-OSFCR dataset released by Wuhan University. The SEN12MS-CR dataset was publicly released by the Technical University of Munich in Germany, and the LuojiaSET-OSFCR dataset was publicly released by Wuhan University. Both are standard benchmark datasets for remote sensing image declouding tasks. This embodiment randomly selects samples from them to ensure the generalization of training and testing. The SEN12MS-CR dataset contains Sentinel-2 optical cloud and cloud-free images acquired in adjacent time periods, as well as Sentinel-1 SAR images. The optical images contain 13 bands, and the SAR images contain two channels, VV and VH. The size of each image is 256×256 pixels. The LuojiaSET-OSFCR dataset also contains Sentinel-2 optical cloud and cloud-free images acquired in adjacent time periods, as well as Sentinel-1 SAR images. The optical images contain 13 bands, and the SAR images contain two channels, VV and VH. The size is also 256×256 pixels. Unlike SEN12MS-CR, the optical images in the LuojiaSET-OSFCR dataset are images after atmospheric correction, and the SAR images are preliminarily de-noised, so the overall quality of the dataset is high. At the same time, the LuojiaSET-OSFCR dataset divides all triples into 5 categories based on the different cloud amounts in the cloud images, with the cloud amount range from 0% to 100% evenly divided, and there are 4000 triple data in each category, for a total of 20,000 data pairs. The present invention randomly selected 20,000 pairs, 2,000 pairs, and 500 pairs of triple images from the SEN12MS-CR dataset, and randomly selected 12,000 pairs, 1,000 pairs, and 500 pairs of triple images from the LuojiaSET-OSFCR dataset as training sets, validation sets, and test sets, respectively.

[0078] The cloud removal effect of the present invention on cloud-containing optical remote sensing images was compared with five existing methods published in authoritative journals, namely: [A. Meraner, P. Ebel, XX Zhu, and M. Schmitt, “Cloud removalin Sentinel-2 imagery using a deep residual neural network and SAR-opticaldata fusion,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 166,pp. 333–346, Aug. 2020.] (comparison method 1), [F. Xu et al., “GLF-CR: SAR-enhancedcloud removal with global–local fusion,” ISPRS Journal of Photogrammetry andRemote Sensing, vol. 192, pp. 268–278, Oct. 2022.] (comparison method 2), [Y. Li, F. Wei,Y. Zhang, W. Chen, and J. Ma, “HS2P: Hierarchical spectral and structure-preserving fusion network for multimodal remote sensing image cloud and shadow removal," Information Fusion, vol. 94, pp. 215–228, Jun. 2023.] (Comparison method 3), [Y. Wang, B. Zhang, W. Zhang, D. Hong, B. Zhao, and Z. Li, "CloudRemoval With SAR-Optical Data Fusion Using a Unified Spatial–SpectralResidual Network," IEEE Transactions on Geoscience and Remote Sensing, vol.62, pp. 1–20, 2024.] (Comparative Method 4), [J. Pan et al., "HDRSA-Net: Hybrid dynamicresidual self-attention network for SAR-assisted optical image cloud and shadow removal," ISPRS Journal of Photogrammetry and Remote Sensing, vol.218, pp. 258–275, Dec. 2024.] (Compare method 5). .

[0079] This paper selects four different land cover types, namely farmland, water, mountain and urban land, from two datasets for comparison. The corresponding reconstruction results are shown in Figure 2. Figure 6 、 Figure 7 shown. Figure 6 、、 Figure 7 From left to right, each column is: the cloud-containing optical image of Sentinel-2 to be restored, the SAR image of Sentinel-1, the cloud-free optical image of Sentinel-2 in the near-term as a standard reference, the results of comparison method 1, the results of comparison method 2, the results of comparison method 3, the results of comparison method 4, the results of comparison method 5, and the results of the method of the present invention. The comparison results show that compared with other methods, the image declouding method designed by the present invention based on optical SAR remote sensing image fusion presents the best results in spectral fidelity and detailed texture in different datasets and different types of objects. Other methods have varying degrees of blur and artifacts in the reconstruction process, but the present invention can suppress the influence of useless features through local gating mechanisms and global sparse mechanisms, minimizing blur and artifacts in the reconstructed image.

[0080] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0081] The image declouding electronic device based on optical SAR remote sensing image fusion provided by the present invention is described below. The image declouding electronic device based on optical SAR remote sensing image fusion described below and the image declouding method based on optical SAR remote sensing image fusion described above can be referenced to each other.

[0082] The electronic device may include a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may invoke logic instructions in the memory to execute an image declouding method based on optical SAR remote sensing image fusion, which primarily includes the software processing portion of the aforementioned steps.

[0083] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the software processing part of the image declouding method based on optical SAR remote sensing image fusion provided by the above methods.

[0085] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the software processing part of the image declouding method based on optical SAR remote sensing image fusion provided by the above-mentioned methods.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0087] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image declouding method based on optical SAR remote sensing image fusion, characterized in that: Construct an image declouding network, which sequentially sets up an image fusion shallow feature extraction module, a dual-dimensional deep feature extraction module, and a cloud-free image reconstruction module. The image fusion shallow feature extraction module performs preliminary feature interaction and fusion on the cloud optical image and the SAR image; The dual-dimensional deep feature extraction module extracts deep features using a stacked dual sparse Transformer residual group. The residual group contains alternating sparse spatial Transformer modules and sparse channel Transformer modules to extract spatial domain features and channel domain features, respectively. A gated local feature extraction module is introduced into the sparse spatial Transformer module and the sparse channel Transformer module to dynamically enhance effective local features and suppress cloud and speckle noise interference. The cloud-free image reconstruction module reconstructs the cloud-free optical image through a channel attention mechanism and a deconvolution operation; The multi-perceptual loss function is used to supervise the training of the image declouding network, and the cloud-free optical image reconstruction results are extracted based on the trained image declouding network.

2. The image declouding method based on optical SAR remote sensing image fusion according to claim 1, characterized in that: The image fusion shallow feature extraction module connects the optical image and the SAR image in series, and then performs downsampling convolution and feature dimension-raising convolution to achieve preliminary fusion.

3. The image cloud removal method based on optical SAR remote sensing image fusion according to claim 1 is characterized by: The sparse spatial Transformer module adopts a window partitioning mechanism to calculate spatial attention, and the sparse channel Transformer module calculates the through-hole global attention based on the number of channels.

4. The image cloud removal method based on optical SAR remote sensing image fusion according to claim 1, characterized in that: The gated local feature extraction module generates main features and gated features through dual-branch convolution, and outputs dynamic local features after weighting by an activation function.

5. The image cloud removal method based on optical SAR remote sensing image fusion according to claim 1, characterized in that: The sparse spatial Transformer module and the sparse channel Transformer module further include a dual-gated feedforward network module, which fuses dual-dimensional features through a spatial gating branch and a channel gating branch.

6. The image cloud removal method based on optical SAR remote sensing image fusion according to claim 1, characterized in that: The cloud-free image reconstruction module optimizes spectral dependencies through channel attention.

7. The image cloud removal method based on optical SAR remote sensing image fusion according to claim 1, characterized in that: The loss function is composed of a weighted combination of L1 loss, LPIPS loss and DISTS loss.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image declouding method based on optical SAR remote sensing image fusion as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image declouding method based on optical SAR remote sensing image fusion as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the image declouding method based on optical SAR remote sensing image fusion as described in any one of claims 1 to 7 is implemented.