SAR fusion thick cloud removal method based on Transform and potential diffusion

Abstract

The invention relates to an SAR (synthetic aperture radar) fusion thick cloud removal method based on Transform and potential diffusion, which is used for realizing high-performance thick cloud removal processing on a satellite remote sensing image. The method comprises the steps that a satellite remote sensing image and an SAR image are input into a coarse cloud removal module, a cloud attention image and a coarse cloud-removed image are obtained through output, and the coarse cloud removal module comprises a three-layer cascade Swin Transformer layer and a space attention layer; the SAR image, the cloud attention image and the coarse cloud-removed image are input into an SAR fusion module, a fused image is obtained through output, and the SAR fusion module comprises a preset mixing formula; the fused image is input into a cloudless diffusion module, a cloudless image is obtained through output, and the cloudless diffusion module comprises a parameterized 50-step DDIM sampling iteration denoising process of conditional potential diffusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing optical image processing, in particular to a SAR fusion thick cloud removal method based on Transformer and latent diffusion. BACKGROUND

[0002] Satellite remote sensing images capture information of the Earth's surface through sensors installed on satellites in orbit. These images are crucial for a wide range of Earth observation and monitoring applications. However, the presence of clouds can severely interfere with the imaging process. Specifically, thick clouds can obscure the Earth's surface, making it difficult for satellites to collect accurate data. Addressing and mitigating the impact of clouds remains a challenging task. In the aspect of thick cloud removal, Synthetic Aperture Radar (SAR) has shown great potential in providing clear ground information, thus being widely used in this field. Various techniques have been developed to utilize SAR ground-guided data to improve the efficiency of thick cloud removal operations. SAR data is particularly valuable because it provides high-resolution surface information independent of weather and lighting conditions. Optical satellite remote sensing images have the advantages of high resolution, clear visual features, and strong interpretability, but are susceptible to cloud and fog coverage. SAR has the advantage of all-weather and all-time imaging, which can provide effective auxiliary information for cloud and fog removal in optical remote sensing images. However, the imaging methods of optical remote sensing images and SAR images are quite different, and there are significant differences in the expression of ground features in their images. Therefore, it is of great significance to achieve optical remote sensing image cloud removal under SAR fusion and obtain cloud-free optical remote sensing images.

[0003] Through literature retrieval of existing technologies, it is found that the patent with the application number 202311704091.2 and the patent name A progressive double-decoupling SAR-assisted remote sensing image thick cloud removal method obtains a three-data set of thick cloud-shielded optical remote sensing image, cloud-free reference image, and SAR image; constructs a progressive double-decoupling SAR-assisted remote sensing image thick cloud removal model; trains the progressive double-decoupling SAR-assisted remote sensing image thick cloud removal model; obtains and preprocesses the thick cloud-shielded optical remote sensing image to be removed and the corresponding SAR image data; and obtains the optical remote sensing image thick cloud removal result. However, this patent has problems such as insufficient remote sensing image processing performance. SUMMARY

[0004] Therefore, it is necessary to provide a SAR fusion thick cloud removal method and system based on Transformer and latent diffusion to achieve high-performance thick cloud removal processing of satellite remote sensing images.

[0005] To solve the above technical problems, the present application provides the following technical solutions: The application provides a SAR fusion thick cloud removal method based on a Transformer and potential diffusion, and the method comprises the following steps: Satellite remote sensing images and SAR images are input into a coarse cloud removal module to respectively output cloud attention images and coarse cloud removal images, wherein the coarse cloud removal module comprises three cascaded Swin Transformer layers and a spatial attention layer; The SAR images, the cloud attention images and the coarse cloud removal images are input into a SAR fusion module to output a fusion image, wherein the SAR fusion module comprises a preset hybrid formula; The fusion image is input into a cloud-free diffusion module to output a cloud-free image, wherein the cloud-free diffusion module comprises a parameterized iterative denoising process of conditional potential diffusion.

[0006] Preferably, the coarse cloud removal module comprises the following steps: the satellite remote sensing images and the SAR images sequentially pass through a channel splicing layer and a convolution layer to obtain initial feature images, the initial feature images pass through three cascaded Swin Transformer layers to obtain high-order feature images, the high-order feature images pass through a spatial attention layer to obtain spatial attention weighted feature images, the high-order feature images and the spatial attention weighted feature images are subjected to preset times of iterative residual connection to obtain final feature images, the final feature images pass through a spatial attention layer to obtain cloud attention images, and the final feature images pass through an image reconstruction layer to obtain coarse cloud removal images.

[0007] Preferably, the residual connection calculation expression is as follows:

[0008] wherein, represents the i-th feature image, represents a spatial attention module of the spatial attention layer, represents feature extraction of the i-th layer Swin Transformer, represents the i-th feature image, is a maximum of , represents the final feature image.

[0009] Preferably, the SAR fusion module comprises the following steps: The cloud attention images are subjected to a preprocessing operation to obtain binary cloud mask images; The SAR images, the binary cloud mask images and the coarse cloud removal images are subjected to image fusion by using a preset hybrid formula to obtain a fusion image.

[0010] Preferably, the preset mixed formula calculation expression is as follows:

[0011] wherein, denotes a fusion image, denotes a fusion weight coefficient, denotes a coarse cloud-removed image, denotes a binary cloud mask image, denotes a SAR image.

[0012] Preferably, the cloud-free diffusion module comprises: inputting the fusion image into a pre-trained variational autoencoder to obtain a conditional vector through encoding; randomly obtaining a latent variable at a time step according to the latent variable at the time step and the conditional vector to obtain a predicted Gaussian noise; calculating a denoising cosine scheduling coefficient according to the current time step, and performing iterative denoising on the latent variable at the time step according to the predicted Gaussian noise and the denoising cosine scheduling coefficient, to finally obtain a clean latent variable; inputting the clean latent variable into a variational auto-decoder to obtain a cloud-free image.

[0013] Preferably, before inputting the fusion image into the cloud-free diffusion module, the method further comprises training the cloud-free diffusion module, and the training process comprises: obtaining a satellite remote sensing image sample, inputting the satellite remote sensing image sample into a pre-trained variational autoencoder to obtain a latent variable at a time step of zero; presetting a number of iteration time steps, randomly obtaining a Gaussian noise, calculating a noise-adding cosine scheduling coefficient according to the current time step, and performing iterative noise-adding on the latent variable at the time step of zero according to the Gaussian noise and the noise-adding cosine scheduling coefficient to obtain a trained cloud-free diffusion module.

[0014] Preferably, the iterative denoising calculation expression is as follows:

[0015] wherein, denotes a latent variable at a time step , denotes a latent variable at a time step , denotes a denoising cosine scheduling coefficient at a time step , denotes a denoising cosine scheduling coefficient at a time step , denotes a predicted Gaussian noise.

[0016] Preferably, the iterative noise adding calculation expression is:

[0017] wherein, denotes a latent variable at time step denotes a latent variable at time step denotes a latent variable at time step denotes a noise-added cosine scheduling coefficient at time step denotes a noise-added cosine scheduling coefficient at time step denotes a Gaussian noise.

[0018] Preferably, the predicted Gaussian noise is obtained by joint prediction of a U-Net and a ControlNet branch.

[0019] Compared with the prior art, the present application has the beneficial effects that: The present application provides a SAR fusion thick cloud removal method based on Transformer and latent diffusion, which adopts three key modules to process satellite remote sensing images and SAR images, including a thick cloud removal module with three cascaded Swin Transformer layers, a SAR fusion module with a preset hybrid formula, and a cloud-free diffusion module with a parameterized iterative denoising process based on conditional latent diffusion, finally obtaining a high-definition cloud-free image, improving the calculation efficiency, and at the same time maintaining the quality and detail integrity of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 Fig. 1 is a flowchart of a SAR fusion thick cloud removal method based on Transformer and latent diffusion in an embodiment; Figure 2 Fig. 2 is a quantitative comparison result diagram of different cloud removal methods of the SAR fusion thick cloud removal method based on Transformer and latent diffusion in an embodiment on the SEN12MS-CR dataset; Figure 3 Fig. 3 is a performance comparison result diagram of three cloud removal methods of the SAR fusion thick cloud removal method based on Transformer and latent diffusion in an embodiment under different cloud amount proportions; Figure 4 Fig. 4 is a performance difference diagram of different cloud removal methods of the SAR fusion thick cloud removal method based on Transformer and latent diffusion in an embodiment in remote sensing image processing; Figure 5 ​This is a schematic diagram illustrating the processing effects of different cloud removal methods based on the SAR fusion method of Transformer and potential diffusion in one embodiment, under three cloud cover conditions of 30% / 60% / 90%. Figure 6 This is a schematic diagram illustrating the impact of different module combinations on the cloud removal effect of a SAR fusion thick cloud removal method based on Transformer and potential diffusion in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] Example 1 like Figure 1 As shown, this embodiment proposes a SAR fusion method for thick cloud removal based on Transformer and potential diffusion, the method comprising: S1: Input satellite remote sensing images and SAR images into the coarse cloud removal module (CCR module), and output cloud attention images and coarse cloud removal images respectively. The coarse cloud removal module includes a three-layer cascaded Swin Transformer layer and a spatial attention layer. The specific implementation method for this step is as follows: (1) Multimodal feature fusion First, the cloud-covered satellite remote sensing image (3 channels) and the SAR image (1 channel) are stitched together along the channel dimension (channel stitching layer) to generate a 4-channel stitched fused image. Then, the stitched image is processed by a shallow convolutional module (convolutional layer, 3×3 convolutional kernel and LeakyReLU activation function) to output 64 initial features. This step, through channel stitching and convolution operations, fuses the texture information of the optical image with the geometric structure of the SAR image into a unified feature representation, while strictly maintaining the spatial resolution of the original image (H×W unchanged), providing high-fidelity basic features for subsequent cloud detection and image reconstruction.

[0023] (2) Refinement of attention characteristics The initial features output from the multimodal feature fusion are used for deep feature extraction using a three-layer cascaded Swing Transformer. Each layer extracts high-order feature images through an 8×8 local window attention mechanism (spatial attention layer). The high-order feature images are then passed through the spatial attention layer to obtain spatial attention-weighted feature images, which are then processed... The next iteration residual connection mode adds the high-order feature image and the spatial attention weighted feature image, ensuring that important information is not lost. This hierarchical processing structure can effectively capture cloud layer features of different scales while ensuring computational efficiency, accurately identifying from local details to overall distribution. Through three layers of step-by-step refinement, the system can balance the accuracy of cloud detection and the integrity of non-cloud features, providing high-quality feature representation for subsequent processing.

[0024] (3) Dual-task collaborative output Based on the residual connection, the final feature map is obtained, and based on the final feature map, the cloud attention map of the pixel-level cloud probability distribution is generated in parallel: the final feature map is compressed through 1x1 convolution and activated by Sigmoid (spatial attention layer), and the cloud attention map of the pixel-level cloud probability distribution is output; the final feature map is reconstructed by an image reconstruction layer (3x3 convolution + Tanh activation) to obtain a coarse cloud removal image, which is suitable for input of a subsequent diffusion model. The two-way output is optimized end-to-end by a joint loss function LCCR, which ensures the accuracy of cloud detection while maintaining the color authenticity of the image.

[0025] S2: inputting the SAR image, the cloud attention image and the coarse cloud removal image into a SAR fusion module (SAR-F module) to output a fused image, wherein the SAR fusion module comprises a preset hybrid formula; The specific implementation of this step is as follows: (1) Generation of binary cloud mask image The cloud attention image output by the coarse cloud removal module is preprocessed, 5x5 median filtering is performed to eliminate noise and smooth the cloud area boundary, then a dynamic binarization method based on Otsu threshold calculation combined with local gray mean optimization is used to generate a preliminary cloud area label, and finally morphological closing operation (filling holes and connecting broken edges) and area filtering (removing isolated noise points) are used for optimization processing, outputting a high-quality binary cloud mask image, wherein the black area (value 0) represents the cloud area, and the white area (value 1) represents the non-cloud area.

[0026] (2) Generation of fused image On the basis of the above, a preset hybrid formula is used for image fusion, wherein λ is a fusion weight coefficient, and the formula ensures that the geometric features of the SAR image are mainly retained in the cloud area, and the texture details of the optical image are preferentially maintained in the non-cloud area, and finally a high-quality fused image is output. Through accurate cloud area detection and intelligent weighted fusion, the advantages of optical and SAR data are complementary.

[0027] S3: inputting the fused image into a cloud-free diffusion module to output a cloud-free image, wherein the cloud-free diffusion module comprises a parameterized iterative denoising process of conditional latent diffusion.

[0028] The specific implementation of this step is as follows: The high-quality image generation is realized through the following technical process: first, a pre-trained variational autoencoder (VAE) is used to encode the fusion image into a latent space to generate a 64x64x8-dimensional conditional vector. Then, the core process of iterative denoising is entered, and an improved DDIM sampling algorithm is adopted to randomly obtain a latent variable that obeys a Gaussian distribution, and from the latent variable that obeys a Gaussian distribution and the conditional vector, a U-Net main trunk and a ControlNet conditional branch are combined to predict Gaussian noise . The denoising cosine scheduling coefficient adopted in the denoising process is calculated according to the predicted Gaussian noise and the denoising cosine scheduling coefficient, and the latent variable is gradually removed from the noise according to the iterative denoising calculation expression, and the clean latent variable is obtained after 50 iterations. Finally, the clean latent variable is converted back to the pixel space through the VAE decoder to output the final high-definition cloud-free image. The whole process significantly improves the calculation efficiency through the latent space operation, while maintaining the quality and detail integrity of the generated image.

[0029] In terms of experimental results, the effect measurement indicators include: PSNR (peak signal-to-noise ratio): measures the image reconstruction quality, the higher the value, the better; SSIM (structural similarity): evaluates the structure retention, the higher the value, the better; NIQE (natural image quality assessment): reflects the visual naturalness, the lower the value, the better; MANIQA (image quality assessment): comprehensive quality indicator, the higher the value, the better.

[0030] As shown in Figure 2 , the quantitative comparison experimental results of different cloud removal methods on the SEN12MS-CR dataset show that the method adopted by the present application is better than other cloud removal methods in each indicator; as shown in Figure 3 , the performance comparison results of three cloud removal methods under different cloud amount ratios show that under different cloud amount ratios, the method adopted by the present application is better than the other two cloud removal methods in each indicator; as shown in Figure 4As shown in the figure, the performance of different cloud removal methods in remote sensing image processing is different, among which the SPA-GAN processing result has obvious artifacts (fuzzy cloud edge), the DSen2-CR has information loss in thick cloud area (missing ground texture), the GLF-CR produces unnatural color deviation (abnormal blue tone), the UnCRtainTS L2 performs well in thin cloud area but is insufficient in thick cloud processing, and the method (Ours) adopted by the application is closest to the real label (Label), especially in the preservation of building contours and vegetation texture; as Figure 5 As shown in the figure, the processing effect of different cloud removal methods under 30% / 60% / 90% cloud cover shows that the effect diagram processed by the method (Ours) adopted by the application is closest to the real label (Label); as Figure 6 As shown in the figure, the influence of different module combinations on cloud removal effect is different, among which, in the complete method (ours full), the index effect of each item is best.

[0031] Embodiment 2 This embodiment further supplements the SAR fusion thick cloud removal method based on Transformer and latent diffusion proposed in embodiment 1.

[0032] The coarse cloud removal module comprises: the satellite remote sensing image and the SAR image sequentially passing through a channel splicing layer and a convolution layer to obtain an initial feature image, the initial feature image passing through three cascaded Swin Transformer layers to obtain a high-order feature image, the high-order feature image passing through a spatial attention layer to obtain a spatial attention weighted feature image, the high-order feature image and the spatial attention weighted feature image being pre-set times of iteration residual connection to obtain a final feature image, the final feature image passing through a spatial attention layer to obtain a cloud attention image, and the final feature image passing through an image reconstruction layer to obtain a coarse cloud removal image.

[0033] The residual connection calculation expression is as follows:

[0034] Among them, represents the first feature image, represents a spatial attention module of the spatial attention layer, represents the first feature extraction of the Swin Transformer, represents the first feature image, is the maximum , represents the final feature image.

[0035] The SAR fusion module includes: The cloud attention image is preprocessed to obtain a binary cloud mask image; The SAR image, the binary cloud mask image, and the coarse cloud-removed image are fused using a preset mixing formula to obtain a fused image.

[0036] The preset mixed formula calculation expression is as follows:

[0037] in, Indicates a fused image. Indicates the fusion weight coefficient. This represents a coarsely cloud-removed image. Represents a binary cloud mask image. Represents a SAR image.

[0038] The cloudless diffusion module includes: The fused image is input into a pre-trained variational autoencoder to encode a conditional vector; Randomly select a time step that follows a Gaussian distribution. The latent variables, based on the time step, are The latent variables and conditional vectors are used to obtain the predicted Gaussian noise; The denoised cosine scheduling coefficients are calculated based on the current time step. Then, based on the predicted Gaussian noise and the denoised cosine scheduling coefficients, the time step is... The latent variables are iteratively denoised to obtain clean latent variables; The clean latent variables are input into the variational automatic decoder to obtain a cloud-free image.

[0039] Before inputting the fused image into the cloudless diffusion module, the process also includes training the cloudless diffusion module, the training process of which includes: Acquire satellite remote sensing image samples and input the satellite remote sensing image samples into a pre-trained variational autoencoder to obtain latent variables with a time step of zero; A preset number of iteration time steps are set, a Gaussian noise is randomly obtained, and the noise-adding cosine scheduling coefficient is calculated based on the current time step. Based on the Gaussian noise and the noise-adding cosine scheduling coefficient, the potential variables with zero time step are iteratively noise-adding is performed to obtain the trained cloudless diffusion module.

[0040] The iterative denoising calculation expression is:

[0041] in, Indicates the time step as Potential variables, Indicates the time step as Potential variables, Indicates the time step as The denoised cosine scheduling coefficients, Indicates the time step as The denoised cosine scheduling coefficients, This represents the predicted Gaussian noise.

[0042] The expression for the iterative noise addition calculation is:

[0043] in, Indicates the time step as Potential variables, Indicates the time step as Potential variables, Indicates the time step as The noisy cosine scheduling coefficient, This represents Gaussian noise.

[0044] The predicted Gaussian noise is obtained through joint prediction using U-Net and ControlNet branches.

[0045] Example 3 This embodiment further supplements the explanation of step S3 in embodiment 1, which includes training the cloudless diffusion module before applying it. First, a satellite remote sensing image sample is used for module training. A pre-trained variational autoencoder (VAE) is used to encode the satellite remote sensing image sample into a latent space, generating 64×64×8-dimensional latent variables with zero time steps. Simultaneously, Gaussian noise that conforms to a Gaussian distribution is randomly injected. This is used to destroy image information, and then a noisy cosine scheduling coefficient is applied. , The total number of iterations is set to T=50. Noise is added iteratively using the forward diffusion formula, ultimately transforming the input data into pure noise. From the training process, we can obtain the relationship between the input and the corresponding data. Therefore, during inference, we can use the input time step as... latent variables And conditional vectors, jointly predict Gaussian noise through the U-Net backbone and ControlNet conditional branches. Let the predicted Gaussian noise as close as possible This way, during reasoning, we can restore as many clean latent variables as possible.

Claims

1. A SAR fusion thick cloud removal method based on Transformer and latent diffusion, characterized in that, The method comprises the following steps: S1: inputting a satellite remote sensing image and a SAR image into a coarse cloud removal module to respectively output a cloud attention image and a coarse cloud removal image, wherein the coarse cloud removal module comprises three cascaded Swin Transformer layers and a spatial attention layer; S2: inputting the SAR image, the cloud attention image and the coarse cloud removal image into a SAR fusion module to output a fusion image, wherein the SAR fusion module comprises a preset hybrid formula; S3: inputting the fusion image into a cloud-free diffusion module to output a cloud-free image, wherein the cloud-free diffusion module comprises a parameterized iterative denoising process of conditional latent diffusion.

2. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 1, characterized in that, The coarse cloud removal module comprises that the satellite remote sensing image and the SAR image sequentially pass through a channel splicing layer and a convolution layer to obtain an initial feature image, the initial feature image passes through three cascaded Swin Transformer layers to obtain a high-order feature image, the high-order feature image passes through a spatial attention layer to obtain a spatial attention weighted feature image, the high-order feature image and the spatial attention weighted feature image are pre-set times of iterative residual connection to obtain a final feature image, the final feature image passes through a spatial attention layer to obtain a cloud attention image, and the final feature image passes through an image reconstruction layer to obtain a coarse cloud removal image.

3. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 2, characterized in that, The residual connection calculation expression is as follows: wherein, represents the first feature image, represents a spatial attention module of the spatial attention layer, represents the first feature extraction of the layer Swin Transformer, represents the first feature image, is at most , represents the final feature image.

4. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 1, characterized in that, The SAR fusion module comprises: performing a preprocessing operation on the cloud attention image to obtain a binary cloud mask image; performing image fusion on the SAR image, the binary cloud mask image and the coarse cloud removal image by using a preset hybrid formula to obtain a fusion image.

5. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 4, characterized in that, The preset hybrid formula calculation expression is as follows: wherein, denotes a fused image, denotes a fusion weight coefficient, denotes a coarse cloud removed image, denotes a binary cloud mask image, denotes a SAR image.

6. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 1, characterized in that, The cloud-free diffusion module comprises: inputting the fusion image into a pre-trained variational autoencoder to encode to obtain a conditional vector; A time step of a latent variable following a Gaussian distribution is randomly obtained A predicted Gaussian noise is obtained from the latent variable of the time step and the conditional vector. According to the current time step, a denoising cosine scheduling coefficient is calculated, and according to the predicted Gaussian noise and the denoising cosine scheduling coefficient, the latent variable of the time step is iteratively denoised, and finally a clean latent variable is obtained. ​ inputting the clean latent variable into a variational auto-decoder to obtain a cloud-free image.

7. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 6, characterized in that, Before the fusion image is input into the cloud-free diffusion module, the method further comprises training the cloud-free diffusion module, and the training process comprises: obtaining a satellite remote sensing image sample, inputting the satellite remote sensing image sample into a pre-trained variational autoencoder to obtain a latent variable with a time step of zero; presetting an iteration time step number, randomly obtaining a Gaussian noise, calculating a noise-added cosine scheduling coefficient according to a current time step, and iteratively adding noise to the latent variable with the time step of zero according to the Gaussian noise and the noise-added cosine scheduling coefficient to obtain a trained cloud-free diffusion module.

8. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 6, characterized in that, The iterative denoising calculation expression is as follows: wherein, denotes the latent variable at time step denotes the latent variable at time step denotes the denoising cosine schedule coefficient at time step denotes the denoising cosine schedule coefficient at time step denotes the predicted Gaussian noise.​​​​ 9. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 7, characterized in that, The iterative noise addition calculation expression is as follows: where, represents the latent variable at time step , represents the latent variable at time step , represents the noisy cosine scheduling coefficient at time step , represents the Gaussian noise.

10. The SAR fusion thick cloud removal method based on the Transformer and latent diffusion according to claim 6, characterized in that, The predicted Gaussian noise is obtained by joint prediction of a U-Net and a ControlNet branch.

Images