Remote sensing image cloud removal method based on conditional diffusion model

Abstract

The invention discloses a remote sensing image cloud removal method based on a conditional diffusion model. The remote sensing image cloud removal method comprises the following steps of S1, preprocessing data of a remote sensing image; s2, combining the data of the remote sensing image to construct a model; s3, performing experimental verification on the constructed model; the problems that in the prior art, a cloud removal task cannot be completed, the color consistency of a cloud removal result and a cloud image cannot be ensured, space position guidance of a cloud coverage area can be provided, a denoising sampling process of the cloud removal result is ensured to be matched with a forward propagation process, and a model training process cannot be stabilized are solved.

Description

Technical Field

[0001] This invention belongs to the field of cloud removal in remote sensing images, and specifically relates to a method for cloud removal in remote sensing images based on a conditional diffusion model. Background Technology

[0002] Remote sensing imagery, characterized by its strong intuitiveness, rich surface information, and high spatiotemporal resolution, is widely used in resource surveys, agricultural surveys, and military reconnaissance. However, due to limitations in its imaging principles and sensors, remote sensing imagery is highly susceptible to cloud cover when acquiring surface information. Clouds reduce surface radiation, leading to the loss or distortion of ground feature information in the imagery, severely impacting its usability and interpretation accuracy. Therefore, how to restore and reconstruct surface information in these areas is a highly significant research area.

[0003] Cloud removal methods for remote sensing images can be categorized into three types: methods based on traditional image processing, methods based on multi-source data fusion, and methods based on deep learning. Traditional image processing methods utilize techniques such as filtering, interpolation, and morphological operations to detect and repair cloud-covered areas in remote sensing images. Common methods include Haze Optimization Transform (HOT), Dark Channel Prior (DCP), and homomorphic filtering. However, these methods still have significant limitations when dealing with thick cloud cover and extreme weather scenarios. Thick clouds absorb surface radiation, resulting in severely insufficient spectral information, making detailed reconstruction of surface objects difficult. Furthermore, traditional image processing methods perform poorly when handling multi-scale clouds and complex cloud shapes, and have high requirements for data quality and temporal consistency, making it difficult to meet the efficiency demands of practical applications.

[0004] Multi-source data fusion-based methods integrate remote sensing data from different sensors, at different times, or with different spectral ranges (such as optical, infrared, and radar) to improve cloud removal accuracy by leveraging the complementarity between the multiple data sources. However, this method may still be limited by insufficient information from single-temporal images when dealing with thick cloud cover or full-temporal cloud-covered scenes.

[0005] In recent years, deep learning-based methods have made groundbreaking progress in image reconstruction and have gradually become the mainstream technical approach for cloud removal in remote sensing images. Compared with traditional methods based on physical models or statistical learning, deep learning methods, especially Generative Adversarial Networks (GANs), have shown significant advantages in this field. Numerous studies have shown that GAN-based cloud removal methods can better preserve image detail features, outperforming traditional methods in both visual effects and quantitative metrics. However, GAN-based cloud removal methods lack solid mathematical theoretical support and suffer from limitations such as pattern collapse and unstable training processes, potentially leading to color distortion or loss of detail in the restored image. Furthermore, deep learning-based methods typically only extract features from the bands of remote sensing images, ignoring important geospatial data such as altitude, longitude, and latitude contained within the images. This one-sided focus on band features limits the ability to obtain accurate cloud detection results in complex scenes.

[0006] In summary, this patent proposes a novel cloud detection method that combines remote sensing image geospatial data. Addressing the complex distribution and irregular shapes of cloud regions in remote sensing images, an interleaved self-attention mechanism module is designed to extract multi-scale features. A bidirectional multi-scale feature fusion module helps the model aggregate high-level and low-level semantic information, improving the model's ability to learn subtle differences and overall information within features. A cloud boundary extraction module is used for residual network learning to obtain high-precision cloud mask results with clear boundary details. A multi-level supervised two-layer loss function is also designed to achieve convergence between feature maps at different scales and the cloud mask. Furthermore, unlike previous methods that only focus on image data while neglecting geospatial data, this method fuses the altitude, longitude, and latitude of remote sensing images into a geospatial dataset, helping the deep learning network understand the relationship between cloud pixels and geospatial locations.

[0007] The existing technical solution proposes a cloud removal method called SPA-Net based on generative adversarial networks. This method consists of two parts: a generator is responsible for reconstructing cloudless images from cloudy images, while a discriminator ensures the authenticity of the generated results. Figure 1 The specific implementation process of this technical solution is shown.

[0008] (1) The generator of SPA-Net is a convolutional neural network called Spatial Attention Network (SPANet), whose structure includes several key modules. The input cloudy image first passes through a convolutional layer to extract an initial feature map, and then passes through three standard residual blocks (RBs) to further extract features. Next, the feature map enters four spatial attention blocks (SABs), each SAB containing three spatial attention residual blocks (SARBs) and one spatial attention module. Guided by the attention map generated by the spatial attention module, the SARB module removes clouds through negative residual learning, enhancing the information recovery of cloudy areas. Finally, after passing through two standard residual blocks and one convolutional layer, the generator outputs a cloudless image. The entire generator is designed to focus on cloudy areas and generate high-quality cloudless images through a local-to-global spatial attention mechanism.

[0009] (2) The discriminator of SPA-Net is a typical convolutional neural network, consisting of multiple convolutional layers (C), batch normalization layers (B), and Leaky ReLU activation layers (R). The input is a three-channel image. The network progressively extracts image features through the convolutional layers and performs feature normalization and nonlinear transformation through batch normalization and Leaky ReLU activation functions. Finally, the discriminator outputs a flag indicating whether the input image is a real cloudless image or an image generated by the generator. The discriminator aims to distinguish between real and generated images as accurately as possible, thus providing feedback to the generator and prompting it to produce more realistic cloudless images.

[0010] The shortcomings of existing technology one are significant. While it utilizes generative networks for cloud removal, its adversarial training mechanism between the generator and discriminator has major drawbacks. During training, the balance between the generator and discriminator must be continuously adjusted; any misstep can lead to complex training processes and difficulty in achieving stable convergence. The generative model lacks robust mathematical support, making it highly susceptible to pattern collapse or training non-convergence during training, further limiting its reliability and effectiveness in practical applications. Furthermore, existing technology one relies solely on the spectral features of remote sensing images for cloud removal, failing to fully explore and utilize key geographic information such as longitude, latitude, altitude, and climate, which to some extent limits its ability to improve cloud removal performance. Summary of the Invention

[0011] To address the aforementioned technical problems, this invention proposes a remote sensing image cloud removal method based on a conditional diffusion model. This method solves the problems in existing technologies, such as the inability to complete the cloud removal task, ensuring color consistency between the cloud removal result and the clouded image, providing spatial location guidance for the cloud-covered area, ensuring that the denoising sampling process of the cloud removal result matches the forward propagation process, and stabilizing the model training process.

[0012] The technical solution provided by this invention is as follows:

[0013] A method for cloud removal in remote sensing images based on a conditional diffusion model includes the following steps:

[0014] Step S1: Preprocess the remote sensing image data;

[0015] Step S2: Construct a model by combining data from remote sensing images;

[0016] Step S3: Conduct experimental verification of the constructed model.

[0017] Preferably, step S1 includes the following sub-steps:

[0018] Sub-step S1-1: Construct geospatial data by combining the global elevation data model with the global Köppen climate classification map and remote sensing dataset;

[0019] Sub-step S1-2: Perform radiometric transformation and coordinate system transformation on the geographic coordinates of the remote sensing image to ensure that it matches the coordinate system of the global elevation data and the Köppen climate type data;

[0020] Sub-steps S1-3: After completing the coordinate system matching, further acquire the elevation map, latitude map, longitude map and climate map corresponding to the remote sensing image as geographic information data.

[0021] Preferably, step S2 includes the following sub-steps:

[0022] Sub-step S2-1: Diffusion of forward noise;

[0023] Sub-step S2-2: Denoise the diffused noise and sample it to obtain a cloudless image.

[0024] Preferably, the forward noise diffusion in sub-step S2-1 specifically involves:

[0025] The cloudless image is progressively improved by adding specific noise at time step t. Degraded to a noisy image, and trained using an intermediate transition distribution, a parameterized neural network p θ .

[0026] Preferably, sub-step S2-2 specifically includes:

[0027] Sub-step S2-2-1: Transform the image Y containing noise... t p, as a denoising neural network θ Input;

[0028] Sub-step S2-2-2: Given a cloud image Together with geospatial data, these are inputs into a denoising neural network p that is combined with a DPM-Solver sampler. θ Generate cloudless images;

[0029] Sub-step S2-2-3: The framework for generating cloudless images can be iteratively optimized and generate high-quality cloudless images;

[0030] Sub-step S2-2-4: Backsample the cloudless image.

[0031] Preferably, the backward sampling in sub-step S2-2-4 includes the following structure:

[0032] Conditional encoders, time encoders, and encoder-decoders.

[0033] The beneficial effects of the remote sensing image cloud removal method based on the conditional diffusion model of this invention are as follows:

[0034] 1. This invention utilizes geographic information such as longitude, latitude, altitude, and climate type contained in remote sensing images to construct geospatial data. It combines remote sensing image band information to assist a deep learning network in understanding the relationship between cloud regions and geographic spatial locations, effectively improving the accuracy of cloud removal results.

[0035] 2. This invention addresses the problems of incomplete cloud removal results in remote sensing images and the instability of the learning process in generative models. In terms of the model, based on the "forward diffusion-backward sampling" training process, a data prediction-type conditional denoising network with cross-attention blocks as the backbone is designed. Skip links ensure color consistency between the de-clouded result and the clouded image. A conditional encoder is designed to introduce clouded image and geospatial data to provide spatial location guidance for the cloud-covered area of ​​the model. A temporal encoder is designed to ensure that the denoising sampling process of the de-clouded result matches the forward propagation process and stabilizes the model training process.

[0036] 3. In terms of data augmentation, this invention combines remote sensing imagery with global elevation data and global Köppen climate type data to generate longitude maps, latitude maps, elevation maps, and climate maps of corresponding sizes, which serve as auxiliary models for remote sensing imagery geospatial data to perform cloud removal tasks.

[0037] 4. In terms of the model, this invention designs a data prediction-type conditional denoising network with cross-attention blocks as the backbone based on the training process of "forward diffusion-backward sampling". Skip links ensure the color consistency between the de-clouding result and the clouded image. A conditional encoder is designed to introduce clouded image and geospatial data to provide spatial location guidance for the cloud-covered area of ​​the model. A temporal encoder is designed to ensure that the denoising sampling process of the de-clouding result matches the forward propagation process and stabilizes the model training process. Attached Figure Description

[0038] Figure 1 This is a structural diagram of SPA-Net, a prior art technology of the present invention.

[0039] Figure 2 This is a flowchart illustrating the construction process of geospatial data according to the present invention.

[0040] Figure 3 This is a flowchart of the forward diffusion and backward sampling process of the CDCR-Net of the present invention.

[0041] Figure 4 This is the conditional denoising network diagram of the present invention.

[0042] Figure 5 This is a structural diagram of the cross-attention block of the present invention.

[0043] Figure 6 This is a diagram of the adaptive cross-attention mechanism of the present invention.

[0044] Figure 7 This is a visualization of the cloud removal effect of the present invention on the SEN12MS-CR dataset.

[0045] Figure 8 This is a visualization of the cloud removal effect of the present invention on the RICE dataset. Detailed Implementation

[0046] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0048] A method for cloud removal in remote sensing images based on a conditional diffusion model includes the following steps:

[0049] Step S1: Preprocess the remote sensing image data;

[0050] Step S2: Construct a model by combining data from remote sensing images;

[0051] Step S3: Conduct experimental verification of the constructed model.

[0052] Step S1 of this implementation plan includes the following sub-steps:

[0053] Sub-step S1-1: Construct geospatial data by combining the global elevation data model with the global Köppen climate classification map and remote sensing dataset;

[0054] Sub-step S1-2: Perform radiometric transformation and coordinate system transformation on the geographic coordinates of the remote sensing image to ensure that it matches the coordinate system of the global elevation data and the Köppen climate type data;

[0055] Sub-steps S1-3: After completing the coordinate system matching, further acquire the elevation map, latitude map, longitude map and climate map corresponding to the remote sensing image as geographic information data.

[0056] Step S2 of this implementation plan includes the following sub-steps:

[0057] Sub-step S2-1: Diffusion of forward noise;

[0058] Sub-step S2-2: Denoise the diffused noise and sample it to obtain a cloudless image.

[0059] The forward noise diffusion in sub-step S2-1 of this implementation scheme is as follows:

[0060] The cloudless image is progressively improved by adding specific noise at time step t. Degraded to a noisy image, and trained using an intermediate transition distribution, a parameterized neural network p θ .

[0061] Sub-step S2-2 of this implementation plan is as follows:

[0062] Sub-step S2-2-1: Transform the image Y containing noise... t p, as a denoising neural network θ Input;

[0063] Sub-step S2-2-2: Given a cloud image Together with geospatial data, these are inputs into a denoising neural network p that is combined with a DPM-Solver sampler. θ Generate cloudless images;

[0064] Sub-step S2-2-3: The framework for generating cloudless images can be iteratively optimized and generate high-quality cloudless images;

[0065] Sub-step S2-2-4: Backsample the cloudless image.

[0066] The backward sampling of sub-step S2-2-4 in this implementation scheme includes the following structure:

[0067] Conditional encoders, time encoders, and encoder-decoders.

[0068] When this implementation plan is implemented,

[0069] This invention is implemented using the existing deep learning framework PyTorch. The complete technical solution consists of three steps: data augmentation, model building, and experimental analysis.

[0070] Remote sensing image data preprocessing

[0071] like Figure 2 As shown, this patent uses a global elevation data model combined with a global Köppen climate type map and a remote sensing dataset to construct geospatial data. The global elevation data is primarily based on the WGS84 geographic coordinate system and covers approximately 80% of the global land surface. Furthermore, the global Köppen climate type data provides detailed climate zoning information. These two data sources provide crucial data support and scientific basis for studying the relationship between cloud distribution and the geographic environment. However, remote sensing image datasets from different sensor sources exhibit significant differences in coordinate systems. Therefore, before constructing the geospatial data, the geographic coordinates of the remote sensing images must undergo radiometric transformation and coordinate system conversion to ensure they match the coordinate systems of the global elevation data and the Köppen climate type data. After coordinate conversion, elevation maps, latitude maps, longitude maps, and climate maps corresponding to the remote sensing images can be further obtained as geographic information data.

[0072] Model building

[0073] This patent proposes a remote sensing image cloud removal method based on a conditional diffusion model, namely CDCR-Net (Conditional Diffusion Cloud Removal Network). Figure 3 As shown, the cloud removal process of this method includes two steps: a forward noise diffusion process from left to right and a denoising sampling process from right to left.

[0074] During the forward diffusion process, specific noise at time step t is added to progressively diffuse the cloud-free image. Degraded to a noisy image, and trained using an intermediate transition distribution, a parameterized neural network p θ .

[0075] During the sampling process, the image Y containing noise is... t As p θ The input is a cloud image. Together with geospatial data, these are inputs into a denoising neural network p that is combined with a DPM-Solver sampler. θ The framework generates cloudless images and can iteratively optimize and generate high-quality cloudless images.

[0076] The goal of the forward diffusion process is to gradually degenerate the input data Y0 into Gaussian noise Y. T By adding a small amount of Gaussian noise at each step, the structure and features of the data are gradually destroyed, eventually degenerating into pure noise. This process is usually represented as a Markov chain, so the result at a later time step depends only on the previous time step. During the iteration process, the noise-added result at any time step can be directly calculated from the input. Based on the reparameterization technique, the recursive formula for iteratively calculating the forward diffusion process is:

[0077]

[0078] Among them, Y t Let be the degenerate image at step t, t∈{1,2,3,...,T}, β t These are scheduling parameters representing the noise variance, and as the number of steps increases, β1 < β2 < ... < β. t The relationship, ∈ t ~N(0,1) represents standard Gaussian noise, and I is the identity matrix. Therefore, as the step number t increases, Y t It will be very close to a standard normal distribution.

[0079] As can be seen from formula (1), all parameters of forward diffusion are known and there is no learnability. With the passage of t, Y... t This will become random noise that perfectly follows a Gaussian distribution. Furthermore, the noise addition result at each step t can be derived from the previous step; the noisy image at any step can be obtained from the initial state Y0. The formula is as follows:

[0080]

[0081] in, And α i =1-β t .

[0082] Equation (2) shows that for any noise-free image Y0, a scheduling parameter β with a fixed noise variance can be used. t Therefore, a noisy image at any stage can be obtained through single-step diffusion.

[0083] Therefore, the diffusion process can be achieved through the inverse process of the reparameterization technique. Generally, for a normal distribution Y... t ~N(μ, σ 2The input of ·I) is standardized by subtracting the mean μ and dividing by the standard deviation σ, resulting in a standard normally distributed sample z = (Y - μ) / (σ) ~ N(0,1). Therefore, by reversing this method and applying it to the diffusion process, the following formula can be derived:

[0084]

[0085] The inverse diffusion process is the core generation mechanism of the diffusion model, aiming to gradually recover the structure and features of the original data from Gaussian noise. This process is the reverse of the forward diffusion process, recovering the structure and features of the original data from Gaussian noise Y. t There is a process of progressively removing noise to recover the original image Y0. The inverse diffusion process is often referred to as the sampling process because it generates the final sample by progressively refining the noisy data.

[0086] Based on the formulaic reasoning of the diffusion process described above,

[0087] The sampling process can be set as approximating Y. t ~N(0,1) through P θ (Y t-1 |Y t )=N(Y t-1 μ θ (Y t ,t),∑ θ (Y t ,t)) to generate Y from the previous step t-1 However, due to the randomness of noise and the irreversibility of its diffusion, the sampling process cannot directly iterate over Y. t We obtain Y0. Therefore, we can design a neural network to handle μ. θ (Y t ,t) and ∑ θ (Y t ,t) to perform reasoning.

[0088] The sampling phase of traditional diffusion models requires starting from Y t Initially, the model iterates through the parameter renormalization formula until a noise-free image is obtained. The number of iterations, T, is directly proportional to the time cost, resulting in a slow inference speed for the diffusion model. Therefore, some researchers have proposed methods to improve the sampling mechanism to increase the sampling speed.

[0089] DPM-Solver is a method for accelerating sampling in diffusion models. It significantly improves the generation speed by optimizing the numerical solver in the backdiffusion process while maintaining the quality of the generated samples.

[0090] Its core idea is to use the semi-linear structure of the high-order ordinary differential equation (ODE) solver to approximate the reverse diffusion process, thereby reducing the number of sampling steps.

[0091]

[0092] Where, x s and x t These are noise samples at steps s and t, respectively, α t λ is the coefficient related to time step t. t and λ s The logarithmic signal-to-noise ratios corresponding to times t and s, respectively, e -λ The term is the exponential decay term, used for noise prediction in the weighted integral. Equation (4) expresses the solution of the diffusion ODE as an exponentially weighted integral. By approximating this integral using numerical methods, the DPM-Solver is not only applicable to continuous-time models, but can also be applied to discrete-time models through simple transformations, thus having higher flexibility and efficiency.

[0093] As mentioned above, backsampling is more challenging because it requires recovering the original information from complex, diffused data. This necessitates designing deep learning neural networks for prediction and reconstruction. Figure 4 As shown, this patent designs a conditional denoising network for the prediction process of noisy images, which can be divided into three main components: a conditional encoder, a temporal encoder, and an encoder-decoder structure built based on cross-attention blocks and skip links.

[0094] (1) Conditional encoder

[0095] Traditional diffusion models lack fine-grained control over the generated content, resulting in highly random outputs that are difficult to meet specific needs. However, for cloud removal tasks in remote sensing imagery, the distribution and shape of clouds are more complex, making it impossible to classify and label all clouds using a single sequence label. In contrast, clouded images themselves contain relevant cloud information, providing the model with a more direct and richer context, making it equivalent to a classification label. Therefore, geospatial data and clouded images are conditionally encoded and fused, serving as conditional input to a conditional denoising network. The specific implementation process is as follows:

[0096] CDN = cat(Proj_C(X) t )+Proj_G(GS t (5)

[0097] Among them, X t and GS t These represent cloud imagery and geospatial data, respectively, with t representing the time step.

[0098] It is worth noting that the noisy image y corresponds to any time step t X t GS t and Xp No forward diffusion processing is performed. Proj_C(·), Proj_P(·), and Proj_P(·) represent the 3×3 convolution and layer normalization operations for the three conditions, respectively, and the conditional features are represented by CDN.

[0099] (2) Time encoder

[0100] In the diffusion model, the forward diffusion and backward sampling processes of the noisy image strictly follow the variation law of the noise level. The input of CDCR-Net is an image y with varying degrees of random noise added. t t∈{1, 2, ..., T}. Accordingly, to prevent the model from learning a fixed paradigm, a time step t reflecting the noise level is used as an additional input to the network to guide the entire learning process. Specifically, for a time step t, sine and cosine encoding is first performed and fed into a multilayer perceptron (MLP) to obtain implicit temporal embeddings. The specific formula is as follows:

[0101]

[0102] embedding=cat(cos(args),sin(args)) (6)

[0103] Where t is the specified time vector, dim is the dimension of the feature to be incorporated, max_preiod is a constant that determines the frequency of the sine and cosine functions, and embedding is the embedding vector containing the sequence information of time step t.

[0104] Embedding can provide static time step information, but it is difficult to dynamically adapt to complex changes under different noise levels. Therefore, this patent introduces a design for dynamically adjusting parameters.

[0105] This design is partly inspired by adaptive control theory, similar to dynamically adjusting parameters in a control system to optimize control performance in real time. Specifically, it processes the time embedding through nonlinear transformation to generate a series of dynamically adjustable parameters. These parameters can dynamically adjust the model's behavior according to the current time step to better adapt to complex distributions under different noise levels, while enhancing the model's expressive power and generating richer contextual information. This effectively avoids the mode collapse problem common in generative models, thereby improving the diversity and robustness of the generated results. The formula is as follows:

[0106] α1, γ1, β1, α2, γ2, β2=SiLU(Linear(embedding)).chunck(6) (7)

[0107] The nonlinear transformation consists of the SiLU(·) activation function and the Linear(·) linear transformation layer. Chunck(6) is used to divide the feature vector into 6 parts along the feature dimension. α1, γ1, β1, α2, γ2, and β2 are the dynamic adjustment parameters.

[0108] (3) Cross-attention blocks

[0109] Traditional diffusion models typically use U-Net as the backbone network. However, U-Net's convolutional operations have a limited receptive field, making it difficult to model long-range dependencies and limiting its performance when processing complex cloud distributions in remote sensing images. Figure 4 and 5 As shown, this patent utilizes cross-attention blocks combined with an encoder-decoder structure as the backbone network of the conditional denoising network to ensure color consistency between the de-clouded and cloudless images. Simultaneously, a dedicated cross-attention block is designed to process conditional information, thereby better integrating multi-source conditional information. For the encoder part responsible for feature extraction, the cross-attention block uses the noisy image Y... t CDN and dynamically adjusted parameters generated by time step t are used as inputs to Y t Feature segmentation and downsampling operations are performed on both the CDN and the encoder to ensure consistency across the channel dimension. After extracting position-related features through an adaptive cross-attention mechanism, layer normalization is further applied to stabilize the training process and improve model convergence speed. Linear transformation and reshaping operations are used to adjust the size of the output features, allowing the output to fit the next cross-attention block. The encoder process can be summarized by the following formula:

[0110]

[0111] Where θ = {α1, γ1, β1, α2, γ2, β2} are dynamically adjusted parameters, i represents the number of layers in the cross-attention blocks of the decoder, and feature segmentation is essentially a Conv(·) convolution operation. 2× (·) represents double downsampling, Mutli_CA[·] represents adaptive cross attention, and RLL[·] is a unified abbreviation for layer normalization, linear transformation, and reshape.

[0112] For the latter half of the decoder in the U-shaped structure, the input features consist of the output of the previous cross-attention block and the output obtained from the corresponding coding layer through skip connections. Upsampling is then used to progressively restore the feature size to the original image size. It is worth noting that conditional features are not required as guidance during the decoding stage. The formula is as follows:

[0113]

[0114] in, This represents the output of the previous cross-attention block. Up represents the intermediate features of the encoder part corresponding to the current cross-attention block. 2× (·) indicates double upsampling.

[0115] like Figure 6 As shown, the adaptive cross-attention mechanism constructed in this section differs from the conventional multi-head self-attention mechanism. Traditional ViT's self-attention mechanism primarily calculates the correlation within a single feature, while the conditional cross-attention mechanism constructed in this section focuses on calculating the correlation between input noisy features and conditional features (cloudy images and geospatial data). Specifically, the conditional cross-attention mechanism uses the noisy image to generate a query vector Q, and simultaneously uses the conditional features to generate a key vector (K) and a value vector (V). By calculating the dot product of Q and K and normalizing it using Softmax, the attention weight matrix is ​​obtained. Finally, the weighted aggregated value vector V generates the fused feature representation. This feature representation allows the model to dynamically focus on y. t The most relevant part of the CDN adaptively assigns weights based on the conditional features of the current input, effectively utilizing these features to guide the geospatial location information of clouds in noisy images, thereby more accurately locating and removing clouds. The formula is as follows:

[0116]

[0117] Among them, Q h K h and V h Let W represent the query, key, and value matrices of the h-th attention head, respectively. Q W K and W V This is the corresponding weight matrix. Softmax(·) represents the normalization function, d k For K h Vector dimension, Attention h This represents the weight result of the h-th attention head. Finally, the attention values ​​of all attention heads are concatenated and appended to the weight matrix W. O Dot product yields cross-attention features.

[0118] To address issues such as mode collapse in generative models, this chapter designs dynamic control of the residual path and adaptive adjustment of layer normalization parameters in adaptive cross-attention. Before discussing the adjustment parameters, it is necessary to explain layer normalization (LN). LN normalizes each feature dimension of the input to stabilize the model's training process and accelerate its convergence. Specifically, layer normalization first calculates the mean and standard deviation of the input features, then uses these statistics to standardize the input, and finally performs a linear transformation on the standardized features using scaling and offset parameters. The formula is as follows:

[0119]

[0120] Where x is the input feature, μ and σ are the mean and standard deviation, and γ and β are the scaling and offset parameters for learning.

[0121] In the Adaptive Layer Normalization (AdaLN) introduced in this patent, adjustment parameters γ1, β1, γ2, and β2, which contain time-step information, are dynamically incorporated into the layer normalization parameters of conditional cross-attention and MLP, thereby improving the stability and diversity of the generated results. α1 and α2 serve as residual scaling parameters, used to dynamically adjust the input-output ratio in the residual structure. Furthermore, α is initialized to zero in the early stages of network training, and the initial state of the network is set to an identity mapping. This not only ensures the stability of training but also improves the overall convergence speed of the network. The above process can be summarized by the following formula:

[0122]

[0123] Where AdaLN(·) represents the adaptive cross-attention mechanism, Attention(·) represents the conditional cross-attention mechanism, and γ i and β i It is an adjustment parameter that includes time step information, α i These are the residual scaling parameters, where μ and σ are the mean and standard deviation of the input, respectively.

[0124] Experimental Analysis

[0125] To verify the effectiveness of the proposed model, the method was compared with other cloud removal methods on the SEN12MS-CR and RICE datasets, including Spa-GAN, CTGAN, DIFFCR, UnCRtainTS, and PMAA.

[0126] Table 1 shows the comparative experimental results on the SEN12MS-CR dataset.

[0127]

[0128] As shown in Tables 1 and 2, CDCR-Net achieved a PSNR of 44.18 dB and an SSIM of 96.99% on the SEN12MS-CR dataset. Compared to cloud removal methods based on generative adversarial networks (Spa-GAN and CTGAN), it improved by 5.75 dB and 4.4% in PSNR and SSIM, respectively, compared to Spa-GAN, and by 7.05 dB and 6.1%, respectively, compared to CTGAN. Compared to cloud removal methods combining attention mechanisms and convolutional neural networks (UnCRtainTS and PMAA), CDCR-Net also showed significant advantages in both metrics. Analysis suggests this is mainly because the conditional cross-attention used in the conditional denoising network effectively utilizes the latent spatial information of clouded images and geospatial data to guide the reconstruction of cloudless images, allowing the model to more accurately focus on surface information and cloud regions in the generated image to recover details obscured by clouds. Compared to DIFFCR, another diffusion model, CDCR-Net also performed excellently in both metrics. On the RICE dataset, the trends of the evaluation metrics are consistent with those on the SEN12MS-CR dataset, but most metrics are higher than those in Table 1. Further analysis suggests this phenomenon is mainly due to the relatively simple scenarios and more consistent background features of the RICE dataset, which helps reduce potential errors in the model when handling complex scenarios.

[0129] Table 2 shows the comparative experimental results on the RICE dataset.

[0130]

[0131] To further evaluate and verify the performance of the comparative method, experimental results were visualized and compared on the SEN12MS-CR and RICE datasets. Figure 7 and Figure 8 As shown in the table, from an overall sensory analysis, CDCR-Net is closest to the cloudless image in terms of color and detail. CTGAN and Spa-GAN have poor visualization effects and exhibit significant color differences and image distortion. UnCRTainTS and PMAA can recover surface information relatively clearly, but still have some blurring issues in pixel details. DIFFCR is closest to the proposed method in terms of cloud removal effect, but still has color differences compared to the cloudless image. In summary, the visualization results are consistent with the indicators in the table and conform to the previous description of generative models, proving the effectiveness of CDCR-Net in the cloud removal task.

[0132] To verify that geospatial data constructed from longitude, latitude, elevation, and climate maps can effectively guide the model in understanding the spatial information of cloud pixels, an ablation experiment was designed to verify its effectiveness. The results are shown in Figure 3.

[0133] Table 3 Evaluation of Geographic Spatial Data Ablation Experiment Indicators

[0134]

[0135] The results demonstrate that geospatial data constructed from longitude, latitude, and elevation maps effectively improves the cloud removal accuracy of conditional denoising models. Compared to directly using remote sensing image datasets, geospatial data stitched with Köppen climate type data significantly enhances model performance. Through reasoning and analysis, it is believed that an implicit relationship exists between climate type and latitude / longitude, enabling the model to better capture cloud distribution patterns, thereby improving cloud removal accuracy.

[0136] Table 4. Evaluation of ablation experiment metrics for efficiency and generated image quality at different inference iterations.

[0137]

[0138] Traditional diffusion models are extremely slow inference models, requiring thousands of iterations to generate high-quality images. Even on high-performance devices, the inference speed for a single sample can be slower than that for a single training cycle, severely hindering the application of diffusion models in computer vision. In contrast, CDCR-Net, through a combination of data prediction and the DPM-Solver sampler, generates high-quality samples with only a few inference steps. Therefore, quantitative results comparing the generated image quality and inference time for a single image at different sampling steps are presented in Table 4.

[0139] In the inference experiments for a single image, the number of inferences was positively correlated with time. As the number of inferences increased, the inference time became increasingly longer. The table shows that a good quality generated image can be obtained in just one step, with improvements gradually increasing from 1 to 5 steps. However, the cloud removal performance gradually decreased as the number of inferences increased. Analysis suggests this is because oversampling introduces excessive noise into the model during generation, thus destroying image details and structure, leading to performance degradation.

[0140] This invention combines geographic information from remote sensing images with a global digital elevation model and a global Köppen climate classification map to construct geospatial data, which is then used in conjunction with band information from remote sensing images for cloud removal tasks. A novel cloud removal method based on a conditional diffusion attention mechanism is proposed. Specific protection technologies are as follows:

[0141] By utilizing the geographic location information inherent in remote sensing images, and through operations such as affine transformation formulas and coordinate system transformations, these images are matched with global elevation data and global Köppen climate classification data to generate longitude maps, latitude maps, elevation maps, and climate maps. These four maps are then combined to construct a geospatial data-assisted deep learning model for cloud removal.

[0142] The CDCR-Net (Conditional Diffusion Cloud Removal Network) is a remote sensing image cloud removal method based on the conditional diffusion model, and the conditional denoising network, conditional encoder, temporal encoder and adaptive cross-attention mechanism constructed by the method.

Claims

1. A method for cloud removal in remote sensing images based on a conditional diffusion model, characterized in that, Includes the following steps: Step S1: Preprocess the remote sensing image data; Step S2: Construct a model by combining data from remote sensing images; Step S3: Conduct experimental verification of the constructed model.

2. The remote sensing image cloud removal method based on the conditional diffusion model according to claim 1, characterized in that, Step S1 includes the following sub-steps: Sub-step S1-1: Construct geospatial data by combining the global elevation data model with the global Köppen climate classification map and remote sensing dataset; Sub-step S1-2: Perform radiometric transformation and coordinate system transformation on the geographic coordinates of the remote sensing image to ensure that it matches the coordinate system of the global elevation data and the Köppen climate type data; Sub-steps S1-3: After completing the coordinate system matching, further acquire the elevation map, latitude map, longitude map and climate map corresponding to the remote sensing image as geographic information data.

3. The remote sensing image cloud removal method based on the conditional diffusion model according to claim 1, characterized in that, Step S2 includes the following sub-steps: Sub-step S2-1: Diffusion of forward noise; Sub-step S2-2: Denoise the diffused noise and sample it to obtain a cloudless image.

4. The remote sensing image cloud removal method based on the conditional diffusion model according to claim 1, characterized in that, The forward noise diffusion in sub-step S2-1 specifically involves: The cloudless image is progressively improved by adding specific noise at time step t. Degraded to a noisy image, and trained using an intermediate transition distribution, a parameterized neural network p θ .

5. The remote sensing image cloud removal method based on the conditional diffusion model according to claim 1, characterized in that, The sub-step S2-2 is specifically as follows: Sub-step S2-2-1: Transform the image Y containing noise... t p, as a denoising neural network θ Input; Sub-step S2-2-2: Given a cloud image Together with geospatial data, these are inputs into a denoising neural network p that is combined with a DPM-Solver sampler. θ Generate cloudless images; Sub-step S2-2-3: The framework for generating cloudless images can be iteratively optimized and generate high-quality cloudless images; Sub-step S2-2-4: Backsample the cloudless image.

6. The remote sensing image cloud removal method based on the conditional diffusion model according to claim 5, characterized in that, The backward sampling in sub-step S2-2-4 includes the following structure: Conditional encoders, time encoders, and encoder-decoders.

Images