A method and system for declouding optical remote sensing images based on diffusion model

Abstract

The present invention provides an optical remote sensing image declouding method based on a diffusion model. The method comprises the following steps: inputting a foggy image into a conditional encoder for conditional encoding to obtain a first latent space vector; the conditional encoder comprises a plurality of image downsampling-feature attention extraction structures; encoding an irrelevant clear image using the encoder to obtain a second latent space vector; forward diffusion is performed on the second latent space vector through a diffusion model to gradually add noise to obtain a noisy vector; the noisy vector is concatenated with the first latent space vector and reversely diffused through the diffusion model to remove noise to obtain a third latent space vector; and the third latent space vector is input into a decoder for decoding to obtain a declouded image of the foggy image.

Description

Technical Field

[0001] The present invention relates to the technical field of optical remote sensing image defogging, and in particular to a method and system for defogging optical remote sensing images based on a diffusion model. Background Art

[0002] Since remote sensing technology first demonstrated its unique advantages in military reconnaissance in the last century, it has entered a golden age of rapid development. In recent years, my country has launched numerous remote sensing satellites for resource exploration, urban planning, military defense, and post-disaster reconstruction. However, in practical applications, atmospheric clouds can affect the sensors of remote sensing satellites, reducing the contrast of captured images, altering features such as color and texture, and reducing image quality. This makes it difficult to capture effective information from optical remote sensing images, hindering subsequent work such as ground-based resource surveys, land change detection, and target detection. Therefore, the question of how to effectively remove cloud and haze from optical remote sensing images to obtain clear, high-quality images is of vital research significance.

[0003] Although traditional optical remote sensing image declouding algorithms have achieved certain results, the results still show problems such as cloud residue, loss of detail information, and local color distortion of the image to varying degrees. In addition, some deep learning-based optical remote sensing image declouding methods usually require a large number of real labeled declouding datasets, which are difficult to obtain in actual situations. Summary of the Invention

[0004] The present invention proposes a method and system for declouding optical remote sensing images based on a diffusion model to solve the technical problems of cloud occlusion, loss of detail information, and local color distortion of declouded images in declouding tasks.

[0005] To solve the above technical problems, the present invention provides a method for declouding an optical remote sensing image based on a diffusion model, which is special in that it includes the following steps:

[0006] Step S1: Inputting the cloud image into a conditional encoder for conditional encoding to obtain a first latent space vector of the conditional variable; the conditional encoder includes multiple sets of image downsampling-feature attention extraction structures;

[0007] Step S2: Encode the irrelevant clear image using an encoder to obtain a second latent space vector;

[0008] Step S3: forward diffusion is performed on the second latent space vector through a diffusion model to gradually add noise to obtain a noisy vector;

[0009] Step S4: concatenate the noise vector with the first latent space vector, and perform reverse diffusion through a diffusion model to remove noise, thereby obtaining a third latent space vector;

[0010] Step S5: input the third latent space vector into a decoder for decoding to obtain a declouded image of the foggy image.

[0011] Preferably, the conditional encoder in step S1 includes two sets of image downsampling-feature attention extraction structures.

[0012] Preferably, the pre-training process of the conditional encoder comprises the following steps:

[0013] Step S11: constructing a pre-training network, the pre-training network includes five stages, each stage consists of a sampling module and a feature attention module, wherein the first three sampling modules are downsampling, and the last two sampling modules are upsampling;

[0014] Step S12: inputting a clear optical remote sensing image into the pre-trained network to obtain a pre-trained latent space vector;

[0015] Step S13: Decoding the pre-trained latent space vector to obtain a reconstructed image;

[0016] Step S14: Compare the reconstructed image with the optical remote sensing image. When the reconstructed image is less than a set threshold, the training is completed and the parameters of the conditional encoder are obtained. Otherwise, repeat steps S12 to S14 until the training is completed.

[0017] Preferably, the pre-training process of the diffusion model includes the following steps:

[0018] Step S21: input an irrelevant clear optical remote sensing image into the diffusion model to obtain a noise latent space vector;

[0019] Step S22: inputting the foggy image into the trained conditional encoder to obtain a foggy latent space vector;

[0020] Step S23: concatenating the cloud and fog latent space vector and the noise latent space vector and performing denoising to obtain a result image;

[0021] Step S24: setting a loss function, and updating the parameters of the diffusion model through a back propagation algorithm based on the result image and the cloud-free image corresponding to the cloud-enriched image, so as to complete the pre-training of the diffusion model.

[0022] Preferably, the loss function L in step S24 is total The expression is:

[0023]

[0024] Where θ represents the network parameters, ε∈N(0,I), ε t is the value of ε at step t, λ represents the hyperparameter, T represents the number of diffusion steps of the diffusion model, and Z t represents the latent space vector after t steps of noise addition, λ=0.3, O clear Represents the output clear fog-free image, and GT is the label data GroundTruth corresponding to the input foggy image.

[0025] Preferably, the optical remote sensing image declouding method based on a diffusion model further comprises the following steps:

[0026] Step S61: setting the conditional encoder to three sets of image downsampling-feature attention extraction structures;

[0027] Step S62: extracting features from the cloud image in sequence through three sets of image downsampling-feature attention extraction structures to obtain a first feature map F1, a second feature map F2, and a third feature map F3;

[0028] Step S63: performing feature fusion on the first feature map F1, the second feature map F2 and the third feature map F3 to obtain a multi-scale feature fusion feature map F0;

[0029] Step S64: inputting the fused feature map F0 into a dense residual block for feature extraction to obtain a feature extraction map;

[0030] Step S65: inputting the feature extraction image into an image reconstruction module to calculate a latent space vector to obtain a fourth latent space vector;

[0031] Step S65: splicing the third latent space vector and the fourth latent space vector in step S4, and inputting them into a decoder for decoding to obtain a declouded image of the foggy image.

[0032] Preferably, the method for performing feature fusion in step S63 includes the following steps:

[0033] Step S631: scaling the first feature map F1, the second feature map F2 and the third feature map F3 to the same size S1 、F S2 and F S3 , and perform weighted summation to obtain the sum feature map F g ;

[0034] Step S632: The summed feature map F g Perform convolution to obtain the convolution feature map F r ;

[0035] Step S633: Convolution feature map F r Perform weighted addition processing with the second feature map F2 to obtain a fusion feature map F t ;

[0036] Step S634: The fusion feature map F t Perform batch normalization and image downsampling calculations to obtain the multi-scale feature fusion feature map F0.

[0037] Preferably, the expression for weighted summation in step S631 is:

[0038]

[0039] Where, τ i Represents the weight corresponding to the feature map.

[0040] Preferably, the expression for weighted addition processing in step S633 is:

[0041]

[0042] Where, F tj Represents the fusion feature map F t The jth channel, F 2j represents the jth channel of the second feature map F2, F rj Represents the convolution feature map F r The jth channel of Represents the weight parameter corresponding to the j-th channel of the second feature map F2, Represents the convolution feature map F r The parameter corresponding to the j-th channel.

[0043] The present invention also provides an optical remote sensing image declouding system based on a diffusion model, characterized in that: the system comprises: a conditional encoding module, an irrelevant clear image encoding module, a diffusion module and a decoding module;

[0044] The conditional encoding module is used to conditionally encode the cloud image to obtain a first latent space vector;

[0045] The irrelevant clear image encoding module is used to encode the irrelevant clear image to obtain a second latent space vector;

[0046] The diffusion module is configured to perform forward diffusion on the second latent space vector to gradually add noise to obtain a noisy vector; and perform reverse diffusion on a vector obtained by concatenating the noisy vector and the first latent space vector to remove noise to obtain a third latent space vector;

[0047] The decoding module is used to decode the third latent space vector to obtain a declouded image of the cloud image.

[0048] The beneficial effects of the present invention include at least: generating high-quality images through a traditional diffusion model, and controlling the generated image to tend toward a clear, fog-free image corresponding to an input foggy image, thereby solving the problems of fog occlusion, loss of detail information, and local color distortion of the declouded image in the declouding task, solving the problem of lack of a data set in the remote sensing image declouding task, and improving the generation quality of the remote sensing image declouding results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0050] Figure 2 Schematic diagram of the pre-training process of the conditional encoder according to an embodiment of the present invention;

[0051] Figure 3 Schematic diagram of the pre-training process of the diffusion model according to an embodiment of the present invention;

[0052] Figure 4 A schematic flow chart of a method for adding a super-resolution module according to an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the feature fusion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, the embodiment of the present invention provides a method for declouding optical remote sensing images based on a diffusion model. The method is mainly based on the diffusion model. On this basis, this paper uses a conditional coding module as a constraint mechanism to control the output of the diffusion model so that the generated image is a cloud-free optical remote sensing image corresponding to the input cloud-containing optical remote sensing image, rather than an uncontrollable image. The method includes the following steps:

[0056] Step S1: Transform the cloud image I haze The input conditional encoder is conditionally encoded to obtain the first latent space vector c of the conditional variable; the conditional encoder includes multiple sets of image downsampling-feature attention extraction structures.

[0057] Specifically, in the embodiment of the present invention, it is more preferred that the conditional encoder includes two sets of image downsampling-feature attention extraction structures, which can improve the encoding efficiency while ensuring the encoding effect.

[0058] Step S2: Use the encoder to process the irrelevant clear image I clear Encode to obtain the second latent space vector z0;

[0059] Step S3: Forward diffuse the second latent space vector z0 through the diffusion model to gradually add noise to obtain the noisy vector z T ;

[0060] Step S4: Add the noise vector z T It is concatenated with the first latent space vector c and subjected to reverse diffusion through the diffusion model to denoise, obtaining the third latent space vector

[0061] Step S5: Decode the third latent space vector input to the decoder to obtain the de-clouded image O of the fog image. clear .

[0062] In order to effectively encode foggy images and further improve the encoding effect, a new conditional encoder is designed in the embodiment of the present invention. The specific network structure is as follows: Figure 2 The figure shows the entire conditional encoder pre-trained network. The conditional encoding module is composed of a set of encoders in an encoder-decoder structure, with the conditional encoder enclosed by the dashed box. The entire pre-trained network consists of five stages, each consisting of a sampling module and a feature attention module. The first three stages are downsampling stages, and the last two are upsampling stages.

[0063] Among them, the training process of the conditional encoder is as follows Figure 3 As shown in the upper part, it is mainly divided into two steps. The first step is to pre-train the conditional coding module. Since the conditional coding module itself is derived from a part of the encoder-decoder structure, a large number of clear optical remote sensing images I need to be trained in the pre-training process. pre-clear Input into the encoder-decoder network structure, so that the latent space vector of the clear image encoded by it can be successfully restored to the original image by the decoder. When the encoder-decoder network structure can successfully reconstruct the input clear image, that is, I pre-clear With O pre-clear When the difference is less than the set threshold, it is considered that the network structure has learned a lot of feature information about the clear image and has the ability to restore the clear image.

[0064] At the same time, the present invention also follows the same idea and trains the diffusion model to be able to reconstruct a clear image from a clear image to achieve the same purpose.

[0065] The trained conditional encoder is then added to the diffusion model and trained with the help of a small number of paired optical remote sensing image declouding datasets. In this step of training, an unrelated clear optical remote sensing image is first used as the input of the diffusion model I clear Transmitted to the diffusion model network to obtain the noise latent space vector z T , then there will be a cloud image I haze As a conditional image, it is fed into the conditional encoder and compared with the latent space vector z T Splicing and joint participation in the subsequent denoising process to obtain the final de-clouded image O clear .

[0066] In this embodiment of the present invention, in order to optimize the prediction performance of the model and drive the generated results closer to the corresponding clear and cloud-free image, the following formula is used as the loss function, and the back propagation algorithm is used to update the model parameters during the training process.

[0067]

[0068] Where θ represents the network parameters, ε∈N(0,I), ε t is the value of ε at step t, λ represents the hyperparameter, T represents the number of diffusion steps of the diffusion model, and Z t represents the latent space vector after t steps of noise addition, λ=0.3, O clear Represents the output clear fog-free image, and GT is the label data GroundTruth corresponding to the input foggy image.

[0069] In order to further mine the detail feature information in the input foggy image to supplement the detail information in the diffusion model generation result, the embodiment of the present invention also introduces a super-resolution module. The overall network structure is as follows: Figure 4 As shown in Figure 1, the module consists of three parts: feature fusion module, dense residual block and reconstruction module. The specific implementation method includes the following steps:

[0070] Step S61: setting the conditional encoder to three sets of image downsampling-feature attention extraction structures;

[0071] Step S62: extract features from the cloud image in sequence through three sets of image downsampling-feature attention extraction structures to obtain a first feature map F1, a second feature map F2, and a third feature map F3;

[0072] Step S63: Fusing the first feature map F1, the second feature map F2, and the third feature map F3 to obtain a multi-scale feature fusion feature map F0. This operation can fuse low-frequency detail features and high-frequency semantic features into one feature map.

[0073] Specifically, the structure diagram of the feature fusion module is as follows: Figure 5 As shown, the fusion method includes the following steps:

[0074] Step S631: Scale the first feature map F1, the second feature map F2, and the third feature map F3 to the same size. S1 、F S2 and F S3 , and perform weighted summation to obtain the sum feature map F g .

[0075] In the embodiment of the present invention, the weighted sum is calculated using the following formula:

[0076]

[0077] Where, τ i Represents the weight corresponding to the feature map.

[0078] Step S632: sum the feature map F g Perform convolution to obtain the convolution feature map F r .

[0079] Step S633: Convolution feature map F r Perform weighted addition processing with the second feature map F2 to obtain the fusion feature map F t ;

[0080] In the embodiment of the present invention, weighted addition processing is performed using the following formula:

[0081]

[0082] Where, F tj Represents the fusion feature map F t The jth channel, F 2j represents the jth channel of the second feature map F2, F rj Represents the convolution feature map F r The jth channel of Represents the weight parameter corresponding to the j-th channel of the second feature map F2, Represents the convolution feature map F r The parameter corresponding to the j-th channel.

[0083] Step S634: Fusing the feature map F tPerform batch normalization and image downsampling calculations to obtain the multi-scale feature fusion feature map F0.

[0084] Step S64: Input the fused feature map F0 into the dense residual block for feature extraction to obtain a feature extraction map. The dense residual block has a deep network structure, so it has a strong ability to mine image features. This model can also use more effective information when reconstructing a fog-free image. The dense residual block is a conventional technical means for those skilled in the art and will not be described in detail here.

[0085] Step S65: Input the feature extraction image into the image reconstruction module to calculate the latent space vector to obtain the fourth latent space vector;

[0086] Step S65: concatenate the third latent space vector and the fourth latent space vector in step S4, and input the concatenated vectors into a decoder for decoding to obtain a declouded image of the clouded image.

[0087] Through the above super-resolution modules, the generation quality of remote sensing image declouding results is further improved.

[0088] The present invention also provides an optical remote sensing image declouding system based on a diffusion model, which is characterized in that the system includes: a conditional encoding module, an irrelevant clear image encoding module, a diffusion module and a decoding module;

[0089] The conditional encoding module is used to conditionally encode the cloud image to obtain the first latent space vector;

[0090] An irrelevant clear image encoding module, used for encoding the irrelevant clear image to obtain a second latent space vector;

[0091] The diffusion module is used to perform forward diffusion on the second latent space vector to gradually add noise to obtain a noisy vector; and to perform reverse diffusion on the concatenation of the noisy vector and the first latent space vector to remove noise, to obtain a third latent space vector;

[0092] The decoding module is used to decode the third latent space vector to obtain a declouded image of the foggy image.

[0093] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. Only preferred embodiments of the present invention are presented. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. As long as there are no contradictions in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be based on the appended claims.

Claims

1. A method for declouding optical remote sensing images based on a diffusion model, characterized by: The following steps are involved: Step S1: Inputting the cloud image into a conditional encoder for conditional encoding to obtain a first latent space vector of the conditional variable; the conditional encoder includes multiple sets of image downsampling-feature attention extraction structures; Step S2: Encode the irrelevant clear image using an encoder to obtain a second latent space vector; Step S3: forward diffusion is performed on the second latent space vector through a diffusion model to gradually add noise to obtain a noisy vector; Step S4: concatenate the noise vector with the first latent space vector, and perform reverse diffusion through a diffusion model to remove noise, thereby obtaining a third latent space vector; Step S5: inputting the third latent space vector into a decoder for decoding to obtain a declouded image of the cloud image; The pre-training process of the conditional encoder includes the following steps: Step S11: constructing a pre-training network, the pre-training network includes five stages, each stage consists of a sampling module and a feature attention module, wherein the first three sampling modules are downsampling, and the last two sampling modules are upsampling; Step S12: inputting a clear optical remote sensing image into the pre-trained network to obtain a pre-trained latent space vector; Step S13: Decoding the pre-trained latent space vector to obtain a reconstructed image; Step S14: Compare the reconstructed image with the optical remote sensing image. If the reconstructed image is less than a set threshold, the training is completed and the parameters of the conditional encoder are obtained. Otherwise, repeat steps S12 to S14 until the training is completed. The pre-training process of the diffusion model includes the following steps: Step S21: input an irrelevant clear optical remote sensing image into the diffusion model to obtain a noise latent space vector; Step S22: inputting the foggy image into the trained conditional encoder to obtain a foggy latent space vector; Step S23: concatenating the cloud and fog latent space vector and the noise latent space vector and performing denoising to obtain a result image; Step S24: setting a loss function, and updating the parameters of the diffusion model through a back propagation algorithm based on the result image and the cloud-free image corresponding to the cloud-enriched image, so as to complete the pre-training of the diffusion model.

2. The method for declouding an optical remote sensing image based on a diffusion model according to claim 1, characterized in that: The conditional encoder in step S1 includes two sets of image downsampling-feature attention extraction structures.

3. The method for declouding an optical remote sensing image based on a diffusion model according to claim 2, characterized in that: The loss function L in step S24 total The expression is: Where θ represents the network parameters, ε∈N(0,I), ε t is the value of ε at step t, λ represents the hyperparameter, T represents the number of diffusion steps of the diffusion model, and Z t represents the latent space vector after t steps of noise addition, λ=0.3, O clear Represents the output clear fog-free image, and GT is the label data GroundTruth corresponding to the input foggy image.

4. The method for declouding an optical remote sensing image based on a diffusion model according to claim 1, characterized in that: The optical remote sensing image declouding method based on the diffusion model further comprises the following steps: Step S61: setting the conditional encoder to three sets of image downsampling-feature attention extraction structures; Step S62: extracting features from the cloud image in sequence through three sets of image downsampling-feature attention extraction structures to obtain a first feature map F1, a second feature map F2, and a third feature map F3; Step S63: performing feature fusion on the first feature map F1, the second feature map F2 and the third feature map F3 to obtain a multi-scale feature fusion feature map F0; Step S64: inputting the fused feature map F0 into a dense residual block for feature extraction to obtain a feature extraction map; Step S65: inputting the feature extraction image into an image reconstruction module to calculate a latent space vector to obtain a fourth latent space vector; Step S65: splicing the third latent space vector and the fourth latent space vector in step S4, and inputting them into a decoder for decoding to obtain a declouded image of the foggy image.

5. The optical remote sensing image declouding method based on a diffusion model according to claim 4, characterized in that: The method for performing feature fusion in step S63 includes the following steps: Step S631: scaling the first feature map F1, the second feature map F2 and the third feature map F3 to the same size S1 、F S2 and F S3 , and perform weighted summation to obtain the sum feature map F g ; Step S632: The summed feature map F g Perform convolution to obtain the convolution feature map F r ; Step S633: Convolution feature map F r Perform weighted addition processing with the second feature map F2 to obtain a fusion feature map F t ; Step S634: The fusion feature map F t Perform batch normalization and image downsampling calculations to obtain the multi-scale feature fusion feature map F0.

6. The method for declouding an optical remote sensing image based on a diffusion model according to claim 5, characterized in that: The expression for weighted summation in step S631 is: Where, τ i Represents the weight corresponding to the feature map.

7. The method for declouding an optical remote sensing image based on a diffusion model according to claim 5, characterized in that: The expression for weighted addition processing in step S633 is: Where, F tj Represents the fusion feature map F t The jth channel, F 2j represents the jth channel of the second feature map F2, F rj Represents the convolution feature map F r The jth channel of Represents the weight parameter corresponding to the j-th channel of the second feature map F2, Represents the convolution feature map F r The parameter corresponding to the j-th channel.

8. A diffusion model-based optical remote sensing image declouding system, implemented based on the diffusion model-based optical remote sensing image declouding method according to any one of claims 1 to 7, characterized in that: The system includes: a conditional encoding module, an irrelevant clear image encoding module, a diffusion module and a decoding module; The conditional encoding module is used to conditionally encode the cloud image to obtain a first latent space vector; The irrelevant clear image encoding module is used to encode the irrelevant clear image to obtain a second latent space vector; The diffusion module is configured to perform forward diffusion on the second latent space vector to gradually add noise to obtain a noisy vector; and perform reverse diffusion on a vector obtained by concatenating the noisy vector and the first latent space vector to remove noise to obtain a third latent space vector; The decoding module is used to decode the third latent space vector to obtain a declouded image of the cloud image.

Images