A cloud removal method for optical remote sensing images based on SAR images

Abstract

The present invention relates to a method for declouding optical remote sensing images based on SAR images, which belongs to the field of image processing technology and solves the problems of long training time and poor declouding effect of existing declouding models. The method comprises: collecting SAR images and optical remote sensing images with clouds from the same area and at the same time; passing the SAR images into an improved GAN network to obtain a simulated optical image without clouds; the improved GAN network comprises two generators and a discriminator; passing the optical remote sensing image with clouds into a cloud detection model, and obtaining the cloud area based on the output image segmentation result; intercepting the image corresponding to the cloud area from the simulated optical image without clouds and replacing it with the optical remote sensing image with clouds, and performing local filtering processing on the replaced optical remote sensing image to obtain the optical remote sensing image after declouding. This achieves a fast and efficient declouding method and improves the clarity of the declouded image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for removing cloud from optical remote sensing images based on SAR images. Background Art

[0002] During the optical remote sensing imagery process, thick cloud areas can completely obscure ground feature information, making it difficult for simple filtering methods to effectively remove the cloud layer, significantly reducing the utilization rate of remote sensing images. During routine remote sensing surveillance, if adverse weather conditions occur, remote sensing images acquired over long periods of time may be frequently obscured by cloud cover, hindering the acquisition of ground information and further reducing the practical value of remote sensing images.

[0003] SAR imagery, or synthetic aperture radar imagery, demonstrates significant advantages in remote sensing due to its unique penetrating, polarimetric, coherent, and high-resolution characteristics. Unrestricted by lighting and weather conditions, it can acquire surface information around the clock, with excellent penetration into clouds, fog, forests, and soil. Furthermore, the polarimetric nature of SAR imagery provides rich information about surface structure, while its coherence enables interferometric measurement for high-precision surface deformation monitoring. SAR technology also boasts high resolution and a wide swath, enabling rapid acquisition of detailed surface information over a wide area. These characteristics give SAR imagery broad application prospects in a variety of fields, including military reconnaissance, environmental monitoring, land use planning, and natural disaster warning.

[0004] Existing technologies typically use SAR images and optical remote sensing images with clouds to train neural network models to directly generate cloud-free images. The processes involved in the neural network model are cumbersome, the training time is long, and it is highly dependent on the training effect of the neural network model. In some cases, the generated cloud-free images are not clear and cannot locate the problem. Summary of the Invention

[0005] In view of the above analysis, the embodiment of the present invention aims to provide a method for declouding optical remote sensing images based on SAR images, so as to solve the problem that the existing declouding model has a long training time and poor declouding effect.

[0006] The embodiment of the present invention provides a method for removing cloud from optical remote sensing images based on SAR images, comprising the following steps:

[0007] Collect SAR images and cloud-covered optical remote sensing images of the same area and time. Pass the SAR images into an improved GAN network to obtain cloud-free simulated optical images. The improved GAN network consists of two generators and one discriminator.

[0008] The cloud-containing optical remote sensing image is fed into the cloud detection model, and the cloud area is obtained based on the output image segmentation result.

[0009] The image corresponding to the cloud area is intercepted from the cloudless simulated optical image and replaced with the clouded optical remote sensing image. The replaced optical remote sensing image is subjected to local filtering to obtain the cloud-removed optical remote sensing image.

[0010] Based on the further improvement of the above method, the improved GAN network is trained using a dataset constructed from multiple image pairs consisting of historical SAR images and historical cloud-free optical remote sensing images of the same area and time, where the historical SAR images are used as input samples and the historical cloud-free optical remote sensing images are used as real data.

[0011] Based on a further improvement of the above method, the first generator in the improved GAN network receives historical SAR images and random noise, uses an encoder-decoder-based network to extract overall structural features, generates a first optical image, and passes it to the second generator; the second generator takes the first optical image and random noise as input, uses a residual network that introduces multi-scale fusion to extract local texture features and fuse them with overall structural features to generate a second optical image, which is passed to the discriminator; the discriminator receives the second optical image and a historical cloud-free optical remote sensing image, and outputs the identification result.

[0012] Based on the further improvement of the above method, the loss function of the discriminator includes: real data identification loss and generated data identification loss, where the generated data identification loss is obtained by calculating the weights of the first generator and the second generator according to the average information entropy value of the historical SAR image, and then taking the weighted sum of the losses of the two generators.

[0013] Based on the further improvement of the above method, the average information entropy value of historical SAR images is calculated by the following formula:

[0014]

[0015] Where ENT represents the average information entropy value, p(i, j) represents the probability of gray levels i and j appearing simultaneously in historical SAR images, and k represents the total number of gray levels.

[0016] Based on the further improvement of the above method, the weights of the first generator and the second generator are calculated according to the average information entropy value of the historical SAR images using the following formula:

[0017]

[0018] Among them, λ1 represents the weight of the first generator, λ2 represents the second generator, n represents the total number of historical SAR images in the dataset, m1 represents the number of historical SAR images whose average information entropy value is less than the average information entropy value of the dataset, m2 represents the number of historical SAR images whose average information entropy value is greater than the average information entropy value of the dataset, and the average information entropy value of the dataset is the average of the average information entropy values ​​of all historical SAR images in the dataset.

[0019] Based on the further improvement of the above method, the cloud detection model is obtained by training the UNET network by collecting historical optical remote sensing images with clouds, and updating the network parameters by minimizing the weighted cross entropy loss function.

[0020] Based on the further improvement of the above method, the weights in the weighted cross entropy loss function are calculated by the following formula:

[0021]

[0022] Among them, w c represents the weight of category c, category c corresponds to cloud or background, N represents the total number of pixels in the optical remote sensing image with clouds in history; N c represents the number of pixels predicted to be of category c, and exp(·) represents the exponential function.

[0023] Based on a further improvement of the above method, a local filtering process is performed on the replaced optical remote sensing image to obtain a cloud-removed optical remote sensing image, which is obtained by performing a median filtering operation on the image within the minimum circumscribed rectangular frame of each replacement area.

[0024] Based on the further improvement of the above method, the encoder and decoder in the UNET network adopt the ELU activation function in the convolutional layer.

[0025] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0026] 1. Taking advantage of the fact that SAR images can penetrate clouds and are not restricted by weather conditions, a GAN network with two generators is used to convert SAR images into cloud-free optical remote sensing images. A deep learning model is then used to detect clouds in optical remote sensing images with clouds. Finally, a replacement method is used to remove clouds, which improves the utilization rate of optical remote sensing images and enhances the clarity of de-clouded images.

[0027] 2. Two generators are used to extract and fuse features from the overall structure and local texture respectively, and the average information entropy is used to reflect the different texture characteristics of different SAR images. At the same time, the weights of the two generators are dynamically allocated according to the scale and texture characteristics of the dataset, which better captures the data characteristics, enhances the network's generation ability and quality, and improves the network's accuracy and generalization ability.

[0028] 3. During cloud detection using optical remote sensing images, weights are set based on the proportion of each category, so that the model weights favor the cloud area, better learning cloud feature information, improving the accuracy of cloud detection, effectively reducing the impact of high-frequency categories in data samples, and balancing data samples of each category.

[0029] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0031] Figure 1 This is a flow chart of a method for declouding an optical remote sensing image based on SAR images in an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of the improved GAN network architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0034] A specific embodiment of the present invention discloses a method for removing cloud from optical remote sensing images based on SAR images, such as Figure 1 As shown, the following steps are included:

[0035] S1. Collect SAR images and optical remote sensing images with clouds in the same area at the same time; pass the SAR images into the improved GAN network to obtain cloud-free simulated optical images; the improved GAN network contains two generators and one discriminator.

[0036] It should be noted that SAR imagery can penetrate clouds and most weather interference. Due to the long wavelength of radar, it is not limited by the wavelength of visible light, allowing SAR imagery to obtain effective data even in cloudy areas and inclement weather. Compared to optical remote sensing imagery, SAR imagery has lower resolution and is more difficult to interpret. Therefore, this embodiment utilizes an improved GAN (Generative Adversarial Network) network to convert SAR imagery into optical imagery, which serves as an auxiliary image for cloud removal.

[0037] Existing GAN networks consist of a generator and a discriminator. The generator generates fake data, while the discriminator determines whether the input data is real or generated by the generator. The generator continuously optimizes its parameters to ensure that the generated data is authenticated by the discriminator, and the discriminator also optimizes its parameters to ensure more accurate authentication. This embodiment adds a new generator to the existing GAN network, forming a network model with two generators.

[0038] It should be noted that the improved GAN network is trained using a dataset constructed using multiple image pairs consisting of historical SAR images and historical cloud-free optical remote sensing images of the same area and time, where the historical SAR images serve as input samples and the historical cloud-free optical remote sensing images serve as real data.

[0039] The improved GAN network architecture diagram is as follows Figure 2 Specifically, the first generator in the improved GAN network receives historical SAR images and random noise, focusing on the overall structure of the SAR image and extracting global texture features using an encoder-decoder network. The encoder gradually reduces the spatial dimension through convolution operations and uses Leaky ReLU as the activation function to promote nonlinear transformation and enhance the representation ability of the network, thereby capturing the overall structural characteristics of the SAR image. The decoder gradually increases the spatial dimension through deconvolution operations to generate a first optical image with overall structural characteristics.

[0040] Exemplarily, the first generator adopts a UNET network or a Transformer network.

[0041] The second generator takes the first optical image and random noise as input, focusing on local texture features. It then uses a residual network with multi-scale fusion to further extract features from the first optical image, fusing local texture features with overall structural features to refine the texture features and generate the second optical image. The second generator uses ReLU as the activation function.

[0042] Exemplarily, the second generator adopts a residual network FResNet combined with a feature pyramid.

[0043] The discriminator receives the second optical image and the historical cloud-free optical remote sensing image, and outputs a discrimination result.

[0044] When using the dataset to train the improved GAN network, both generators try to generate real optical remote sensing images to deceive the discriminator. The loss functions of the two generators are as follows:

[0045]

[0046] Where L(G1) and L(G2) represent the loss functions of the first generator G1 and the second generator G2 respectively; Indicates the calculation expectation, z1 and z2 represent the prior noise distribution and is a random noise vector sampled in , x represents the historical SAR image in the SAR dataset pdata(x), G1(·) represents the first optical image generated by the first generator; G2(·) represents the second optical image generated by the second generator, and D(·) represents the identification result of the discriminator.

[0047] It should be noted that the discriminator's loss function includes the real data identification loss and the generated data identification loss. In this embodiment, two generators sequentially generate data. Given their different focuses and the different SAR imagery, the generated data identification loss in this embodiment is calculated by weighting the first and second generators based on the average information entropy of the historical SAR images, and then taking the weighted sum of the losses of the two generators.

[0048] Specifically, the average information entropy value of historical SAR images is calculated by the following formula:

[0049]

[0050] Where ENT represents the average information entropy value, p(i, j) represents the probability of gray levels i and j appearing simultaneously in historical SAR images, and k represents the total number of gray levels.

[0051] Furthermore, according to the average information entropy value of historical SAR images, the weights of the first generator and the second generator are calculated by the following formula:

[0052]

[0053] Among them, λ1 represents the weight of the first generator, λ2 represents the weight of the second generator, n represents the total number of historical SAR images in the dataset, m1 represents the number of historical SAR images whose average information entropy value is less than the average information entropy value of the dataset, m2 represents the number of historical SAR images whose average information entropy value is greater than the average information entropy value of the dataset, and the average information entropy value of the dataset is the average of the average information entropy values ​​of all historical SAR images in the dataset.

[0054] Finally, the loss function of the discriminator is expressed by the following formula:

[0055]

[0056] Among them, L(D,G1,G2) represents the loss function of the discriminator, and y represents the historical cloud-free optical remote sensing image in the optical remote sensing image dataset ptarget(y).

[0057] After the improved GAN network is trained, the actual collected SAR image is passed into the improved GAN network. After passing through the first generator and the second generator in sequence, the image output by the second generator is used as a cloud-free simulated optical image.

[0058] Compared with the existing technology, this embodiment uses two generators to extract and fuse features from the overall structure and local texture respectively, and uses the average information entropy to reflect the different texture features of different SAR images. At the same time, the weights of the two generators are dynamically allocated according to the scale and texture features of the data set, which better captures the data features, enhances the network's generation ability and generation quality, and improves the network's accuracy and generalization ability.

[0059] S2. The optical remote sensing image with clouds is passed into the cloud detection model, and the cloud area is obtained based on the output image segmentation result.

[0060] It should be noted that this embodiment uses a cloud detection model to identify whether each pixel in an optical remote sensing image with clouds is cloud or background (non-cloud). The cloud detection model is obtained by training a UNET network by collecting historical optical remote sensing images with clouds and updating the network parameters by minimizing a weighted cross-entropy loss function.

[0061] Specifically, the UNET network consists of an encoder, a bottleneck layer, a decoder, skip connection layers, and an output layer. The encoder reduces the spatial dimension of the image by downsampling while increasing the number of feature channels, thereby extracting deep features. In this embodiment, the encoder includes four first convolutional modules, each consisting of two 3×3 convolutional layers and a 2×2 pooling layer with a stride of 2. The bottleneck layer further extracts and fuses image features from the final feature map output by the encoder and passes the output feature map to the decoder. The decoder gradually restores the spatial dimension of the image by upsampling while reducing the number of feature channels, thereby gradually reconstructing the image. In this embodiment, the decoder is symmetrical to the encoder and includes four second convolutional modules, each consisting of two 3×3 transposed convolutional layers. Skip connection layers directly connect feature maps from the encoder to the corresponding layers of the decoder, helping to preserve image details. The output layer is a convolutional layer that converts the decoder output into a segmentation map with the same size as the input image. Each pixel in the segmentation map corresponds to a predicted class. Cloud regions are then derived from the segmentation map.

[0062] It should be noted that the encoder and decoder in the UNET network use the ELU activation function in the convolutional layer. Considering that the ELU activation function has negative values, it will make the average activation close to 0, making the network learn faster and the gradient closer to the natural gradient.

[0063] Furthermore, in order to avoid the problem of network weight offset caused by excessive background pixels in optical remote sensing images, this embodiment adopts a weighted cross entropy loss function in back propagation, and the formula is as follows:

[0064]

[0065] Among them, L(Unet) represents the weighted cross entropy loss function, w c represents the weight of category c, category c corresponds to cloud or background, M represents the number of categories; N represents the total number of pixels in a historical optical remote sensing image with clouds; N c represents the number of pixels predicted to be of category c, y c Indicates the unique hot encoding of the true label. If the category of the pixel is consistent with the true category, it takes 1, otherwise it takes 0; p c represents the probability of predicting category c; exp(·) represents the exponential function.

[0066] As the above formula shows, the greater the number of background pixels in the optical image, the smaller its weight coefficient. This biases the model's weight toward cloud regions, better learning cloud features, improving the model's robustness and cloud detection accuracy. Furthermore, increasing the weight effectively mitigates the influence of high-frequency categories in the data samples, balancing the data samples across categories.

[0067] S3. An image corresponding to the cloud area is intercepted from the cloudless simulated optical image and replaced with the clouded optical remote sensing image, and a local filtering process is performed on the replaced optical remote sensing image to obtain a cloud-removed optical remote sensing image.

[0068] It should be noted that the cloud area in the optical remote sensing image with clouds is determined through step S2, and the cloudless simulated optical image converted from the SAR image is obtained through step S1. Then, according to the coordinate position of the cloud pixels in the cloud area, the image of the same position in the cloudless simulated optical image is obtained and replaced in the optical remote sensing image with clouds.

[0069] Since the gradient difference of the boundary pixel values ​​of the replacement area in the replaced optical remote sensing image is too large, a large amount of noise exists in the replaced optical remote sensing image. In this embodiment, the final de-clouded optical remote sensing image is obtained by performing a median filtering operation on the image within the minimum circumscribed rectangular frame of each replacement area.

[0070] Specifically, according to the upper left corner pixel coordinate (x min ,y min ) and the lower right pixel coordinate (x max ,y max ) to make a rectangular frame to obtain the minimum circumscribed rectangular frame of each replacement area; in each minimum circumscribed rectangular frame, a pixel grayscale value replacement operation is performed according to a preset odd-sized filter window, which is performed by sorting the grayscale values ​​of each pixel in the filter window to obtain a median value, and replacing the grayscale value of the center pixel of the filter window with the median value; then sliding the filter window with a step size of 1, and continuing the pixel grayscale value replacement operation until the filter window covers all pixels in the minimum circumscribed rectangular frame and completes the replacement of the grayscale values ​​of all center pixels.

[0071] For example, the filtering window is a 3×3 window, the grayscale value of the central pixel of the window is 89, the grayscale values ​​of the pixels in the window are sorted as follows: 646569717382888995, and the median is 73, then the grayscale value of the central pixel of the window is replaced from 89 to 73.

[0072] Compared to existing technologies, this embodiment provides a method for removing cloud from optical remote sensing images based on SAR images. Leveraging the advantages of SAR images, which can penetrate clouds and are unrestricted by weather conditions, this method uses a GAN network with two generators to convert SAR images into cloud-free optical remote sensing images. Furthermore, a deep learning model is used to detect clouds in cloud-covered optical remote sensing images. Finally, a replacement approach is used to remove clouds, improving the utilization rate of optical remote sensing images and enhancing the clarity of declouded images. The two generators extract and fuse features from both the overall structure and local texture perspectives, respectively. Furthermore, average information entropy is used to reflect the different texture features of different SAR images. The weights of the two generators are dynamically allocated based on the size and texture features of the dataset, better capturing data features, enhancing the network's generation capacity and quality, and improving its accuracy and generalization capabilities. During cloud detection in optical remote sensing images, weights are set based on the proportion of each category, biasing the model weights toward cloud regions. This allows for better learning of cloud feature information, improving cloud detection accuracy, and effectively reducing the influence of high-frequency categories in the data samples, balancing data samples across categories.

[0073] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0074] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for removing cloud from optical remote sensing images based on SAR images, characterized in that: The following steps are involved: SAR images and cloud-covered optical remote sensing images of the same area and time are collected; the SAR images are fed into an improved GAN network to obtain cloud-free simulated optical images; the improved GAN network comprises two generators and a discriminator; the first generator in the improved GAN network uses an encoder-decoder-based network to extract global structural features, while the second generator uses a residual network that introduces multi-scale fusion to extract local texture features and fuse them with global structural features; when constructing a dataset to train the improved GAN network, the weights of the two generators are dynamically assigned based on the dataset size and texture features, where the texture features are represented by the average information entropy of the SAR image; The cloud-containing optical remote sensing image is fed into a cloud detection model, and the cloud area is obtained based on the output image segmentation result; the cloud detection model is obtained by training a UNET network by collecting historical cloud-containing optical remote sensing images and updating the network parameters by minimizing a weighted cross-entropy loss function; the weights are set according to the proportion of each category; An image corresponding to the cloud area is captured from the cloudless simulated optical image and replaced in the optical remote sensing image with clouds. The replaced optical remote sensing image is subjected to local filtering to obtain a declouded optical remote sensing image, specifically by performing a median filtering operation on the image within the minimum circumscribed rectangular frame of each replacement area.

2. The cloud removal method for optical remote sensing images based on SAR images according to claim 1 is characterized in that: The improved GAN network is trained using a dataset constructed using multiple image pairs consisting of historical SAR images and historical cloud-free optical remote sensing images of the same area and time, where the historical SAR images serve as input samples and the historical cloud-free optical remote sensing images serve as real data.

3. The cloud removal method for optical remote sensing images based on SAR images according to claim 2 is characterized in that: In the improved GAN network, the first generator receives historical SAR images and random noise, uses an encoder-decoder-based network to extract overall structural features, generates a first optical image, and transmits it to the second generator; the second generator takes the first optical image and random noise as input, uses a residual network that introduces multi-scale fusion to extract local texture features and fuse them with overall structural features to generate a second optical image, which is transmitted to the discriminator; the discriminator receives the second optical image and the historical cloud-free optical remote sensing image, and outputs an identification result.

4. The method for removing cloud from optical remote sensing images based on SAR images according to claim 3, characterized in that: The loss function of the discriminator includes: real data identification loss and generated data identification loss, wherein the generated data identification loss is obtained by calculating the weights of the first generator and the second generator based on the average information entropy value of historical SAR images, and then weighted summing the losses of the two generators.

5. The method for removing cloud from optical remote sensing images based on SAR images according to claim 4, characterized in that: The average information entropy value of the historical SAR image is calculated by the following formula: Where ENT represents the average information entropy value, p(i, j) represents the probability of gray levels i and j appearing simultaneously in historical SAR images, and k represents the total number of gray levels.

6. The method for removing cloud from optical remote sensing images based on SAR images according to claim 4, characterized in that: According to the average information entropy value of the historical SAR images, the weights of the first generator and the second generator are calculated by the following formula: Among them, λ1 represents the weight of the first generator, λ2 represents the weight of the second generator, n represents the total number of historical SAR images in the dataset, m1 represents the number of historical SAR images whose average information entropy value is less than the average information entropy value of the dataset, and m2 represents the number of historical SAR images whose average information entropy value is greater than the average information entropy value of the dataset. The average information entropy value of the dataset is the average of the average information entropy values ​​of all historical SAR images in the dataset.

7. The method for removing cloud from optical remote sensing images based on SAR images according to claim 1 or 6, characterized in that: The weights in the weighted cross entropy loss function are calculated using the following formula: Among them, w c represents the weight of category c, category c corresponds to cloud or background, N represents the total number of pixels in the optical remote sensing image with clouds in history; N c represents the number of pixels predicted to be of category c, and exp(·) represents the exponential function.

8. The method for removing cloud from optical remote sensing images based on SAR images according to claim 1 or 6, characterized in that: The encoder and decoder in the UNET network adopt the ELU activation function in the convolutional layer.

Images