Cloud removal method for single remote sensing image based on edge information

Abstract

A cloud removal method based on edge information combined with a generative adversarial network (GAN) process separately processes edge and color information in remote sensing images, making the cloud removal process more efficient. Cloud occlusion simulation is first performed, and then two GAN networks are trained separately to obtain two generative models with different functions, improving the quality of the final cloud removal results.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing and pattern recognition, and in particular to a single remote sensing image cloud removal method based on edge information. Background Art

[0002] Remote sensing imagery opens the door to analyzing objects on the Earth's surface. It plays a key role in studying physical environmental systems, human systems, and the interactions between them. Examples of using remote sensing imagery for physical environmental research include forest inventory, wetland identification, grassland mapping, and water resource statistics. Regarding human systems, remote sensing has been used to study urban surfaces, public health, and epidemiology. Regarding human-environment interactions, high-resolution remote sensing has been used to investigate land degradation, precision agriculture, water and water pollution, and natural disasters (e.g., earthquakes, hurricanes, floods, and landslides). Cloud cover over the Earth's surface is a common natural phenomenon. Cloud obstruction significantly reduces the effective information obtained from remote sensing images, hindering their post-processing and further analysis and interpretation. Cloud obstruction is particularly problematic in real-time feature extraction tasks such as military strikes and disaster relief. Therefore, how to detect the cloud position of remote sensing images more comprehensively and accurately, and remove cloud occlusions in the images as much as possible to show clear ground object information, has very important application value for further analysis and research of remote sensing images.

[0003] Cloud removal from single remote sensing images primarily relies on spatial-based cloud removal methods. Traditional spatial-based cloud removal methods can be further categorized as diffusion-based and patch-based. Diffusion-based methods propagate data from unobstructed areas to cloud-obstructed areas through a diffusion process modeled using typical differential operators. Sample-based methods, on the other hand, extract the most similar sample blocks from a set of uncontaminated cloud patches to fill in the missing areas. The Criminisi algorithm is a classic patch-based method that surpasses other traditional algorithms in restoration accuracy and speed, but suffers from the problem of undesirable artifacts. Over the past two decades, deep learning has flourished in the field of image processing. Deep learning-based technologies have been widely applied in object detection, image segmentation, and image editing, bringing greater convenience and intelligence to people's lives. In remote sensing image processing, there are relatively mature remote sensing image processing methods. Therefore, deep learning networks can naturally achieve better performance in remote sensing image preprocessing. For example, deep neural networks for image inpainting can be used to remove clouds. Before 2018, deep learning methods used in image inpainting primarily focused on filling regular rectangular holes. For irregularly shaped areas, these methods required significant post-processing, requiring significant time and effort. In 2018, the NVIDIA team proposed a new network based on partial convolutional layers that can inpaint irregular holes. This made it possible to apply this network to remove irregularly shaped clouds from remote sensing images. However, inpainting based solely on color information can produce blurred boundaries.

[0004] Edges are an important component of images, and edge information is critical in the practice of cloud removal. Currently, there is no research on using edge information for cloud removal in deep neural networks. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a single remote sensing image cloud removal method based on edge information. The method uses two GAN networks, edge generation GAN and color filling GAN, to decouple the high-frequency edge information and low-frequency color information of the remote sensing image, making the cloud removal process more reasonable. It also trains two generative adversarial networks and ultimately achieves an effective improvement in the quality of the cloud-removed image.

[0006] The present invention comprises two parts: a training phase and a cloud removal phase. The training phase comprises the following steps:

[0007] The first step is to obtain cloud-free remote sensing data;

[0008] The second step is to perform cloud occlusion simulation processing on the data, which includes first using Perlin noise to generate a simulated cloud layer and its corresponding cloud mask, and then combining the cloud mask with cloud-free data to simulate the cloud occlusion scene to obtain a cloud simulation dataset;

[0009] The third step is to perform image preprocessing on the cloud simulation dataset to obtain the grayscale image of the remote sensing image after the simulated cloud occlusion, and use the Canny edge detection algorithm to obtain the edge map of the remote sensing image after the cloud occlusion;

[0010] In the fourth step, the cloud occlusion mask, the grayscale image of the remote sensing image after simulated cloud occlusion, and the edge map of the remote sensing image after cloud occlusion are superimposed in the channel dimension as the input of the edge generation GAN; the complete edge map of the corresponding cloud-free remote sensing image is used as the true value label data;

[0011] The fifth step is to input the training data and label data into the edge generation GAN to train the network parameters. The network structure is as follows Figure 1 G1 and D1 in:

[0012] The first part of G1 is the encoder, which primarily consists of a spectral normalization convolutional layer. This spectral normalization stabilizes the GAN training process. The middle part consists of eight residual convolutional blocks with dilated convolutions, corresponding to a dilation factor of 2. This is followed by an upsampling inverse convolution, and finally by a sigmoid activation layer that outputs the edge prediction map. The discriminator follows the same architecture as the Markovian discriminator PatchGAN and can judge 70×70 image patches within an image.

[0013] Step 6: Save the network parameters trained in step 5 as our edge generation model.

[0014] In the seventh step, the generator model of the edge generation GAN trained in the sixth step is used to obtain a more complete edge map predicted by the cloud simulation remote sensing image;

[0015] In the eighth step, the edge map predicted in the seventh step and the cloud occlusion simulation remote sensing image are connected in the channel dimension as the input of the color filling GAN, and the corresponding cloud-free remote sensing image is used as the true value label data;

[0016] The ninth step is to input the training data and label data into the color filling GAN to train the network parameters. The network structure is as follows Figure 1 G2 and D2 in the figure are consistent with the network structure in the fifth step;

[0017] Step 10. Save the network parameters trained in step 9 as our color filling model.

[0018] The cloud removal phase includes the following steps:

[0019] The first step is to obtain the remote sensing image to be cloud removed;

[0020] The second step is to perform cloud detection on the cloud-containing remote sensing image to obtain the cloud mask;

[0021] The third step is to superimpose a cloud mask on the real cloud-containing remote sensing image and obtain the corresponding grayscale image. The Canny edge detection algorithm is used to obtain the edge map of the remote sensing image after cloud occlusion.

[0022] The fourth step is to superimpose the cloud mask, the grayscale image of the remote sensing image after the cloud mask is blocked, and the edge map of the remote sensing image after the cloud mask is blocked in the channel dimension as the input of the trained edge generation GAN model, and obtain a more complete predicted edge map through the model;

[0023] In the fifth step, the edge map predicted in the fourth step and the remote sensing image occluded by the cloud mask are connected in the channel dimension as the input of the trained color filling GAN model, and the result image after cloud removal is obtained through the model.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, compared with multispectral and multi-temporal cloud removal methods, the present invention trains a neural network on a single RGB remote sensing image. Users only need to input a single remote sensing image of the RGB channel to obtain the result, which solves the problem of difficulty in acquiring multispectral and multi-temporal remote sensing images.

[0026] Second, the latest deep learning methods are applied to the field of remote sensing image cloud removal to generate cloud occlusion simulation masks to simulate cloud occlusion scenes, solving the problem that remote sensing image cloud removal datasets are difficult to organize and obtain.

[0027] Third, we apply advanced GAN networks to the cloud removal process. The generator in the GAN network transforms the feature representation of the sample in the original space into a higher-level feature space, automatically learning a hierarchical feature representation, and finally obtaining the final cloud-removed image through the decoder. Furthermore, the GAN network utilizes game theory methods and a Markov discriminator, making the images generated by the trained generator model even clearer and more realistic.

[0028] Fourth, the edge and color information in remote sensing images are processed separately, resulting in more reasonable results. Using a dedicated GAN model to refine the edge information and then another GAN model to fill in the color information provides a more refined approach. Furthermore, due to the appropriate decoupling approach, the algorithm remains robust to large cloud occlusions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the training phase of the present invention.

[0030] Figure 2 This is the overall structure diagram of the GAN network used in this invention. G1 and D1 are the generator and discriminator of the edge generation GAN network, and G2 and D2 are the generator and discriminator of the color filling GAN network.

[0031] Figure 3 1 is a flow chart of the cloud removal phase of the present invention.

[0032] Figure 4 This is a schematic diagram of the simulated cloud removal results of the present invention, where a is the simulated cloud occlusion map; b is the Criminisi algorithm result; c is the partial convolutional network result; d is the edge generation GAN network result map; e is the color filling GAN network result map; and f is the true value map.

[0033] Figure 5 This is a schematic diagram of the real cloud removal results of the present invention, where a is the real cloud-containing image; b is the combined image after cloud detection; c is the real cloud-free image; d is the result of the Criminisi algorithm; e is the result of a partial convolutional network; and f is the result of the GAN network used in the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below with reference to the accompanying drawings and embodiments, but the scope of protection of the present invention should not be limited thereto.

[0035] The present invention comprises two parts: a training phase and a cloud removal phase. The training phase comprises the following steps:

[0036] The first step is to obtain cloud-free remote sensing data;

[0037] The second step is to perform cloud occlusion simulation processing on the data, which includes first using Perlin noise to generate a simulated cloud layer and its corresponding cloud mask, and then combining the cloud mask with cloud-free data to simulate the cloud occlusion scene to obtain a cloud simulation dataset;

[0038] The third step is to perform image preprocessing on the cloud simulation dataset to obtain the grayscale image of the remote sensing image after the simulated cloud occlusion, and use the Canny edge detection algorithm to obtain the edge map of the remote sensing image after the cloud occlusion;

[0039] In the fourth step, the cloud occlusion mask, the grayscale image of the remote sensing image after simulated cloud occlusion, and the edge map of the remote sensing image after cloud occlusion are superimposed in the channel dimension as the input of the edge generation GAN; the complete edge map of the corresponding cloud-free remote sensing image is used as the true value label data;

[0040] The fifth step is to input the training data and label data into the edge generation GAN to train the network parameters. The network structure is as follows Figure 2 G1 and D1 in:

[0041] The first part of G1 is the encoder, which is mainly a spectral normalization convolution layer. Adding spectral normalization can stabilize the training process of the GAN network; the middle part is 8 residual convolution blocks with dilated convolutions, and the dilation coefficient corresponding to dilated convolutions is 2. Then there is the up-sampled inverse convolution, and finally the sigmoid activation layer that outputs the edge prediction map. As for the discriminator, it follows the same architecture as the Markov discriminator PatchGAN and can judge the 70×70 image blocks in the image; the Adma training method is used in training, and the loss function is the generation adversarial loss (L adv,1 ) and feature matching loss (L fm );

[0042] Step 6: Save the network parameters trained in step 5 as our edge generation model;

[0043] In the seventh step, the generator model of the edge generation GAN trained in the sixth step is used to obtain a more complete edge map predicted by the cloud simulation remote sensing image;

[0044] In the eighth step, the edge map predicted in the seventh step and the cloud occlusion simulation remote sensing image are connected in the channel dimension as the input of the color filling GAN, and the corresponding cloud-free remote sensing image is used as the true value label data;

[0045] The ninth step is to input the training data and label data into the color filling GAN to train the network parameters. The network structure is as follows Figure 1 G2 and D2 in the network structure are consistent with the fifth step; the Adma training method is used in training, and the loss function is the generation of adversarial loss (L adv,2 ), L1 loss, perceptual loss and style loss.

[0046] Step 10: Save the network parameters trained in step 9 as the color filling model.

[0047] Edge coefficient comparison test:

[0048] The experimental data consists of remote sensing images from the Sentinel-2 satellite. We simulated cloud occlusion on the cloud-free remote sensing images in this dataset. Since we first detect clouds to determine the cloud mask position and then remove clouds, the cloud occlusion simulation here directly overlays the cloud-free remote sensing images with a cloud occlusion mask with a pixel value of 255 to simulate cloud contamination. The training dataset consists of 79,085 remote sensing images randomly combined with 20,000 cloud occlusion masks. The validation and test sets consist of 5,764 and 1,000 remote sensing images, respectively, combined in a specific order with a fixed cloud occlusion mask. Each remote sensing image has a pixel size of 256.

[0049] Edge detection in edge-connected generative adversarial networks is a key component of the network. Remote sensing images differ from conventional images in their complex and diverse content, so the setting of the Canny edge detection coefficient α requires experimental analysis. A larger α coefficient results in fewer detected edges, while a larger α coefficient results in more detected edges. The edge detection results directly influence the edge generation results of the edge generation GAN network. This experiment compared the α coefficients to 1, 1.5, and 2. The evaluation metrics used were MSE, PSNR, and SSIM. The results are shown in Table 1.

[0050] Table 1 Quantitative evaluation results of cloud removal simulation using three different α coefficients

[0051] α MSE PSNR(dB) SSIM (%) 1 38.75 34.44 91.62 1.5 36.48 35.40 92.31 2 36.78 35.20 92.13

[0052] It can be seen that different α coefficients also have an impact on the final cloud removal results. Figure 3 The results of α = 1.5 show that the best cloud removal result is achieved in the simulation dataset. Therefore, the α coefficient is set to 1.5.

[0053] Comparison of cloud removal performance using different methods:

[0054] Similarly, a cloud removal comparison experiment was conducted on the previous dataset using the Criminisi algorithm, a partial convolutional network, and the edge-connected generative adversarial network of the present invention. The quantitative evaluation results are shown in Table 2.

[0055] Table 2 Quantitative evaluation results of simulated cloud removal using different methods

[0056] method MSE PSNR(dB) SSIM (%) Criminisi algorithm 51.75 34.16 90.52 Partial Convolutional Network 35.25 34.88 90.47 Edge-Connected Generative Adversarial Networks 36.48 35.40 92.31

[0057] It can be seen that the test results of cloud removal based on edge connection generative adversarial network are better than the other two methods in quantitative evaluation results on PSNR and SSIM, which verifies that it is a very effective idea to separate edge information from other color information for prediction.

[0058] Real cloud removal experiment:

[0059] In order to verify the cloud removal performance of different methods on real cloud-containing remote sensing images, a remote sensing image with a large area is taken as an example to compare the methods. The obtained visualization results are shown in the figure below. Figure 3 As shown in the figure, it can be seen that the method based on edge information generative adversarial network is significantly better than the other two methods in terms of cloud removal visual effect, without very obvious flaws, so this method still has certain practical value in the actual application of cloud removal.

[0060] Further quantitative evaluation and comparison are shown in Table 3.

[0061] Table 3 Quantitative evaluation results of real cloud removal by different methods

[0062] method MSE PSNR(dB) SSIM (%) Original cloud map 6114.73 10.27 66.33 Criminisi algorithm 199.47 25.13 79.32 Partial Convolutional Network 263.70 23.92 80.74 Edge-Connected Generative Adversarial Networks 137.54 26.75 86.20

[0063] It can be seen that compared with the evaluation indicators of cloud images and cloud-free images, the images after cloud removal by the three methods have all improved in these evaluation indicators, but the method of the present invention has a more obvious improvement and is more robust.

Claims

1. A method for removing cloud from a single remote sensing image based on edge information, characterized in that: include: Obtain and process the edge information of remote sensing images to obtain the edge generation GAN network; Obtain and process the color information of remote sensing images to obtain a color filling GAN network; Decoupling the high-frequency edge information and low-frequency color information of remote sensing images makes the cloud removal process more reasonable; The edge generation GAN network and color filling GAN network are trained separately to generate two adversarial networks, effectively improving the image quality after cloud removal. The method specifically includes a training phase and a cloud removal phase, and the steps are as follows: Training phase: The first step is to obtain cloud-free remote sensing data; The second step is to perform cloud occlusion simulation processing on the cloud-free remote sensing data to obtain a cloud simulation data set; The third step is to perform image preprocessing on the cloud simulation dataset to obtain the grayscale image of the remote sensing image after the simulated cloud occlusion, and use the Canny edge detection method to obtain the edge map of the remote sensing image after the cloud occlusion; In the fourth step, the cloud occlusion mask, the grayscale image of the remote sensing image after simulated cloud occlusion, and the edge map of the remote sensing image after cloud occlusion are superimposed in the channel dimension as the input of the edge generation GAN; the complete edge map of the corresponding cloud-free remote sensing image is used as the true value label data; The fifth step is to input the training data and label data into the edge generation GAN to train the network parameters; Step 6: Save the network parameters trained in step 5 as the edge generation model; In the seventh step, the generator model of the edge generation GAN trained in the sixth step is used to obtain a more complete edge map predicted by the cloud simulation remote sensing image; In the eighth step, the edge map predicted in the seventh step and the cloud occlusion simulation remote sensing image are connected in the channel dimension as the input of the color filling GAN, and the corresponding cloud-free remote sensing image is used as the true value label data; Step 9: Input the training data and label data into the color filling GAN to train the network parameters; Step 10: Save the network parameters trained in step 9 as the color filling model; Cloud removal phase: The first step is to obtain the remote sensing image to be cloud removed; The second step is to perform cloud detection on the cloud-containing remote sensing image to obtain the cloud mask; The third step is to superimpose a cloud mask on the real cloud-containing remote sensing image and obtain the corresponding grayscale image. The Canny edge detection method is used to obtain the edge map of the remote sensing image after cloud occlusion. The fourth step is to superimpose the cloud mask, the grayscale image of the remote sensing image after the cloud mask is blocked, and the edge map of the remote sensing image after the cloud mask is blocked in the channel dimension as the input of the trained edge generation GAN model, and obtain a more complete predicted edge map through the model; In the fifth step, the edge map predicted in the fourth step and the remote sensing image occluded by the cloud mask are connected in the channel dimension as the input of the trained color filling GAN model, and the result image after cloud removal is obtained through the model.

2. The cloud removal method for a single remote sensing image based on edge information according to claim 1 is characterized in that: The training network parameter structure includes: The encoder is a spectral normalization convolutional layer. Adding spectral normalization can stabilize the training process of the GAN network; The discriminator is used to judge the image blocks in the image. The Adma training method is used in training, and the loss function is the generation of adversarial loss L adv,1 and feature matching loss L fm .

3. The cloud removal method for a single remote sensing image based on edge information according to claim 1, characterized in that: The larger the coefficient α of the Canny edge detection is, the fewer edge lines are detected, and vice versa. The result of edge detection directly affects the edge generation result of the edge generation GAN network.

Images