A cloud removal method for remote sensing images based on selective information fusion and generative adversarial networks

Abstract

This invention discloses a remote sensing image declouding method based on selective information fusion generative adversarial networks (GANs), belonging to the field of image processing technology. The method includes inputting cloudy remote sensing images from different time periods into a generator; generating feature maps through an initial module; extracting high-level features from the feature maps using an intermediate module, preserving information while reducing the feature map size; learning the importance of different channels using a channel attention mechanism, enhancing important features of the feature map by adjusting channel weights; then performing feature fusion using a selective information fusion module; and upgrading the feature mapping to the size of the original input image using multiple deconvolution layers. Finally, the generator gradually improves the quality of the generated image, resulting in improved declouding performance. This invention autonomously and selectively focuses on features in cloudless areas, thereby improving the restoration effect of landscapes in cloud-covered areas.

Description

Technical Field

[0001] This invention discloses a method for cloud removal from remote sensing images based on selective information fusion generative adversarial networks, belonging to the field of image processing technology. Background Technology

[0002] Remote sensing imagery plays a crucial role in acquiring comprehensive information about the Earth's surface, including topography, vegetation distribution, water resources, agriculture, and urban planning. However, due to natural weather factors, cloud cover often blurs these images, significantly reducing their quality and impacting their use in downstream tasks. Therefore, cloud removal has become a key aspect of remote sensing image processing. With the emergence of advanced image processing techniques, cloud removal has become a popular research area, and its importance and urgency have received increasing attention in recent years.

[0003] Cloud removal from remote sensing images can be viewed as a problem of reconstructing missing information. Based on the complementary information sources used in the reconstruction, current cloud removal algorithms can be broadly categorized into four types. These methods include spatial methods that utilize information from the same image, spectral methods that extract complementary information from other spectra, temporal methods that extract information from other data obtained at different time intervals from precise locations, and hybrid methods that combine the first three methods to remove complementary information. Temporal cloud removal methods have potential advantages, but they lack a selection mechanism in information fusion, which may leave cloud interference or redundant information. They can only partially utilize the features of cloudless areas, resulting in unsatisfactory cloud removal results. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud removal method for remote sensing images based on selective information fusion generative adversarial networks, so as to solve the problem that the cloud removal effect of existing remote sensing image cloud removal methods is not ideal.

[0005] A cloud removal method for remote sensing images based on selective information fusion and generative adversarial networks includes:

[0006] S1. Input the remote sensing images of cloudy conditions at different time points into the generator. The initial module generates feature maps, and the intermediate module extracts high-level features from the feature maps, retaining information while reducing the size of the feature maps.

[0007] S2. Use a channel attention mechanism to learn the importance of different channels, and enhance the important features of the feature map by adjusting the channel weights;

[0008] S3. Feature fusion is performed using a selective information fusion module;

[0009] S4. Use multiple deconvolutional layers to upgrade the feature map to the size of the original input image;

[0010] When performing deconvolution, reflection padding is used to ensure that edge pixels are processed. A 7x7 convolutional layer is used to generate the output image, and the number of output channels of the 7x7 convolutional layer matches the number of channels of the original input image.

[0011] The pixel values ​​of the output image are scaled using the Tanh activation function to limit the pixel value range to [-1, 1]. The generator produces a corresponding clear image. The discriminator determines whether the generated image is real or fake by comparing it with the real ground image. The output of the discriminator is the loss function of the generator, which guides the generator to produce remote sensing images. Through iterative training, the generator gradually improves the quality of the generated images, resulting in improved cloud removal.

[0012] The initial module contains a reflection padding layer, a convolutional layer, a normalization layer, and a ReLU activation layer. The reflection padding layer allows edge pixels to participate in the convolution operation, thus making full use of edge information.

[0013] S2 includes:

[0014] The feature map input channel attention mechanism uses an average pooling layer to compress the feature value of each channel into a scalar value. Then, through a convolutional layer, a ReLU activation layer, and another convolutional layer, the scalar value is mapped to a new vector with the same dimension as the number of channels in the input feature map.

[0015] The vector output from the mapping layer is used as input, and the Sigmoid function is applied to generate the corresponding weights for each channel. The original feature map is multiplied by the learned weights for each channel to obtain the adjusted feature map.

[0016] S3 includes:

[0017] The feature maps from remote sensing images at different time states are concatenated along the channel dimension and merged into a larger feature map. For the merged feature map, an average pooling layer operation is applied to reduce the number of channels and capture global information.

[0018] The feature maps are processed using a 1x1 convolutional layer with PReLU activation function to further extract useful features. Interest vectors are generated using a 1x1 convolutional layer with the same number of output channels as the original input features, thus weighing the importance of each feature map.

[0019] The softmax function is applied to the output of each 1x1 convolutional layer to compute an attention vector, which represents the importance weight of each feature map.

[0020] The attention vector is multiplied by the original feature map and summed to generate the final fused feature map.

[0021] Compared to existing technologies, this invention offers the following advantages: It proposes a remote sensing image declouding method based on selective information fusion (SEIF) generative adversarial networks (GANs). This method employs a channel attention mechanism that focuses on features of cloudless areas and a SEIF network that combines features from different temporal states to recover landscapes in cloudy areas. Through the channel attention mechanism, the model can autonomously and selectively focus on features of cloudless areas, thereby improving the recovery of landscapes in cloud-covered regions. The SEIF network can automatically adjust attention weights based on different remote sensing images, adapting to different cloud cover conditions and landscape features. This method demonstrates excellent performance and application potential in remote sensing image declouding, improving the quality and accuracy of cloud removal and providing more reliable results for remote sensing image analysis and applications, reaching the current advanced level. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention.

[0023] Figure 2 It is the channel attention module.

[0024] Figure 3 This refers to the network structure of the generator.

[0025] Figure 4 This is the network structure for the selective information fusion module. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] A cloud removal method for remote sensing images based on selective information fusion and generative adversarial networks includes:

[0028] S1. Input the remote sensing images of cloudy conditions at different time points into the generator. The initial module generates feature maps, and the intermediate module extracts high-level features from the feature maps, retaining information while reducing the size of the feature maps.

[0029] S2. A channel attention mechanism is used to learn the importance of different channels, enhancing important features of the feature map by adjusting channel weights. Channel attention is a deep neural network technique that learns the correlation between channels and enhances important features by adjusting channel weights. It is based on the observation that different channels in a deep neural network have different correlations; some channels are more critical for providing rich information, while others may contain noise or redundant information. Channel attention enhances important features of the deep neural network by learning the correlation between channels, reducing noise and redundant data. To enhance important features during feature extraction and reduce the impact of clouds on feature learning, this invention introduces a channel attention module. By adjusting channel weights, this module pays more attention to features in cloudless areas, thereby obtaining better feature representations for cloud removal.

[0030] S3. Feature fusion is performed using a selective information fusion module. Feature fusion is a technique used to combine information from different sources to improve model performance. However, in cloud removal from remote sensing images, previous feature fusion methods only involve simple summation, which is easily affected by noise interference and redundant information. To overcome this problem, this invention proposes a selective information fusion network that can selectively merge valuable information from multiple sources while minimizing the impact of irrelevant noise and redundant data.

[0031] S4. Use multiple deconvolutional layers to upgrade the feature map to the size of the original input image;

[0032] When performing deconvolution, reflection padding is used to ensure that edge pixels are processed. A 7x7 convolutional layer is used to generate the output image, and the number of output channels of the 7x7 convolutional layer matches the number of channels of the original input image.

[0033] The pixel values ​​of the output image are scaled using the Tanh activation function to limit the pixel value range to [-1, 1]. The generator produces a corresponding clear image. The discriminator determines whether the generated image is real or fake by comparing it with the real ground image. The output of the discriminator is the loss function of the generator, which guides the generator to produce remote sensing images. Through iterative training, the generator gradually improves the quality of the generated images, resulting in improved cloud removal.

[0034] The initial module contains a reflection padding layer, a convolutional layer, a normalization layer, and a ReLU activation layer. The reflection padding layer allows edge pixels to participate in the convolution operation, thus making full use of edge information.

[0035] S2 includes:

[0036] The feature map input channel attention mechanism uses an average pooling layer to compress the feature value of each channel into a scalar value. Then, through a convolutional layer, a ReLU activation layer, and another convolutional layer, the scalar value is mapped to a new vector with the same dimension as the number of channels in the input feature map.

[0037] The vector output from the mapping layer is used as input, and the Sigmoid function is applied to generate the corresponding weights for each channel. The original feature map is multiplied by the learned weights for each channel to obtain the adjusted feature map.

[0038] S3 includes:

[0039] The feature maps from remote sensing images at different time states are concatenated along the channel dimension and merged into a larger feature map. For the merged feature map, an average pooling layer operation is applied to reduce the number of channels and capture global information.

[0040] The feature maps are processed using a 1x1 convolutional layer with PReLU activation function to further extract useful features. Interest vectors are generated using a 1x1 convolutional layer with the same number of output channels as the original input features, thus weighing the importance of each feature map.

[0041] The softmax function is applied to the output of each 1x1 convolutional layer to compute an attention vector, which represents the importance weight of each feature map.

[0042] The attention vector is multiplied by the original feature map and summed to generate the final fused feature map.

[0043] This invention utilizes a loss function from Conditional Generative Adversarial Networks (GANs), consisting of adversarial loss and conditional loss. In GAN training, adversarial loss is a crucial component; the generator aims to produce realistic samples to deceive the discriminator. In contrast, the discriminator attempts to distinguish between real and generated samples. Conditional loss is added to ensure that the images generated by the generator tend to approximate realistic images.

[0044] L GAN (G,D)=E x,y～Pdata(x,y) [log D(x,y)]+E x,z～Pdata(x,z) [log(1-D(x,G(x,z))];

[0045] Conditional loss is typically calculated using the L1 loss function, which can be expressed as follows.

[0046] L1(G) = ||yG(x)||1;

[0047] Therefore, the final loss function consists of two parts: adversarial loss and conditional loss, and their effects are balanced using a hyperparameter λ. In this invention, λ is set to 1.

[0048] Loss = L GAN (G,D)+λL1(G).

[0049] The adversarial loss is denoted by D, the discriminator, which is a model used to distinguish between real and generated samples. D takes an input sample and outputs a probability between 0 and 1, representing the likelihood that the sample is a real sample. G represents the generator, which takes random noise or a latent space vector as input and generates a synthetic sample. The goal of the generator is to generate samples that can fool the discriminator, causing it to be mistaken for a real sample. x is a real sample, which is a sample from the real data distribution. z is the latent space vector or random noise, which is the input to the generator, represented here as the input cloudy image, and y represents the real cloudless image.

[0050] Figure 1 In this invention, the adversarial network consists of a generator and a discriminator. The generator takes cloud images from multiple time periods as input and generates corresponding clear images through multiple convolution and deconvolution operations. The discriminator determines whether the generated image is real or fake by comparing it with real ground images. Figure 2 In this process, by adjusting the channel weights, the channel attention module focuses more on the features of cloudless areas, thereby obtaining better feature representations for cloud removal. Figure 3 In the generator, the network structure consists of an initial, intermediate, channel attention mechanism, selective information fusion network, and final module. Figure 4 In this process, the selective information fusion module receives a list of feature maps from remote sensing images at different time states, stitches them together along the channel dimension, and then applies a set of weighted fusion operations to selectively combine these features.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cloud removal from remote sensing images based on selective information fusion and generative adversarial networks, characterized in that, include: S1. Input the remote sensing images of cloudy conditions under different time states into the generator, which consists of an initial module, an intermediate module, a channel attention module, a selective information fusion module, and a final module; the remote sensing images of cloudy conditions under different time states generate feature maps through the initial module, and the intermediate module extracts high-level features from the feature maps, retaining information while reducing the size of the feature maps to obtain feature maps; S2. Through the channel attention module, the importance of different channels is learned using the channel attention mechanism, and the important features of the feature map are enhanced by adjusting the channel weights; S3. Feature fusion is performed using a selective information fusion module; S4. The final module uses multiple deconvolutional layers to upgrade the feature map to the size of the original input image; When performing deconvolution, reflection padding is used to ensure that edge pixels are processed. A 7x7 convolutional layer is used to generate the output image. The number of output channels of the 7x7 convolutional layer matches the number of channels of the remote sensing image of cloudy conditions at different time points. The pixel values ​​of the output image are scaled using the Tanh activation function to limit the pixel value range to [-1, 1], thus obtaining the image generated by the generator. The discriminator compares the image generated by the generator with the real ground image to determine whether the generated image is a real image or a fake image. Based on the loss function, iterative training is conducted to gradually improve the quality of the image generated by the generator, thereby obtaining an improved cloud removal effect. Step S3 specifically includes: The feature maps from remote sensing images of different time periods are concatenated along the channel dimension to obtain the merged feature maps. For the merged feature maps, an average pooling layer is applied to capture global information. The feature map is then processed by a 1x1 convolutional layer with PReLU activation function to further extract useful features, and an interest vector is generated by a 1x1 convolutional layer with the same number of output channels as the feature map. The attention vector is calculated by applying the softmax function to the attention vector, and the attention vector represents the importance weight of each feature map. The attention vector is multiplied by each feature map in the feature map list and summed to generate the final fused feature map.

2. The remote sensing image cloud removal method based on selective information fusion generative adversarial networks according to claim 1, characterized in that, The initial module contains a reflection padding layer, a convolutional layer, a normalization layer, and a ReLU activation layer. The reflection padding layer allows edge pixels to participate in the convolution operation to make full use of edge information.

3. The remote sensing image cloud removal method based on selective information fusion generative adversarial networks according to claim 1, characterized in that, Step S2 includes: The feature map is input into the channel attention module, and the feature value of each channel is compressed into a scalar value using an average pooling layer. The scalar value is then mapped to a new vector through a convolutional layer, a ReLU activation layer, and another convolutional layer. The dimension of the new vector is the same as the number of channels of the feature map input to the channel attention module. The new vector is used as input, and the Sigmoid function is applied to generate the corresponding weights for each channel. The feature map of the input channel attention module is multiplied by the learning weights of each channel to obtain the adjusted feature map.

Images