A remote sensing image thin cloud removal method and device based on a generative adversarial network

Abstract

The application discloses a remote sensing image thin cloud removing method based on a generative adversarial network, and comprises the following steps: acquiring a remote sensing image to be cloud removed; and acquiring a cloud removing result of the remote sensing image according to the remote sensing image to be cloud removed by using a remote sensing image thin cloud removing model. The remote sensing image thin cloud removing model comprises a feature enhancement dense residual module and a prompt learning module. The feature enhancement dense residual module is used for distinguishing cloud layer features with uneven distribution in the remote sensing image. The prompt learning module is used for extracting ground object color features in the remote sensing image. The cloud removing result of the remote sensing image is obtained. In the thin cloud removing method, the feature enhancement dense residual module and the prompt learning module can overcome the difficulty in recovering ground object color features caused by cloud layer ground object spectrum mixing, accurately remove thin cloud layers, and improve the performance of the model.

Description

Technical Field

[0001] This application relates to the field of computer vision, the field of image restoration technology, and in particular to a method and apparatus for removing thin clouds from remote sensing images based on generative adversarial networks. Background Technology

[0002] With the advancement of remote sensing theory and technology, remote sensing information across various bands is widely used in surface observation. Multi-orbit altitude observation networks composed of satellite constellations provide abundant data for multiple fields, with optical remote sensing satellites particularly excelling in resource, climate, and disaster monitoring. However, the average annual cloud cover exceeds 60%, and optical satellites are unable to penetrate cloud layers due to cloud interference, affecting data acquisition. In remote sensing images, clouds are categorized as thin or thick. Thin clouds specifically refer to low-thickness, highly transparent cloud layers in remote sensing images; in areas covered by thin clouds, the outlines and textures of ground features can be vaguely seen with the naked eye. Addressing the interference of thin clouds in optical remote sensing images, restoring low-quality images affected by clouds, and improving data availability are crucial for the real-time performance and widespread application of remote sensing data.

[0003] Currently, the more mainstream thin cloud removal method is based on deep learning. Its main idea is to use neural networks to remove thin cloud layers from the input data, and to restore pixel-level ground feature recovery in the input image through downsampling and upsampling, thereby achieving effective thin cloud removal from remote sensing images.

[0004] However, due to the mixing of spectral characteristics of clouds and ground features, and the uneven distribution of thin clouds, it is difficult to restore the colors of ground features, which reduces the cloud removal performance of the model. Summary of the Invention

[0005] The purpose of this application is to provide a method for removing thin clouds from remote sensing images based on generative adversarial networks, in order to solve the problems in the existing technology. By employing a cue learning mechanism and a channel attention mechanism to fuse multi-dimensional features, the interaction between cue parameters and input features is realized. The color information generated by the cue learning module is introduced into the image features, effectively restoring the color features of ground objects, improving the accuracy of thin cloud removal, and achieving the goal of effectively removing thin clouds from remote sensing images.

[0006] To solve the above-mentioned technical problems, this application adopts the following technical solution:

[0007] In a first aspect, a method for removing thin clouds from remote sensing images based on generative adversarial networks includes the following steps to obtain a remote sensing image of the clouds to be removed;

[0008] Based on the remote sensing image of the cloud to be removed, the cloud removal result of the remote sensing image is obtained by using the remote sensing image thin cloud removal model;

[0009] The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to distinguish unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image.

[0010] A further aspect of this application is that the training method for the remote sensing image thin cloud removal model includes:

[0011] Acquire multiple labeled remote sensing images and construct a remote sensing image dataset;

[0012] The remote sensing image dataset is divided into a training set and a test set. The training set is input into the remote sensing image thin cloud removal model for training, and the test set is input into the remote sensing image thin cloud removal model for verification.

[0013] A further aspect of this application is that the processing procedure of the feature-enhanced dense residual module includes...

[0014] The feature-enhanced dense residual module is based on the residual structure. It learns the distribution characteristics of the cloud layer through residual connections and adds a channel attention mechanism to separate the cloud layer features of the remote sensing image.

[0015] A further proposed solution is that the processing steps of the prompting learning module include:

[0016] The texture features of ground objects are predicted by a prompting learning mechanism to extract the color features of ground objects in the remote sensing image, and the cloud removal result of the remote sensing image is output by a fusion encoder.

[0017] A further aspect of this application is that the remote sensing image thin cloud removal model processing procedure includes:

[0018] The remote sensing image is input into the feature enhancement dense residual module to obtain feature map F. in ;

[0019] The feature map F in The feature map is obtained by inputting the prompt learning module.

[0020] The feature map obtained from the deepest layer of the network through the prompting learning module The feature map F obtained by the second-deepest layers of the network through enhanced dense residual modules in Cascade to obtain intermediate feature map F i mid ;

[0021] Repeat the above operations to obtain the cloud removal result.

[0022] A further aspect of this application is that the construction process of the feature-enhanced dense residual module includes the following steps.

[0023] The feature map F in The feature map F = ReLU is obtained after processing by convolution and ReLU activation function. in );

[0024] Then the feature map F = ReLU(F in The channel attention weight map F is obtained by sequentially processing the data through average pooling, convolution, ReLU activation function, convolution again, and Sigmoid activation function. channel =Sigmoid(F in );

[0025] Channel attention weight map F channel =Sigmoid(F in For the feature map F = ReLU(F in The intermediate feature map F1 = F is obtained by weighting the features. chamel ·F, · indicates that pixel values ​​are multiplied;

[0026] Finally, F1 = F chamel The feature map F2 is obtained by sequentially processing F through convolution, ReLU activation function, convolution, and Sigmoid activation function. The feature maps are then summed to obtain...

[0027] A further aspect of this application is that the construction process of the prompting learning module includes the following steps.

[0028] The feature map F in The feature map F is obtained by sequentially processing the data through global average pooling, fully connected layers, and the Softmax function. Softmax =Softmax(F);

[0029] The feature map F in The advance vector and F Softmax Multiplying by Softmax(F) yields F = F softmax ·F prompt , where · represents pixel multiplication;

[0030] Let F = F prompt ·F in and F in After convolution and cascading, followed by the FADRB module, deconvolution, and convolution, feature F is obtained. out =Conv 1×1 (DeConv 1×1 (FADRB(F in ))).

[0031] The application itself provides a further solution, wherein obtaining the thin cloud removal model for the remote sensing image includes:

[0032] A thin cloud removal model for remote sensing images is constructed, including a generator and a discriminator. The model in the generator includes a feature enhancement dense residual module and a cue learning module.

[0033] During training, the model parameters are dynamically updated based on the loss, and the model parameters with the smallest loss value are saved to obtain the trained remote sensing image thin cloud removal model.

[0034] Secondly, an apparatus for removing thin clouds from remotely sensed images, characterized in that the apparatus comprises:

[0035] The acquisition module is used to acquire remote sensing image data of the cloud to be removed;

[0036] The cloud removal module is used to obtain the cloud removal results of the remote sensing image based on the remote sensing image to be clouded, using a remote sensing image thin cloud removal model.

[0037] The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to separate the unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image.

[0038] Thirdly, a computer-readable storage medium, characterized in that the storage medium includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-described method.

[0039] Compared with the prior art, the beneficial effects achieved by this application are as follows:

[0040] The thin cloud removal method and apparatus provided in this application can overcome the difficulty of restoring the color features of ground features caused by the spectral mixing of cloud features, realize the accurate removal of thin cloud layers by the model, and improve the performance of the model. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the feature-enhanced dense residual module in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the prompt learning module in the embodiments of this application;

[0043] Figure 3 This is a schematic diagram of the generative adversarial network discriminator process in the embodiments of this application;

[0044] Figure 4 This is a schematic diagram of the remote sensing image thin cloud removal method in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the remote sensing image thin cloud removal device module in the application embodiment. Detailed Implementation

[0046] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0047] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0048] Example 1

[0049] As attached Figures 1 to 4 This application discloses a method for removing thin clouds from remote sensing images based on generative adversarial networks, comprising the following steps: acquiring a remote sensing image with clouds to be removed; and using a remote sensing image thin cloud removal model to obtain the cloud removal result of the remote sensing image based on the remote sensing image with clouds to be removed. The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to distinguish unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image.

[0050] In some embodiments, as shown in the appendix Figure 3 and 4 A method for removing thin clouds from remote sensing images based on generative adversarial networks is presented, and its basic implementation process is as follows:

[0051] (1) Select a dataset. In this example, the RICE1 dataset is selected as the baseline;

[0052] (2) Construct the generator and discriminator in the thin cloud removal model of remote sensing image, wherein the thin cloud removal model of remote sensing image includes a feature enhancement dense residual module and a cue learning module;

[0053] (3) Construct a feature-enhanced dense residual module;

[0054] (4) Construct a prompt learning module.

[0055] Reference Appendix Figure 1 The specific operation of the feature enhancement dense residual module is as follows:

[0056] (31) Let the input feature map be F∈H×W×C. Where H, W, and C are the height, width, and number of channels of the feature map, respectively.

[0057] (32) The feature map first passes through a 1×1 convolutional layer and a ReLU activation function: F = ReLU(F in ). Where ReLU(·) represents the modified linear unit activation function.

[0058] (33) Perform average pooling on the feature map: F avg =AvgPool(F). Where AvgPool(·) represents average pooling of the feature map along the channel dimension, and F... avg Its shape is H×W×1.

[0059] (34) The feature map is passed through a 1×1 convolutional layer, a ReLU activation function, another 1×1 convolutional layer, and a Sigmoid activation function to obtain the channel attention weight feature map: F channel =Sigmoid(F in ). Where Sigmoid(·) represents the Sigmoid activation function.

[0060] (35) Weight the feature map obtained in step 32 using the channel feature map obtained in step 34: F1 = F chamel ·F. Where · represents pixel multiplication.

[0061] (36) The feature map first passes through a 1×1 convolutional layer and a ReLU activation function, then through another 1×1 convolutional layer and a Sigmoid activation function. The resulting feature map F2 is then summed with the final output feature map obtained in step 35: in, This indicates pixel addition. (See attached reference.) Figure 2 The specific steps for building a prompt-learning module are as follows:

[0062] (41) Let the input feature map be F in ∈H×W×C. Where H, W, and C are the height, width, and number of channels of the feature map, respectively.

[0063] (42) Perform global average pooling on the feature map: F avg =AvgPool(F in The pooling result is then passed through a fully connected layer and a Softmax function: F Softmax =Softmax(F avg ). Where Softmax(·) represents the normalized exponential function.

[0064] (43) Use the cue vector F obtained from the input imageprompt Weight the feature map obtained in step 42: F = F softmax ·F prompt Where, · indicates pixel multiplication.

[0065] (44) After passing through a 3×3 convolutional layer, the resulting feature map is concatenated with the original feature map of the input module: F = Concat(F in Conv 3×3 (F)), Concat(·) represents a cascading operation, Conv 3×3 This represents a 1×1 convolution operation.

[0066] The resulting feature map is obtained by passing through a FADRB module, a 3×3 deconvolution layer, and finally a 1×1 convolution layer: F out =Conv 1×1 (DeConv 1×1 (FADRB(F in ))).

[0067] Where FADRB(·) represents the feature-enhanced dense residual module, DeConv 1×1 (·) indicates a 1×1 deconvolution layer.

[0068] Example 2

[0069] This application discloses an apparatus for removing thin clouds from remote sensing images, the apparatus comprising:

[0070] The acquisition module is used to acquire remote sensing image data of the cloud to be removed;

[0071] The cloud removal module is used to obtain the cloud removal results of the remote sensing image based on the remote sensing image to be clouded, using a remote sensing image thin cloud removal model.

[0072] The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to separate the unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image.

[0073] Example 3

[0074] This application discloses a computer-readable storage medium, characterized in that the storage medium includes a stored program, wherein the program controls the device where the storage medium is located to execute the above-described method when it is running.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for removing thin clouds from remote sensing images based on generative adversarial networks, characterized in that, Includes the following steps: Acquire remote sensing images of the clouds to be removed; Based on the remote sensing image of the cloud to be removed, the cloud removal result of the remote sensing image is obtained by using the remote sensing image thin cloud removal model; The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to separate the unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image. The acquisition of the remote sensing image thin cloud removal model includes: A thin cloud removal model for remote sensing images is constructed, including a generator and a discriminator. The model in the generator includes a feature enhancement dense residual module and a cue learning module. During training, the model parameters are dynamically updated based on the loss, and the model parameters with the smallest loss value are saved to obtain the trained remote sensing image thin cloud removal model. The remote sensing image thin cloud removal model processing procedure includes: The remote sensing image is input into the feature enhancement dense residual module to obtain the feature map. F in ; The feature map F in The feature map is obtained by inputting the prompt learning module. ; The feature map obtained from the deepest layer of the network through the prompting learning module The feature maps obtained by the second-deepest layers of the network through the feature enhancement dense residual module F in Cascade to obtain intermediate feature maps ; Repeat the above operations to obtain the cloud removal result; The construction process of the feature-enhanced dense residual module includes the following steps: The feature map F in Feature maps are obtained after processing with convolution and ReLU activation functions. ; Then the feature map The channel attention weight map is obtained by sequentially processing the data through average pooling, convolution, ReLU activation function, convolution again, and Sigmoid activation function. ; Channel attention weight map For feature maps Weighted summation is performed to obtain intermediate feature maps. ; This represents the multiplication of pixel values; Finally The feature map is obtained by sequentially processing the data through convolution, ReLU activation function, convolution, and Sigmoid activation function. F 2 The feature maps are summed to obtain ; The construction process of the prompting learning module includes the following steps: The feature map F in The feature map is obtained by sequentially processing the data through global average pooling, fully connected layers, and the Softmax function. ; The feature map F in The advance vector and Multiplying together ,in This indicates pixel multiplication; Will and F in Features are obtained through convolution and concatenation, followed by the FADRB module, deconvolution, and convolution. .

2. The method for removing thin clouds from remote sensing images based on generative adversarial networks according to claim 1, characterized in that... , The training method for the remote sensing image thin cloud removal model includes: Acquire multiple labeled remote sensing images and construct a remote sensing image dataset; The remote sensing image dataset is divided into a training set and a test set. The training set is input into the remote sensing image thin cloud removal model for training, and the test set is input into the remote sensing image thin cloud removal model for verification.

3. The method for removing thin clouds from remote sensing images based on generative adversarial networks according to claim 1, characterized in that, The processing steps of the feature-enhanced dense residual module include: The feature-enhanced dense residual module is based on the residual structure. It learns the distribution characteristics of the cloud layer through residual connections and adds a channel attention mechanism to separate the cloud layer features of the remote sensing image.

4. The method for removing thin clouds from remote sensing images based on generative adversarial networks according to claim 3, characterized in that, The processing procedure of the prompt learning module includes: The texture features of ground objects are predicted by a prompting learning mechanism to extract the color features of ground objects in the remote sensing image, and the cloud removal result of the remote sensing image is output by a fusion encoder.

5. An apparatus for removing thin clouds from remotely sensed images, used to implement the method as described in any one of claims 1 to 4, characterized in that, The device includes, The acquisition module is used to acquire remote sensing image data of the cloud to be removed; The cloud removal module is used to obtain the cloud removal results of the remote sensing image based on the remote sensing image to be clouded, using a remote sensing image thin cloud removal model. The remote sensing image thin cloud removal model includes a feature enhancement dense residual module and a cue learning module. The feature enhancement dense residual module is used to separate the unevenly distributed cloud features in the remote sensing image, and the cue learning module is used to extract the color features of ground features in the remote sensing image to obtain the cloud removal result of the remote sensing image.

6. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program, when running, controls the device on which the storage medium is located to perform the method as described in any one of claims 1 to 4.

Images