A satellite remote sensing image cloud noise removal method, system and device

Abstract

The present application relates to the technical field of deep learning, in particular to a satellite remote sensing image cloud noise removal method, system and device, through constructing a training data set and a test data set, the cloud image is subjected to HIS color space transformation, a double-channel generative adversarial neural network is constructed, the double-channel generative adversarial neural network is trained, finally, the satellite remote sensing image of the RGB color space and the HIS color space of the test data set is input into the double-channel generator network of the trained double-channel generative adversarial neural network, and the satellite remote sensing image with cloud noise removed is obtained, the double-channel generative adversarial neural network is trained, the threshold setting and continuous adjustment optimization required by the Laplace algorithm can be effectively avoided, even in the case of high complexity and diversification of the satellite space aerial image background, a good satellite remote sensing image with cloud noise removed can be obtained, the clarity of the cloud coverage area is improved, and the effect of target ground object interpretation is achieved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, computer vision, and satellite remote sensing image cloud removal and enhancement processing, specifically to a method, system, and apparatus for removing cloud noise from satellite remote sensing images. Background Technology

[0002] With the continuous development of remote sensing technology, remote sensing images are widely used in military, disaster emergency response, and ecological environment monitoring. Modern remote sensing technology mostly uses visible light, infrared, and microwave bands to acquire images. Except for microwave remote sensing, which can penetrate clouds, most optical remote sensing is affected by cloud noise to varying degrees. Cloud noise interference blurs cloud-covered areas in images and even makes it difficult to obtain ground feature information, seriously affecting the interpretation and analysis of remote sensing images. Removing cloud noise from remote sensing images will improve the clarity of cloud-covered areas, thereby achieving the goal of improving the accuracy of image interpretation and the identification of target ground features.

[0003] In some existing technologies, satellite remote sensing methods for removing cloud noise based on manually designed features, such as the Laplacian algorithm, require setting thresholds and other parameters and making continuous adjustments. These methods have poor versatility and adaptability. Furthermore, for satellite aerial images, the backgrounds are often highly complex and diverse, making it difficult to consistently obtain images with good cloud noise removal. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, system and apparatus for removing cloud noise from satellite remote sensing images, so as to solve the problem in the prior art that it is impossible to reliably obtain images with good cloud noise removal due to the highly complex and diverse backgrounds of satellite aerial images.

[0005] According to a first aspect of the present invention, a method for removing cloud noise from satellite remote sensing images is provided, comprising:

[0006] Step S1: Construct the training dataset and test dataset, including:

[0007] Acquire satellite remote sensing data at the same high resolution at different time phases;

[0008] Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image.

[0009] Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images;

[0010] Step S2: Perform HIS color space transformation on the cloud image;

[0011] Step S3: Construct a dual-channel generative adversarial neural network; wherein the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network;

[0012] The cloudy image is input into a dual-channel generator network to obtain a cloudless image;

[0013] The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image;

[0014] Step S4: Train a dual-channel generative adversarial neural network, including:

[0015] The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images.

[0016] The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network;

[0017] Step S5: Input the satellite remote sensing images in the RGB color space and HIS color space of the test dataset from step S1 into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed.

[0018] Preferably, the HIS color space transformation of the cloud image specifically involves:

[0019] For cloud-covered images, perform HIS color space transformation processing to convert the RGB color space to the HIS color space.

[0020] The formula for calculating hue H is:

[0021]

[0022] in,

[0023]

[0024] Wherein, R, G, and B represent the values ​​of the R, G, and B channels in the RGB color space of the satellite image, respectively;

[0025] The formula for calculating brightness I is:

[0026]

[0027] The formula for calculating saturation S is:

[0028]

[0029] Preferably, the step of inputting the clouded image into a dual-channel generator network to obtain a generated cloudless image specifically involves:

[0030] A cloud-covered image is input into the RGB and HIS color space channels of a dual-channel generator network. Features of the input image are extracted using a multi-scale feature fusion method, and the features of the input image are fused using a multi-source feature fusion method. A cloudless image is then generated through the dual-channel generator network.

[0031] Preferably, the multi-scale feature fusion method specifically includes:

[0032] Features of the input image are extracted using three dilated convolutional layers with dilation factors of 1, 2, and 3, a kernel size of 3×3, and 64 channels, generating a 64-channel feature map. The three channel feature maps are then superimposed to obtain a 192-channel feature map, which is then used to generate another 64-channel feature map.

[0033] Preferably, the multi-source feature fusion method specifically comprises:

[0034] It is composed of three dilated residual convolutional modules with dilation factors of 1, 2 and 3 respectively. Each dilated residual convolutional module includes a dilated convolutional layer, a ReLU activation function layer and a dilated convolutional layer. The kernel size of each dilated convolutional layer is 3×3. The output features of the first and third dilated convolutional layers are superimposed and connected to a dilated convolutional layer with a kernel size of 3×3 to generate a 64-channel feature map.

[0035] Preferably, the dual-channel generator network generates cloudless images by using a convolutional layer with a kernel size of 3×3 to obtain the final output of the network and obtain a cloudless satellite image.

[0036] Preferably, the discriminator neural network consists of convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, convolutional layers, fully connected layers, and fully connected layers in sequence. Each convolutional layer has a 3×3 kernel size and 64, 64, 128, 128, 64, and 64 channels respectively. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The two pooling layers use average pooling with a 2×2 kernel size. The two fully connected layers have 1000 and 1 neurons respectively. Finally, the output uses the sigmoid function for probability analysis and normalization of the discrimination result. The sigmoid function formula is:

[0037]

[0038] Preferably, training the dual-channel generative adversarial neural network includes: inputting satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 into a dual-channel generator network to output cloudless images; inputting the output cloudless images and cloudless images into a discriminator neural network to train the dual-channel generative adversarial neural network, specifically:

[0039] PyTorch was used as the deep neural network learning framework, and NVIDIA TITAN×P was used as the GPU for model training. The entire training process used the Adam optimizer to learn the network parameters. The initial learning rate was set to 0.001, and the learning rate descent method was warm-up. The batch size was set to 12, the maximum number of training epochs was 200 (all training data completed one computation), and the test interval was 1 epoch. During training, the ratio of parameter updates for the generator and the discriminator was 4:1, that is, the generator was updated 4 times and the discriminator was updated 1 time. After multiple iterations, the training was completed when the discriminator's discrimination result for the input cloudless image stabilized at a probability of 0.5, resulting in a trained dual-channel generative adversarial neural network.

[0040] According to a second aspect of the present invention, a satellite remote sensing image cloud noise removal system is provided, comprising:

[0041] The first building module is used to construct the training and testing datasets, including:

[0042] Acquire satellite remote sensing data at the same high resolution at different time phases;

[0043] Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image.

[0044] Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images;

[0045] The transformation module is used to perform HIS color space transformation on the cloud image;

[0046] The second building module is used to build a dual-channel generative adversarial neural network; wherein, the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network;

[0047] The cloudy image is input into a dual-channel generator network to obtain a cloudless image;

[0048] The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image;

[0049] The training module, used to train a dual-channel generative adversarial neural network, includes:

[0050] The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images.

[0051] The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network;

[0052] The removal module is used to input satellite remote sensing images in the RGB color space and HIS color space of the test dataset into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed.

[0053] According to a third aspect of the present invention, a satellite remote sensing image cloud noise removal apparatus is provided, comprising: the method or the system described above.

[0054] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0055] This invention constructs training and testing datasets, performs HIS color space transformation on cloud-covered images, constructs a dual-channel generative adversarial neural network (GAN), trains the GAN, and finally inputs satellite remote sensing images in RGB and HIS color spaces from the testing dataset into the dual-channel generator network of the trained GAN to obtain satellite remote sensing images with cloud noise removed. The technical solution of this invention, by training the dual-channel GAN, effectively avoids the drawbacks of the Laplacian algorithm, which requires setting thresholds and continuous adjustments and optimizations. Even in situations where the background of satellite aerial images is highly complex and diverse, it can obtain satellite remote sensing images with good cloud noise removal, improve the clarity of cloud-covered areas, and achieve the effect of identifying target features.

[0056] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0058] Figure 1This is a flowchart illustrating a method for removing cloud noise from satellite remote sensing images according to an exemplary embodiment;

[0059] Figure 2 This is a flowchart illustrating the construction of a dual-channel generative adversarial neural network according to an exemplary embodiment;

[0060] Figure 3 This is a flowchart illustrating the generation of a cloud-free image using a dual-channel generator network according to an exemplary embodiment;

[0061] Figure 4 This is a flowchart illustrating a multi-scale feature fusion method according to an exemplary embodiment;

[0062] Figure 5 This is a flowchart illustrating a multi-source feature fusion method according to an exemplary embodiment;

[0063] Figure 6 This is a flowchart illustrating the normalization process of a discriminator neural network for the discrimination result, according to an exemplary embodiment.

[0064] Figure 7 This is a schematic block diagram of a satellite remote sensing image cloud noise removal system according to an exemplary embodiment. Detailed Implementation

[0065] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0066] Example 1

[0067] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for removing cloud noise from satellite remote sensing images according to an exemplary embodiment, such as... Figure 1 As shown, the method includes:

[0068] Step S1: Construct the training dataset and test dataset, including:

[0069] Acquire satellite remote sensing data at the same high resolution at different time phases;

[0070] Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image.

[0071] Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images;

[0072] Step S2: Perform HIS color space transformation on the cloud image;

[0073] Step S3: Construct a dual-channel generative adversarial neural network; wherein the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network;

[0074] The cloudy image is input into a dual-channel generator network to obtain a cloudless image;

[0075] The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image;

[0076] Step S4: Train a dual-channel generative adversarial neural network, including:

[0077] The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images.

[0078] The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network;

[0079] Step S5: Input the satellite remote sensing images in the RGB color space and HIS color space of the test dataset from step S1 into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed.

[0080] It is understood that the technical solution provided in this embodiment constructs a training dataset and a test dataset, performs HIS color space transformation on the cloud-covered image, constructs a dual-channel generative adversarial neural network, trains the dual-channel generative adversarial neural network, and finally inputs the satellite remote sensing images in the RGB color space and HIS color space of the test dataset into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain a satellite remote sensing image with cloud noise removed. The technical solution of this embodiment, by training the dual-channel generative adversarial neural network, can effectively avoid the drawbacks of the Laplacian algorithm, which requires setting a threshold and continuous adjustment and optimization. Even in situations where the background of satellite aerial images is highly complex and diverse, it can obtain a good satellite remote sensing image with cloud noise removed, improve the clarity of cloud-covered areas, and achieve the effect of identifying target features.

[0081] In practical application, the HIS color space transformation of the cloud image specifically involves:

[0082] For cloud-covered images, perform HIS color space transformation processing to convert the RGB color space to the HIS color space.

[0083] The formula for calculating hue H is:

[0084]

[0085] in,

[0086]

[0087] Wherein, R, G, and B represent the values ​​of the R, G, and B channels in the RGB color space of the satellite image, respectively;

[0088] The formula for calculating brightness I is:

[0089]

[0090] The formula for calculating saturation S is:

[0091]

[0092] It should be noted that δ has no special meaning in the formula for calculating H; it is used simply to make the formula more harmonious and clear. Therefore, when B ≤ G,

[0093]

[0094] It should be noted that color space transformation is performed on satellite remote sensing images. Images with clouds undergo HIS color space transformation, converting the RGB color space to the HIS color space. The HIS color space can separate color and achromatic information. Its brightness information includes the spatial information contained in the image and all the energy reflected by ground objects. Hue and saturation represent information such as the color category of the target and the purity of the image, respectively. Brightness (I) corresponds to relative brightness. In extreme cases, a brightness of 0 corresponds to black, and a brightness of 1 corresponds to white.

[0095] Please see Figure 2 , Figure 2 This is a flowchart illustrating the construction of a dual-channel generative adversarial neural network according to an exemplary embodiment. It should be noted that the discriminator neural network's role is to determine whether a sample comes from a cloudless image or a cloudless image generated by the generator network, to supervise the generator network in generating higher-quality images, and when the discriminator cannot determine the source of the sample, it indicates that the cloudless image generated by the generator network is better.

[0096] Please see Figure 3 , Figure 3 This is a flowchart illustrating a dual-channel generator network generating a cloud-free image according to an exemplary embodiment.

[0097] In practice, the process of inputting the clouded image into a dual-channel generator network to obtain a cloudless image is as follows:

[0098] Cloudy images are input into the RGB and HIS color space channels of a dual-channel generator network. Features of the input images are extracted using a multi-scale feature fusion method, and the features of the input images are fused using a multi-source feature fusion method. Cloudless images are then generated through the dual-channel generator network.

[0099] Please see Figure 4 , Figure 4 This is a flowchart illustrating a multi-scale feature fusion method according to an exemplary embodiment.

[0100] In practical application, the multi-scale feature fusion method is specifically as follows:

[0101] Features of the input image are extracted using three dilated convolutional layers with dilation factors of 1, 2, and 3, a kernel size of 3×3, and 64 channels, generating a 64-channel feature map. The three channel feature maps are then superimposed to obtain a 192-channel feature map, which is then used to generate another 64-channel feature map.

[0102] Please see Figure 5 , Figure 5 This is a flowchart illustrating a multi-source feature fusion method according to an exemplary embodiment.

[0103] In practical application, the multi-source feature fusion method is specifically as follows:

[0104] It is composed of three dilated residual convolutional modules with dilation factors of 1, 2 and 3 respectively. Each dilated residual convolutional module includes a dilated convolutional layer, a ReLU activation function layer and a dilated convolutional layer. The kernel size of each dilated convolutional layer is 3×3. The output features of the first and third dilated convolutional layers are superimposed and connected to a dilated convolutional layer with a kernel size of 3×3 to generate a 64-channel feature map.

[0105] It should be noted that the features extracted from the input image in this embodiment are specifically features such as brightness, edge, texture, and color, which will not be listed one by one here.

[0106] It should be noted that the technical solution in this embodiment further extracts deeper features from the solution in the previous step, and the structural designs of the two modules are different; the superposition of different channels will produce different effects.

[0107] In practice, the dual-channel generator network generates cloudless images by using a convolutional layer with a kernel size of 3×3 to obtain the final output of the network, thus obtaining a cloudless satellite image.

[0108] It should be noted that the dual-channel generator network is used to remove cloud noise; that is, it takes an image with clouds as input and outputs an image without clouds. The solution in this embodiment is designed to better extract image features.

[0109] Please see Figure 6 , Figure 6 This is a flowchart illustrating the normalization process of a discriminator neural network for the discrimination result, according to an exemplary embodiment.

[0110] In practice, the discriminator neural network consists of convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, convolutional layers, fully connected layers, and fully connected layers. Each convolutional layer has a 3×3 kernel size and 64, 64, 128, 128, 64, and 64 channels respectively. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The two pooling layers use average pooling with 2×2 kernels. The two fully connected layers have 1000 and 1 neurons respectively. Finally, the output uses the sigmoid function for probability analysis and normalization of the discrimination result. The sigmoid function formula is:

[0111]

[0112] It should be noted that the convolution kernels in different convolutional layers are different and are used to extract different features; the output of a convolutional layer is the input of the next convolutional layer or pooling layer, and the resulting features are different. These different features include brightness, edges, texture, and color, etc., which will not be listed one by one in this embodiment.

[0113] In practical application, training the dual-channel generative adversarial neural network includes: inputting satellite remote sensing images in the RGB color space and the HIS color space of the training dataset from step S1 into a dual-channel generator network to output cloudless images; inputting the output cloudless images and cloudless images into a discriminator neural network to train the dual-channel generative adversarial neural network, specifically as follows:

[0114] PyTorch was used as the deep neural network learning framework, and NVIDIA TITAN×P was used as the GPU for model training. The entire training process used the Adam optimizer to learn the network parameters. The initial learning rate was set to 0.001, and the learning rate descent method was warm-up. The batch size was set to 12, the maximum number of training epochs was 200 (all training data completed one computation), and the test interval was 1 epoch. During training, the ratio of parameter updates for the generator and the discriminator was 4:1, that is, the generator was updated 4 times and the discriminator was updated 1 time. After multiple iterations, the training was completed when the discriminator's discrimination result for the input cloudless image stabilized at a probability of 0.5, resulting in a trained dual-channel generative adversarial neural network.

[0115] It should be noted that the above parameters are those listed in this embodiment, and other parameters calculated using the method of this application are all within the protection scope of this invention.

[0116] It should be noted that the technical solution of this embodiment, by introducing a multi-scale feature fusion module and a multi-source feature fusion module, fuses the features of two input feature channels from different scales and different feature levels, fully extracts and fuses image context features, improves the stability of cloud noise removal, and generates clear satellite remote sensing images without cloud noise. It has good versatility and good image de-clouding and denoising performance.

[0117] Example 2

[0118] Please see Figure 7 , Figure 7 This is a schematic block diagram illustrating a satellite remote sensing image cloud noise removal system 700, as shown in an exemplary embodiment. Figure 7 As shown, the satellite remote sensing image cloud noise removal system 700 includes:

[0119] The first building module 701 is used to build the training and testing datasets, including:

[0120] Acquire satellite remote sensing data at the same high resolution at different time phases;

[0121] Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image.

[0122] Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images;

[0123] Transformation module 702 is used to perform HIS color space transformation on the cloud image;

[0124] The second construction module 703 is used to construct a dual-channel generative adversarial neural network; wherein, the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network;

[0125] The cloudy image is input into a dual-channel generator network to obtain a cloudless image;

[0126] The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image;

[0127] Training module 704, used to train a dual-channel generative adversarial neural network, includes:

[0128] The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images.

[0129] The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network;

[0130] The removal module 705 is used to input the satellite remote sensing images in the RGB color space and HIS color space of the test dataset into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed.

[0131] In practice, the dual-channel generator network includes RGB color space channels and HIS color space channels, and combines a multi-scale feature fusion module, a multi-source feature fusion module, and an image reconstruction module to generate cloudless images; wherein, the image reconstruction module is part of the dual-channel generator network module.

[0132] It should be noted that since the implementation methods of each module in this embodiment can be found in the relevant description in Embodiment 1, this embodiment will not repeat them.

[0133] It is understood that the technical solution provided in this embodiment includes a first construction module 701, which constructs training and testing datasets; a transformation module 702, which performs HIS color space transformation on the cloud-covered image; a second construction module 703, which constructs a dual-channel generative adversarial neural network; a training module 704, which trains the dual-channel generative adversarial neural network; and finally, a removal module 705, which inputs satellite remote sensing images in RGB and HIS color spaces from the testing dataset into the dual-channel generator network of the trained dual-channel generative adversarial neural network to obtain a satellite remote sensing image with cloud noise removed. The technical solution of this invention, by training the dual-channel generative adversarial neural network, can effectively avoid the drawbacks of the Laplacian algorithm, which requires setting thresholds and continuous adjustments and optimizations. Even in situations where the background of satellite aerial images is highly complex and diverse, it can obtain a good satellite remote sensing image with cloud noise removed, improving the clarity of cloud-covered areas and achieving the effect of identifying target features.

[0134] Example 3

[0135] A satellite remote sensing image cloud noise removal device includes the above-described method or system.

[0136] It should be noted that since the implementation methods and beneficial effects of each module in this embodiment can be found in the relevant description in Embodiment 1, this embodiment will not repeat them here.

[0137] The computer-readable storage media disclosed in this embodiment include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0138] It should be noted that since the implementation methods and beneficial effects of each module in this embodiment can be found in the relevant description in Embodiment 1, this embodiment will not repeat them here.

[0139] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0140] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0141] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0142] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0143] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0144] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0145] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0146] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for removing cloud noise from satellite remote sensing images, characterized in that, include: Step S1: Construct the training dataset and test dataset, including: Acquire satellite remote sensing data at the same high resolution at different time phases; Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image. Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images; Step S2: Perform HIS color space transformation on the cloud image; Step S3: Construct a dual-channel generative adversarial neural network; wherein the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network; The cloudy image after HIS color space transformation and the cloudy image from the manually labeled satellite remote sensing image are input into a dual-channel generator network to obtain the generated cloudless image. The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image; Step S4: Train a dual-channel generative adversarial neural network, including: The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images. The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network; Step S5: Input the satellite remote sensing images in the RGB color space and HIS color space of the test dataset in step S1 into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed. Specifically, inputting the clouded image into a dual-channel generator network to obtain a cloudless image involves the following steps: A cloud-covered image is input into the RGB and HIS color space channels of a dual-channel generator network. Features of the input image are extracted using a multi-scale feature fusion method, and the features of the input image are fused using a multi-source feature fusion method. A cloudless image is then generated through the dual-channel generator network.

2. The method according to claim 1, characterized in that, The HIS color space transformation of the cloud image is specifically performed as follows: For cloud-covered images, perform HIS color space transformation processing to convert the RGB color space to the HIS color space. The formula for calculating hue H is: ； in, ； Wherein, R, G, and B represent the values ​​of the R, G, and B channels in the RGB color space of the satellite image, respectively; The formula for calculating brightness I is: The formula for calculating saturation S is: 。 3. The method according to claim 1, characterized in that, The multi-scale feature fusion method is specifically as follows: Features of the input image are extracted using three dilated convolutional layers with dilation factors of 1, 2, and 3, a kernel size of 3×3, and 64 channels, generating a 64-channel feature map. The feature maps of the three channels are then superimposed to obtain a 192-channel feature map, which is then used to generate a 64-channel feature map.

4. The method according to claim 1, characterized in that, The multi-source feature fusion method is specifically as follows: It is composed of three dilated residual convolutional modules with dilation factors of 1, 2 and 3 respectively. Each dilated residual convolutional module includes a dilated convolutional layer, a ReLU activation function layer and a dilated convolutional layer. The kernel size of each dilated convolutional layer is 3×3. The output features of the first and third dilated convolutional layers are superimposed and connected to a dilated convolutional layer with a kernel size of 3×3 to generate a 64-channel feature map.

5. The method according to claim 1, characterized in that, The dual-channel generator network generates cloudless images by using a 3×3 convolutional layer to obtain the final output of the network, thus obtaining a cloudless satellite image.

6. The method according to claim 1, characterized in that, The discriminator neural network consists of convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, pooling layers, convolutional layers, convolutional layers, fully connected layers, and fully connected layers. Each convolutional layer has a 3×3 kernel size and 64, 64, 128, 128, 64, and 64 channels respectively. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The two pooling layers use average pooling with 2×2 kernels. The two fully connected layers have 1000 and 1 neurons respectively. Finally, the output uses the sigmoid function for probability analysis and normalization of the discrimination result. The sigmoid function formula is: 。 7. The method according to claim 1, characterized in that, The training of the dual-channel generative adversarial neural network includes: inputting satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 into a dual-channel generator network to output cloudless images; inputting the output cloudless images and cloudless images into a discriminator neural network to train the dual-channel generative adversarial neural network, specifically: PyTorch was used as the deep neural network learning framework, and NVIDIA TITAN ×P was used as the GPU for model training. The entire training process used the Adam optimizer to learn the network parameters. The initial learning rate was set to 0.001, and the learning rate descent method was warm-up. The batch size was set to 12, the maximum number of training epochs was 200, and the test interval was 1 epoch. During training, the ratio of parameter updates for the generator to the discriminator was 4:1, that is, the generator was updated 4 times and the discriminator was updated 1 time. After multiple iterations, the training was completed when the discriminator's discrimination result for the input cloudless image stabilized at a probability of 0.5, resulting in a trained dual-channel generative adversarial neural network.

8. A satellite remote sensing image cloud noise removal system, characterized in that, include: The first building module is used to construct the training and testing datasets, including: Acquire satellite remote sensing data at the same high resolution at different time phases; Satellite remote sensing data from the RGB bands were selected to create a satellite remote sensing image. Based on manually labeled satellite remote sensing images, satellite remote sensing images corresponding to the same geographic coordinates are matched to form cloud-covered and cloudless image pairs, which are then assigned to the training dataset and the test dataset; wherein, the manually labeled satellite remote sensing images include: cloud-covered images and cloudless images; The transformation module is used to perform HIS color space transformation on the cloud image; The second building module is used to build a dual-channel generative adversarial neural network; wherein, the dual-channel generative adversarial neural network includes: a dual-channel generator network and a discriminator neural network; The cloudy image after HIS color space transformation and the cloudy image from the manually labeled satellite remote sensing image are input into a dual-channel generator network to obtain the generated cloudless image. The generated cloudless image and the cloudless image are input into a discriminator neural network to determine the source of the input image; The training module, used to train a dual-channel generative adversarial neural network, includes: The satellite remote sensing images in the RGB color space and the HIS color space of the training dataset in step S1 are input into the dual-channel generator network to output cloudless images. The output cloudless image and cloudless image are input into the discriminator neural network to train the dual-channel generative adversarial neural network; The removal module is used to input satellite remote sensing images in the RGB color space and HIS color space of the test dataset into the dual-channel generator network in the trained dual-channel generative adversarial neural network to obtain satellite remote sensing images with cloud noise removed. The second building module is also used to input the clouded image into the RGB color space channel and HIS color space channel of the dual-channel generator network, extract the features of the input image according to the multi-scale feature fusion method, and fuse the features of the input image using the multi-source feature fusion method to generate a cloudless image through the dual-channel generator network.

9. A satellite remote sensing image cloud noise removal device, characterized in that, include: The method as described in any one of claims 1-7 or the system as described in claim 8.

Images