A method for cloud identification and removal in remote sensing images based on neural network

Through the neural network-based method, a cloud removal network is built, which solves the problem of insufficient adaptability of cloud removal in remote sensing images, and realizes efficient cloud removal and information recovery, which is suitable for remote sensing image processing in multiple scenarios.

CN114663303BActive Publication Date: 2024-11-22AEROSPACE DONGFANGHONG SATELLITE
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210255108.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-15
Publication Date
2024-11-22
Estimated Expiration
2042-03-15

AI Technical Summary

Technical Problem

The existence of clouds in remote sensing images leads to information loss, reducing the utilization efficiency of optical remote sensing images. The prior art has problems of insufficient adaptability and waste of resources in cloud removal.

Method used

Using a neural network-based method, the cloud layer removal network is constructed by building cloud edge estimation unit, cloud layer thickness estimation unit, feature enhancement unit and cloud layer discrimination unit, and the cloud layer removal network is constructed using the feature loss function of the de-cloud map, the de-cloud map reconstruction loss function, the cloud layer thickness loss function and The cloud edge loss function is trained to realize the adaptive removal of clouds in remote sensing images.

Benefits of technology

It realizes automatic discrimination and adaptive removal of cloud layers in remote sensing images, with a wide range of adaptations and can handle cloud-based remote sensing images in multiple scenes, with high efficiency in removing cloud layers, and no color distortion. The cloud-based image is natural and real.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114663303B_ABST
    Figure CN114663303B_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image cloud discrimination and removal method based on a neural network. The steps include: selecting a number of cloud-free remote sensing images, simulating and generating clouds to obtain a cloud thickness map, adding the cloud-free image to obtain a cloud image, extracting the edge of the cloud thickness map to form a remote sensing image data set; building a cloud edge estimation unit and a cloud thickness estimation unit; building a feature enhancement unit and a cloud discrimination unit; building a cloud removal network for generating a de-clouded image from a single cloud image; training the cloud removal network based on a de-clouded image feature loss function, a de-clouded image reconstruction loss function, a cloud thickness loss function and a cloud edge loss function, and training the cloud discrimination unit based on a cloud discrimination unit loss function; importing model parameters after training to obtain a de-clouded remote sensing image from a single cloud image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of image processing technology and deep learning technology, and in particular to a remote sensing image cloud layer identification and removal method based on a neural network. Background Art

[0002] Remote sensing satellite data has the characteristics of high resolution and high information content, and is widely used in many fields, such as traffic monitoring and weather forecasting. However, due to the sensor, optical remote sensing images are often affected by factors such as clouds and fog. Some electromagnetic waves in the visible light band are blocked, and only a part of the electromagnetic waves reach the optical sensor, resulting in the ground information in the image being covered and valuable information in the image being lost, which greatly reduces the utilization efficiency of optical remote sensing images and causes a waste of resources. Therefore, cloud removal in remote sensing images has become an urgent problem to be solved.

[0003] Researchers have studied algorithms for removing clouds from remote sensing images, which are generally divided into transformation methods, hypothesis prior methods, and deep learning methods. Representative transformation methods are generally homomorphic filtering methods. Since information such as clouds exists in the low-frequency domain of images, the influence of clouds can be removed by enhancing high-frequency information and suppressing low-frequency information; the hypothesis prior method proposes hypothesis information through statistical analysis of cloud images and cloud-free images, and uses the hypothesis information as a priori to infer remote sensing images without clouds; the deep learning method uses the powerful feature learning ability of convolutional neural networks to restore cloud-free images, and has the characteristics of strong adaptability and robustness. Therefore, studying remote sensing image cloud removal methods based on deep neural networks has practical significance and application value. Summary of the invention

[0004] The technical solution solved by the present invention is: to overcome the shortcomings of the prior art, and to provide a remote sensing image cloud discrimination and removal method based on a neural network, so as to realize automatic discrimination of cloud thickness in remote sensing images, and to adaptively remove clouds in remote sensing images without the need for additional prior information.

[0005] The technical solution of the present invention is: a method for distinguishing and removing clouds from remote sensing images based on a neural network, the method comprising:

[0006] 1) Select several cloud-free remote sensing images, simulate and generate clouds to obtain cloud thickness maps, and add them to the cloud-free images to obtain cloud images. Extract the edges of the cloud thickness maps to obtain cloud edge maps. The cloud-free images, cloud images, cloud thickness maps, and cloud edge maps form a remote sensing image dataset and are divided into training sets, validation sets, and test sets.

[0007] 2) Build a cloud edge estimation unit to estimate the edge feature information of the cloud layer, and build a cloud thickness estimation unit to generate cloud thickness information and provide it to the subsequent network to complete cloud removal;

[0008] 3) Build a feature enhancement unit to enhance the cloud features of the image, and build a cloud discrimination unit to determine whether the clouds in the image have been completely removed;

[0009] 4) Based on the cloud edge estimation unit, cloud thickness estimation unit, feature enhancement unit and cloud discrimination unit, a cloud removal network is built to generate a de-clouded image from a single cloud image;

[0010] 5) The cloud removal network is trained based on the de-clouding feature loss function, de-clouding reconstruction loss function, cloud thickness loss function and cloud edge loss function, and the cloud discrimination unit is trained based on the cloud discrimination unit loss function;

[0011] 6) Import the trained model parameters and obtain the cloud-free remote sensing image from a single remote sensing image with clouds.

[0012] In step 1), the specific steps of constructing the remote sensing image data set are:

[0013] 11) Select m cloud-free remote sensing images, generate clouds by simulating m cloud thickness maps and add them to the cloud-free remote sensing images to obtain m cloud remote sensing images, extract the edges of the cloud thickness maps using the Sobel operator to obtain m cloud edge maps, segment all images according to the size of N×N, and obtain n cloud-free remote sensing images H, n cloud remote sensing images K, n cloud thickness images G and n cloud edge images J to form a data set U, where m, n and N are all positive integers;

[0014] 12) Expanding the data set, specifically: randomly selecting a% of images from the data set U in step 11), and randomly rotating the images, including 90° rotation, 180° rotation, 270° rotation, and mirror flipping; after the rotation operation, the expanded images are obtained and added to the data set U to form a remote sensing image data set V, where a is a positive integer;

[0015] 13) Divide the remote sensing image dataset V in step 12) into a training set, a validation set and a test set according to z1:z2:z3, where z1, z2 and z3 are all positive integers.

[0016] In step 2), the specific construction process of the cloud edge estimation unit is as follows:

[0017] A cloud edge estimation unit is built based on w residual modules to estimate the edge feature information of the cloud layer. The unit input is a remote sensing image with clouds, and the output is a cloud edge map, where w is a positive integer.

[0018] The residual module includes z convolution blocks Block and 1 residual connection. Each convolution block Block consists of 2 convolutions, 2 LReLU activation functions with slope lr, 1 residual learning and 1 feature enhancement unit. The convolution kernel size is s×s and the stride is e, where z, s and e are all positive integers, and lr is a decimal in the range [0,1].

[0019] In step 2), the specific construction process of the cloud thickness estimation unit is as follows:

[0020] A cloud thickness estimation unit is built based on y double residual modules and 1 residual connection to generate cloud thickness information and provide it to the subsequent network to complete cloud removal. The input of the cloud thickness estimation unit is a remote sensing image with clouds, and the output is a cloud thickness map, where y is a positive integer.

[0021] The double residual module consists of 3 convolutions, 3 LReLU activation functions with slope lr, 2 residual learning and 1 feature enhancement unit. The convolution kernel size is d×d and the stride is t, where d and t are both positive integers and lr is a decimal in the range [0,1].

[0022] In step 3), the specific construction process of the feature enhancement unit is as follows:

[0023] A feature enhancement unit is built based on u convolutions, u-1 LReLU activation functions with a slope of lr, and 1 Sigmoid activation function to enhance image features. The convolution kernel size is f×f and the stride is g, where u, f, and g are all positive integers, and lr is a decimal in the range [0,1].

[0024] The process of enhancing the image features is as follows: after the input of the feature enhancement unit is convolved, subjected to an LReLU activation function with a slope of lr and a Sigmoid activation function, a feature map β1 is obtained, and the feature map β1 is multiplied by the corresponding pixel product of the original input β2 to achieve adaptive element-level feature enhancement; wherein lr is a decimal in the range of [0,1].

[0025] In step 3), the specific construction process of the cloud layer identification unit is as follows:

[0026] A cloud layer discrimination unit is built based on j convolutions, j-1 ReLU activation functions, 1 Tanh activation function and 1 mean pooling to determine whether the clouds in the image are removed cleanly. The convolution kernel sizes of j convolutions are {f i ×f i |i∈(1,2,…,j)}, the stride is {d i ×d i |i∈(1,2,…,j)}, where j, f iand d i are all positive integers;

[0027] The process of determining whether the clouds in the image are completely removed is as follows: a cloud discrimination unit is trained using cloudless remote sensing images and cloud remote sensing images in a data set, the input of the cloud discrimination unit is an image, and the output is a probability x, x∈[0,1]; if the clouds in the input image are completely removed, the output x of the cloud discrimination unit is close to 1, and if the clouds are not completely removed, the output x is close to 0; wherein x is a decimal in the range of [0,1].

[0028] In step 4), the specific process of building the cloud layer removal network is as follows:

[0029] The cloud removal network is trained using cloud remote sensing images, cloud-free remote sensing images, cloud thickness images, and cloud edge images in the dataset;

[0030] The output α1 of the cloud edge estimation unit, the output α2 of the cloud thickness estimation unit and the thin cloud remote sensing image α3 are cascaded on the channel, and the feature map α4 is obtained after passing through the feature enhancement unit. The cloud discrimination unit is used for discrimination. If the output is greater than the threshold k, it is considered that the clouds are completely removed, and the output cloud-free remote sensing image is obtained. If the output is less than or equal to the threshold k, it is considered that the clouds are not completely removed. The above process is repeated to remove the clouds, where k is a decimal in the range of [0,1].

[0031] In the step 5), the de-clouding feature loss function is specifically:

[0032]

[0033] Where c, v and b represent the length, width and channel numbers of the feature map, W, H and C represent the length, width and channel sizes of the feature map, σ(·) represents the output feature map of the lth layer of the VGG16 network, H represents the cloud-free remote sensing image, represents a de-clouded remote sensing image, c∈(1,…,W), v∈(1,…,H), b∈(1,…,C), c, v, b, W, H, C and l are all positive integers.

[0034] In the step 5), the de-clouding image reconstruction loss function is specifically:

[0035]

[0036] In step 5), the cloud thickness loss function is specifically:

[0037]

[0038] Where G represents the label cloud thickness map, Represents a map of predicted cloud thickness.

[0039] In step 5), the cloud edge loss function is specifically:

[0040]

[0041] Where J represents the edge graph of the tag cloud layer, Represents the predicted cloud edge map.

[0042] In step 5), the cloud layer discrimination unit loss function is specifically:

[0043]

[0044] Where H represents a cloud-free remote sensing image, K represents a cloud remote sensing image, p represents the number of images in the training set, D(·) represents the output of the cloud layer discrimination unit, and p is a positive integer.

[0045] The beneficial effects of the technical solution provided by the present invention are:

[0046] 1. The traditional hypothesis prior algorithm is based on the hypothesis information, and then the cloud removal image is derived according to the physical model. However, the hypothesis information is adaptive and has a limited scope of adaptation. It will fail in some cases. The method of the present invention realizes cloud discrimination and removal according to cloud characteristics, has a wide range of adaptability, and can process cloud remote sensing images in more scenes;

[0047] 2. The traditional filtering method generally processes the entire image, which may cause distortion in the cloudless part of the image and cannot be processed adaptively. The present invention can adaptively process the cloud part of the image. While removing the cloud layer, the cloudless part does not produce color distortion and other problems. The cloud-removed image is more natural and realistic.

[0048] 3. The present invention has a wide range of applications, strong robustness, a simple and easy method, and a high efficiency in removing clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a cloud discrimination and removal method for remote sensing images based on neural networks;

[0050] Figure 2 This is a schematic diagram of the cloud edge estimation unit structure;

[0051] Figure 3 This is a schematic diagram of the cloud thickness estimation unit structure;

[0052] Figure 4 It is a schematic diagram of the characteristic strengthening unit structure;

[0053] Figure 5 This is a schematic diagram of the cloud layer discrimination unit structure;

[0054] Figure 6Schematic diagram of the network structure for cloud removal. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0056] Example 1

[0057] The present invention proposes a method for identifying and removing cloud layers in remote sensing images based on a neural network. Figure 1 , see the following description for details:

[0058] 101: Select several cloud-free remote sensing images, simulate and generate clouds to obtain a cloud thickness map, add it to the cloud-free image to obtain a cloud image, extract the edge of the cloud thickness map to obtain a cloud edge map, and combine the cloud-free image, cloud image, cloud thickness map and cloud edge map into a remote sensing image dataset, and divide it into a training set, a validation set and a test set;

[0059] 102: Building a cloud edge estimation unit for estimating edge feature information of the cloud layer, and building a cloud thickness estimation unit for generating cloud thickness information, which is provided to a subsequent network to complete cloud removal;

[0060] 103: Building a feature enhancement unit to enhance the cloud features of the image, and building a cloud discrimination unit to determine whether the clouds in the image are completely removed;

[0061] 104: Building a cloud removal network based on the cloud edge estimation unit and the cloud thickness estimation unit in step 102, and the feature enhancement unit and the cloud discrimination unit in step 103, so as to generate a de-clouded image from a single cloud image;

[0062] 105: Train the cloud removal network based on the de-clouding feature loss function, de-clouding reconstruction loss function, cloud thickness loss function and cloud edge loss function, and train the cloud discrimination unit based on the cloud discrimination unit loss function;

[0063] 106: Import the trained model parameters and obtain the cloud-free remote sensing image from a single remote sensing image with clouds.

[0064] The specific steps in step 101 are:

[0065] 1) Select m cloud-free remote sensing images, generate clouds by simulating m cloud thickness maps and add them to the cloud-free remote sensing images to obtain m cloud remote sensing images, use the Sobel operator to extract the edge of the cloud thickness map to obtain m cloud edge maps, segment all images according to the size of N×N, and obtain n cloud-free remote sensing images H, n cloud remote sensing images K, n cloud thickness images G and n cloud edge images J to form a data set U, where m, n and N are all positive integers;

[0066] 2) Expanding the data set, specifically: randomly selecting a% of images from the data set U in step 1), randomly rotating the images by 90°, 180°, 270°, and mirror flipping, obtaining expanded images and adding them to the data set U to form a remote sensing image data set V, where a is a positive integer

[0067] 3) Divide the remote sensing image dataset V in step 2) into a training set, a validation set and a test set according to z1:z2:z3, where z1, z2 and z3 are all positive integers.

[0068] The specific steps in step 102 are:

[0069] 1) If Figure 2 As shown in the figure, the residual module includes z convolution blocks and 1 residual connection. Each convolution block consists of 2 convolutions, 2 LReLU activation functions with slope lr, 1 residual learning and 1 feature enhancement unit. The convolution kernel size is s×s and the stride is e, where z, s and e are all positive integers and lr is a decimal in the range [0,1].

[0070] 2) If Figure 2 As shown in the figure, a cloud edge estimation unit (CEEU) is built based on w residual modules to estimate the edge feature information of the cloud layer. The unit input is a remote sensing image with clouds, and the output is a cloud edge map, where w is a positive integer.

[0071] 3) If Figure 3 As shown in the figure, the double residual unit consists of 3 convolutions, 3 LReLU activation functions with slope lr, 2 residual learning and 1 feature enhancement unit. The convolution kernel size is d×d and the stride is t, where d and t are positive integers and lr is a decimal in the range [0,1].

[0072] 4) If Figure 3 As shown in the figure, a cloud thickness estimation unit (CloudThickness Estimation Unit, CTEU) is built based on y double residual modules and 1 residual connection to generate cloud thickness information, which is provided to the subsequent network to complete cloud removal. The input of the CTEU unit is a remote sensing image with clouds, and the output is a cloud thickness map, where y is a positive integer.

[0073] The specific steps in step 103 are:

[0074] 1) If Figure 4As shown in the figure, a feature enhancement unit (FEU) is built based on u convolutions, u-1 LReLU activation functions (with a slope of lr) and 1 Sigmoid activation function to enhance image features. The convolution kernel size is f×f and the stride is g, where u, f and g are all positive integers and lr is a decimal in the range [0,1].

[0075] 2) After the input of the feature enhancement unit FEU is convolved, the LReLU activation function with a slope of lr and the Sigmoid activation function, the feature map β1 is obtained. The feature map β1 is multiplied by the corresponding pixel of the original input β2 to achieve adaptive element-level feature enhancement, where lr is a decimal in the range [0,1];

[0076] 3) If Figure 5 As shown in the figure, a cloud discrimination unit (CDU) is built based on j convolutions, j-1 ReLU activation functions, 1 Tanh activation function and 1 mean pooling to determine whether the clouds in the image are removed cleanly. The convolution kernel sizes of the j convolutions are {f i ×f i |i∈(1,2,…,j)}, the stride is {d i ×d i |i∈(1,2,…,j)}, where j, f i and d i are all positive integers;

[0077] 4) The cloud discrimination unit CDU is trained using the cloudless remote sensing images and clouded remote sensing images in the dataset. The input of CDU is an image, and the output is a probability x, x∈[0,1]. If the input image is a cloudless remote sensing image, the output x is close to 1, and if the input image is a clouded remote sensing image, the output x is close to 0. Therefore, if the cloud layer of the input image is completely removed, the output x of the cloud discrimination unit is close to 1, and if the cloud layer is not completely removed, the output x is close to 0. Among them, x is a decimal in the range of [0,1].

[0078] The specific steps in step 104 are:

[0079] 1) If Figure 6 As shown, a cloud removal network is built based on the cloud edge estimation unit CEEU and the cloud thickness estimation unit CTEU in step 102, the feature enhancement unit FEU and the cloud discrimination unit CDU in step 103, which is used to generate a de-clouded image from a single cloud image, and the cloud removal network is trained using a remote sensing image with clouds, a remote sensing image without clouds, a cloud thickness image and a cloud edge image in the data set;

[0080] 2) The output α1 of the cloud edge estimation unit CEEU, the output α2 of the cloud thickness estimation unit CTEU and the thin cloud remote sensing image α3 are cascaded on the channel, and the feature map α4 is obtained after passing through the feature enhancement unit FEU. The cloud discrimination unit CDU is used for discrimination. If the output is greater than the threshold k, it is considered that the clouds are completely removed, and the output cloud-free remote sensing image is obtained. If the output is less than or equal to the threshold k, it is considered that the clouds are not completely removed. The above process is repeated to remove the clouds, where k is a decimal in the range of [0,1].

[0081] The specific steps in step 105 are:

[0082] 1) Training a cloud removal network based on a de-clouding feature loss function, a de-clouding reconstruction loss function, a cloud thickness loss function, and a cloud edge loss function. The specific functions are as described in 2), 3), 4), and 5);

[0083] 2) The specific loss function of the cloud map feature is:

[0084]

[0085] Where c, v and b represent the length, width and channel numbers of the feature map, W, H and C represent the length, width and channel sizes of the feature map, σ(·) represents the output feature map of the lth layer of the VGG16 network, H represents the cloud-free remote sensing image, represents a de-clouded remote sensing image, c∈(1,…,W), v∈(1,…,H), b∈(1,…,C), c, v, b, W, H, C and l are all positive integers;

[0086] 3) The specific loss function of cloud image reconstruction is:

[0087]

[0088] 4) The cloud thickness loss function is specifically:

[0089]

[0090] Where G represents the label cloud thickness map, Represents the predicted cloud thickness map;

[0091] 5) The cloud edge loss function is specifically:

[0092]

[0093] Where J represents the edge graph of the tag cloud layer, Represents the predicted cloud edge map;

[0094] 6) The cloud discrimination unit is trained based on the cloud discrimination unit loss function. The cloud discrimination unit loss function is specifically:

[0095]

[0096] Where H represents a cloud-free remote sensing image, K represents a cloud remote sensing image, p represents the number of images in the training set, D(·) represents the output of the cloud layer discrimination unit, and p is a positive integer.

[0097] The specific steps in step 106 are: importing the trained model parameters, and obtaining a de-clouded remote sensing image from a single remote sensing image with clouds.

[0098] Example 2

[0099] The present invention proposes a method for identifying and removing cloud layers in remote sensing images based on a neural network. Figure 1 , see the following description for details:

[0100] 201: Select several cloud-free remote sensing images, simulate and generate clouds to obtain a cloud thickness map, add it to the cloud-free image to obtain a cloud image, extract the edge of the cloud thickness map to obtain a cloud edge map, and combine the cloud-free image, cloud image, cloud thickness map and cloud edge map into a remote sensing image dataset, and divide it into a training set, a validation set and a test set;

[0101] 202: Building a cloud edge estimation unit for estimating edge feature information of the cloud layer, and building a cloud thickness estimation unit for generating cloud thickness information, which is provided to a subsequent network to complete cloud removal;

[0102] 203: Building a feature enhancement unit to enhance the cloud features of the image, and building a cloud discrimination unit to determine whether the clouds in the image are completely removed;

[0103] 204: Building a cloud removal network based on the cloud edge estimation unit and the cloud thickness estimation unit in step 102, and the feature enhancement unit and the cloud discrimination unit in step 103, for generating a de-clouded image from a single cloud image;

[0104] 205: Train the cloud removal network based on the de-clouding feature loss function, de-clouding reconstruction loss function, cloud thickness loss function and cloud edge loss function, and train the cloud discrimination unit based on the cloud discrimination unit loss function;

[0105] 206: Import the trained model parameters and obtain a de-clouded remote sensing image from a single remote sensing image with clouds.

[0106] The specific steps in step 201 are:

[0107] 1) Select m cloud-free remote sensing images, generate clouds by simulating m cloud thickness maps and add them to the cloud-free remote sensing images to obtain m cloud remote sensing images, use the Sobel operator to extract the edge of the cloud thickness map to obtain m cloud edge maps, segment all images according to the size of N×N, and obtain n cloud-free remote sensing images H, n cloud remote sensing images K, n cloud thickness images G and n cloud edge images J to form a data set U, where m=500, n=5000, and N=256;

[0108] 2) Expanding the data set, specifically: randomly selecting a% of images from the data set U in step 1), randomly performing one of the following operations on the images: 90° rotation, 180° rotation, 270° rotation, and mirror flipping, obtaining expanded images and adding them to the data set U to form a remote sensing image data set V, where a=50;

[0109] 3) Divide the remote sensing image dataset V in step 2) into a training set, a validation set, and a test set according to z1:z2:z3, where z1=7, z2=1, and z3=2.

[0110] The specific steps in step 202 are:

[0111] 1) If Figure 2 As shown in the figure, the residual module includes z convolution blocks and 1 residual connection. Each convolution block consists of 2 convolutions, 2 LReLU activation functions with slope lr, 1 residual learning and 1 feature enhancement unit. The convolution kernel size is s×s and the stride is e, where z=5, lr=0.2, s=4, and e=1.

[0112] 2) If Figure 2 As shown in the figure, a cloud edge estimation unit (CEEU) is built based on w residual modules to estimate the edge feature information of the cloud layer. The unit input is a remote sensing image with clouds, and the output is a cloud edge map, where w = 5;

[0113] 3) If Figure 3 As shown in the figure, the double residual unit consists of 3 convolutions, 3 LReLU activation functions with slope lr, 2 residual learning and 1 feature enhancement unit. The convolution kernel size is d×d and the stride is t, where lr=0.2, d=4, and t=1.

[0114] 4) If Figure 3As shown in the figure, a cloud thickness estimation unit (CloudThickness Estimation Unit, CTEU) is built based on y double residual modules and 1 residual connection to generate cloud thickness information, which is provided to the subsequent network to complete cloud removal. The input of the CTEU unit is a remote sensing image with clouds, and the output is a cloud thickness map, where y = 3.

[0115] The specific steps in step 203 are:

[0116] 1) If Figure 4 As shown in the figure, a feature enhancement unit (FEU) is built based on u convolutions, u-1 LReLU activation functions (slope is lr) and 1 Sigmoid activation function to enhance image features. The convolution kernel size is f×f and the stride is g, where u=7, lr=0.2, f=4, and g=1.

[0117] 2) After the input of the feature enhancement unit FEU is convolved, the LReLU activation function with a slope of lr and the Sigmoid activation function, the feature map β1 is obtained. The feature map β1 is multiplied by the corresponding pixel of the original input β2 to achieve adaptive element-level feature enhancement, where lr = 0.2;

[0118] 3) If Figure 5 As shown in the figure, a cloud discrimination unit (CDU) is built based on j convolutions, j-1 ReLU activation functions, 1 Tanh activation function and 1 mean pooling to determine whether the clouds in the image are removed cleanly. The convolution kernel sizes of the j convolutions are {3×3, 5×5, 7×7, 9×9, 11×11}, and the strides are {2, 2, 2, 1, 1}, where j = 5;

[0119] 4) The cloud discrimination unit CDU is trained using the cloudless remote sensing images and clouded remote sensing images in the dataset. The input of CDU is an image, and the output is a probability x, x∈[0,1]. If the input image is a cloudless remote sensing image, the output x is close to 1, and if the input image is a clouded remote sensing image, the output x is close to 0. Therefore, if the cloud layer of the input image is completely removed, the output x of the cloud discrimination unit is close to 1, and if the cloud layer is not completely removed, the output x is close to 0. Among them, x is a decimal in the range of [0,1].

[0120] The specific steps in step 204 are:

[0121] 1) If Figure 6As shown, a cloud removal network is built based on the cloud edge estimation unit CEEU and the cloud thickness estimation unit CTEU in step 102, the feature enhancement unit FEU and the cloud discrimination unit CDU in step 103, which is used to generate a de-clouded image from a single cloud image, and the cloud removal network is trained using a remote sensing image with clouds, a remote sensing image without clouds, a cloud thickness image and a cloud edge image in the data set;

[0122] 2) The output α1 of the cloud edge estimation unit CEEU, the output α2 of the cloud thickness estimation unit CTEU and the thin cloud remote sensing image α3 are cascaded on the channel, and the feature map α4 is obtained after passing through the feature enhancement unit FEU. The cloud discrimination unit CDU is used for discrimination. If the output is greater than the threshold k, it is considered that the clouds are completely removed, and the output cloud-free remote sensing image is obtained. If the output is less than or equal to the threshold k, it is considered that the clouds are not completely removed. The above process is repeated to remove the clouds, where k = 0.6.

[0123] The specific steps in step 205 are:

[0124] 1) Training a cloud removal network based on a de-clouding feature loss function, a de-clouding reconstruction loss function, a cloud thickness loss function, and a cloud edge loss function. The specific functions are as described in 2), 3), 4), and 5);

[0125] 2) The specific loss function of the cloud map feature is:

[0126]

[0127] Where c, v and b represent the length, width and channel numbers of the feature map, W, H and C represent the length, width and channel sizes of the feature map, σ(·) represents the output feature map of the lth layer of the VGG16 network, H represents the cloud-free remote sensing image, represents a de-clouded remote sensing image, c∈(1,…,W), v∈(1,…,H), b∈(1,…,C), c, v, b, W, H and C are all positive integers, and l=20.

[0128] 3) The specific loss function of cloud image reconstruction is:

[0129]

[0130] 4) The cloud thickness loss function is specifically:

[0131]

[0132] Where G represents the label cloud thickness map, Represents the predicted cloud thickness map;

[0133] 5) The cloud edge loss function is specifically:

[0134]

[0135] Where J represents the edge graph of the tag cloud layer, Represents the predicted cloud edge map;

[0136] 6) The cloud discrimination unit is trained based on the cloud discrimination unit loss function. The cloud discrimination unit loss function is specifically:

[0137]

[0138] Where H represents a cloud-free remote sensing image, K represents a cloud remote sensing image, p represents the number of images in the training set, D(·) represents the output of the cloud layer discrimination unit, and p is a positive integer.

[0139] The specific steps in step 206 are: importing the trained model parameters, and obtaining a de-clouded remote sensing image from a single remote sensing image with clouds.

[0140] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for cloud recognition and removal in remote sensing images based on neural network, characterized in that Here are the steps: 1) Select several cloud-free remote sensing images, simulate and generate clouds to obtain cloud thickness maps, and add them to the cloud-free images to obtain cloud images. Extract the edges of the cloud thickness maps to obtain cloud edge maps. The cloud-free images, cloud images, cloud thickness maps, and cloud edge maps form a remote sensing image dataset and are divided into training sets, validation sets, and test sets. 2) Build a cloud edge estimation unit to estimate the edge feature information of the cloud layer, and build a cloud thickness estimation unit to generate cloud thickness information and provide it to the subsequent network to complete cloud removal; 3) Build a feature enhancement unit to enhance the cloud features of the image, and build a cloud discrimination unit to determine whether the clouds in the image have been completely removed; 4) Based on the cloud edge estimation unit, cloud thickness estimation unit, feature enhancement unit and cloud discrimination unit, a cloud removal network is built to generate a de-clouded image from a single cloud image; 5) The cloud removal network is trained based on the de-clouding feature loss function, de-clouding reconstruction loss function, cloud thickness loss function and cloud edge loss function, and the cloud discrimination unit is trained based on the cloud discrimination unit loss function; 6) Import the trained model parameters and obtain a cloud-free remote sensing image from a single remote sensing image with clouds; In step 1), the specific steps of constructing the remote sensing image data set are: 11) Select m cloud-free remote sensing images, generate clouds by simulating m cloud thickness maps and add them to the cloud-free remote sensing images to obtain m cloud remote sensing images, extract the edges of the cloud thickness maps using the Sobel operator to obtain m cloud edge maps, segment all images according to the size of N×N, and obtain n cloud-free remote sensing images H, n cloud remote sensing images K, n cloud thickness images G and n cloud edge images J to form a data set U, where m, n and N are all positive integers; 12) Expanding the data set, specifically: randomly selecting a% of images from the data set U in step 11), and randomly rotating the images, including 90° rotation, 180° rotation, 270° rotation, and mirror flipping; after the rotation operation, the expanded images are obtained and added to the data set U to form a remote sensing image data set V, where a is a positive integer; 13) Dividing the remote sensing image dataset V in step 12) into a training set, a validation set and a test set according to z1:z2:z3, where z1, z2 and z3 are all positive integers; In step 2), the specific construction process of the cloud edge estimation unit is as follows: A cloud edge estimation unit is built based on w residual modules to estimate the edge feature information of the cloud layer. The unit input is a remote sensing image with clouds, and the output is a cloud edge map, where w is a positive integer. The residual module includes z convolution blocks and 1 residual connection. Each convolution block consists of 2 convolutions, 2 LReLU activation functions with slope lr, 1 residual learning and 1 feature enhancement unit. The convolution kernel size is s×s and the stride is e, where z, s and e are all positive integers and lr is a decimal in the range [0,1]. In step 2), the specific construction process of the cloud thickness estimation unit is as follows: A cloud thickness estimation unit is built based on y double residual modules and 1 residual connection to generate cloud thickness information and provide it to the subsequent network to complete cloud removal. The cloud thickness estimation unit inputs a remote sensing image with clouds and outputs a cloud thickness map, where y is a positive integer. The double residual module consists of 3 convolutions, 3 LReLU activation functions with slope lr, 2 residual learning and 1 feature enhancement unit. The convolution kernel size is d×d and the stride is t, where d and t are positive integers and lr is a decimal in the range [0,1]. In step 3), the specific construction process of the feature enhancement unit is as follows: A feature enhancement unit is built based on u convolutions, u-1 LReLU activation functions with a slope of lr, and 1 Sigmoid activation function to enhance image features. The convolution kernel size is f×f and the stride is g, where u, f, and g are all positive integers, and lr is a decimal in the range [0,1]. The process of enhancing the image features is as follows: after the input of the feature enhancement unit is convolved, the LReLU activation function with a slope of lr and the Sigmoid activation function, a feature map β1 is obtained, and the feature map β1 is multiplied by the corresponding pixel product of the original input β2 to achieve adaptive element-level feature enhancement; wherein lr is a decimal in the range of [0,1]; In step 3), the specific construction process of the cloud layer identification unit is as follows: A cloud layer discrimination unit is built based on j convolutions, j-1 ReLU activation functions, 1 Tanh activation function and 1 mean pooling to determine whether the clouds in the image are removed cleanly. The convolution kernel sizes of j convolutions are {f i ×f i |i∈(1,2,…,j)}, the stride is {d i ×d i |i∈(1,2,…,j)}, where j, f i and d i are all positive integers; The process of judging whether the cloud in the image is completely removed is as follows: a cloud discrimination unit is trained using a cloudless remote sensing image and a cloud remote sensing image in a data set, wherein the input of the cloud discrimination unit is an image, and the output is a probability x, x∈[0,1]; if the cloud in the input image is completely removed, the output x of the cloud discrimination unit is close to 1, and if the cloud is not completely removed, the output x is close to 0; wherein x is a decimal in the range of [0,1]; In step 4), the specific process of building the cloud layer removal network is as follows: The cloud removal network is trained using cloud remote sensing images, cloud-free remote sensing images, cloud thickness images, and cloud edge images in the dataset; The output α1 of the cloud edge estimation unit, the output α2 of the cloud thickness estimation unit and the thin cloud remote sensing image α3 are cascaded on the channel, and the feature map α4 is obtained after passing through the feature enhancement unit. The cloud discrimination unit is used for discrimination. If the output is greater than the threshold k, it is considered that the clouds are completely removed, and the output cloud-free remote sensing image is obtained. If the output is less than or equal to the threshold k, it is considered that the clouds are not completely removed. The above process is repeated to remove the clouds, where k is a decimal in the range of [0,1].

2. The method for cloud layer identification and removal based on neural network in remote sensing images according to claim 1 is characterized in that: In the step 5), the de-clouding feature loss function is specifically: Where c, v and b represent the length, width and channel numbers of the feature map, W, H and C represent the length, width and channel sizes of the feature map, σ(·) represents the output feature map of the lth layer of the VGG16 network, H represents the cloud-free remote sensing image, represents a de-clouded remote sensing image, c∈(1,…,W), v∈(1,…,H), b∈(1,…,C), c, v, b, W, H, C and l are all positive integers.

3. The method for cloud layer identification and removal based on neural network in remote sensing images according to claim 2 is characterized in that: In the step 5), the de-clouding image reconstruction loss function is specifically:

4. The method for cloud layer identification and removal based on neural network in remote sensing images according to claim 2 is characterized in that: In step 5), the cloud thickness loss function is specifically: Where G represents the label cloud thickness map, Represents a map of predicted cloud thickness.

5. The method for cloud layer identification and removal based on neural network in remote sensing images according to claim 2 is characterized in that: In step 5), the cloud edge loss function is specifically: Where J represents the edge graph of the tag cloud layer, Represents the predicted cloud edge map.

6. The method for cloud layer identification and removal based on neural network in remote sensing images according to claim 2 is characterized in that: In step 5), the cloud layer discrimination unit loss function is specifically: Where H represents a cloud-free remote sensing image, K represents a cloud remote sensing image, p represents the number of images in the training set, D(·) represents the output of the cloud layer discrimination unit, and p is a positive integer.

Citation Information

Patent Citations

  • Method for removing thin cloud of remote sensing image

    CN102509262A

  • Remote sensing image cloud detection method based on convolution nerve network

    CN105844228A

  • Cloud detection method based on landsat 8 snow-containing image

    US20210286970A1