Remote sensing image cloud removal method for multi-temporal data

Abstract

The invention relates to a remote sensing image cloud removal method for multi-temporal data. The method comprises the following steps: constructing a remote sensing cloud detection data set; a U-Net neural network is constructed; training the U-Net neural network by using the data; inputting a cloud remote sensing image into the trained U-Net neural network to obtain a cloud picture, and operating by adopting a morphological method to obtain a binary cloud mask; identifying a cloud area and a cloudless area based on the binary cloud mask, and setting balance values respectively; and processing the morphological cloud mask and a given cloud remote sensing image by adopting RPCA, and completing remote sensing image cloud removal after solving. According to the method, the cloud removal effect can be improved, the cloud amount residue is reduced, the cloud and mist shielding is reduced, and the usability of remote sensing data can be greatly improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image cloud removal method for multi-temporal data. Background Art

[0002] The rapid development of satellite technology has enabled remote sensing technology to play a very important role in meteorological research and applications. According to data from the International Satellite Cloud Climatology Project (ISCCP), more than 66% of the world's surface is covered by clouds. Therefore, in the processing of remote sensing images, clouds are an interference factor that cannot be ignored. If only images with relatively low cloud content are used, it will lead to a waste of image data. Moreover, when solar radiation passes through the atmosphere, it will be affected by scattering, reflection and absorption by the atmosphere or clouds. The electromagnetic waves received by remote sensing satellite sensors will have a certain degree of loss. In this case, the information blocked by clouds is not conducive to subsequent image processing and interpretation, and cloud blocking greatly affects the availability of images for further research.

[0003] The current remote sensing image cloud detection methods mainly include spectral threshold-based methods, classical machine learning-based methods, and deep learning-based methods. The spectral threshold-based method realizes automatic cloud detection by setting thresholds for different spectra of the image. This type of method does not require pixel-level labeling and complex model training, and can be deployed and run more efficiently. However, the spectral threshold-based method usually requires the remote sensing image to have rich spectral information, and the generalization ability is poor. The algorithm is often not robust enough for different scenes, especially complex scenes.

[0004] In order to solve the above problems, the classical machine learning method is introduced into the cloud detection task. Considering the brightness features, texture features and local statistical features, the cloud detection is realized by using technical means such as support vector machine and random forest, which improves the mining and utilization of spatial pattern information of remote sensing images and reduces the high dependence of traditional cloud detection methods on the spectral range of satellite image data. However, since the classical machine learning model usually requires manual design of image features, it is difficult for the classical machine learning-based method to extract higher-level semantic information of remote sensing images more efficiently. With the rise of deep learning technology, cloud detection methods based on deep learning are also developing rapidly, and gradually developing from image block classification to image pixel classification.

[0005] Cloud removal refers to the technology of repairing the cloud-covered area of ​​the target image by using the remote sensing image itself or other remote sensing images on the basis of cloud detection, restoring the data to the greatest extent, and obtaining a clear sky image with strong spectral consistency and no cloud pollution. The multi-temporal remote sensing image cloud removal method is studied. The multi-temporal method needs to use multi-temporal images from the same sensor as input. In the time image sequence, the information of the cloud-covered area is replaced or reconstructed by the information of the cloud-free area image at different times of the same scene, so as to achieve the purpose of cloud repair. For multi-temporal images, remote sensing image cloud removal methods mainly include spatiotemporal interpolation, spectral fitting, spatiotemporal filtering, etc. The more popular deep learning method is mainly suitable for single-temporal remote sensing image cloud removal, or can only remove clouds from multiple frames of cloud images with a specific sequence of frames.

[0006] Therefore, designing a remote sensing image cloud removal method for multi-temporal data remains an urgent problem to be solved. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a remote sensing image cloud removal method for multi-temporal data, which can improve the cloud removal effect, reduce residual cloud, reduce cloud and fog obstruction, and greatly improve the availability of remote sensing data.

[0008] To achieve the above-mentioned object of the invention, the present invention provides a remote sensing image cloud removal method for multi-temporal data, comprising the following steps:

[0009] Step S1, constructing a remote sensing cloud detection data set;

[0010] Step S2, constructing a U-Net neural network;

[0011] Step S3, using the data to train the U-Net neural network;

[0012] Step S4, inputting the cloud remote sensing image into the trained U-Net neural network to obtain a cloud image, and using a morphological method to obtain a binary cloud mask;

[0013] Step S5: Based on the binary cloud mask, identify the cloud area and the cloud-free area, and set the balance value respectively;

[0014] Step S6: Using RPCA to process the morphological cloud mask and the given remote sensing image with clouds, and completing the remote sensing image cloud removal after solving.

[0015] According to a technical solution of the present invention, the step S1 specifically includes:

[0016] A cloud satellite image dataset is synthesized using a real cloud image dataset and a real cloud-free ground image dataset with the help of an atmospheric scattering model of satellite images;

[0017] Let S, G, C ∈ R n1×n2×1 , S, G, and C are the synthesized cloud satellite image, cloud-free ground image, and cloud image, respectively. The cloud image C is superimposed on the cloud-free ground image G to synthesize the cloud satellite image S, which is expressed as:

[0018]

[0019] in, Represents the Hadamard product.

[0020] According to a technical solution of the present invention, in step S2, the U-Net neural network has 11 layers, the first 5 layers form a contraction path, which is the encoding stage, and the 6th to 10th layers form an expansion path, which is the decoding stage;

[0021] In the contraction path, the image size is reduced through convolution and downsampling to extract shallow features;

[0022] In the expansion path, deep features are obtained through transposed convolution and upsampling.

[0023] According to a technical solution of the present invention, in step S3, a single-frame satellite image with clouds S∈R is segmented into a frame of cloud image C∈R and a frame of background image G∈R. During the training process, the input is and In each iteration, a batch of images are sampled, and the ground image and cloud image are synthesized into a satellite image with clouds using formula (1). The network weights are updated using the gradient descent method to obtain a trained U-Net neural network.

[0024] According to a technical solution of the present invention, in step S5, a relatively low balance value λ is set for the cloud region. c , ensuring that all clouds and shadows are completely decomposed into an outlier matrix, then λ c =1 / max((m,n)) 1 / 2 -> 0.1 / max((m,n)) 1 / 2 ;

[0025] Set a higher balance value λ for cloud-free areas nc , to ensure the background, we have:

[0026] λ nc =1 / max((m,n)) 1 / 2 -> 1;

[0027] Where D∈R m×n Represents a column-stacked image sequence, where m = rows × cols is the number of pixels in a single image and n is the total number of image sequences.

[0028] According to a technical solution of the present invention, in step S6, RPCA is used to process the morphological cloud mask and the given remote sensing image with clouds, which is expressed as:

[0029]

[0030] Wherein, Ω represents the binary cloud mask obtained in step S4, Ω - Indicates cloud-free areas.

[0031] According to a technical solution of the present invention, in step S6, an inaccurate augmented Lagrangian function is used to solve formula (2), and the augmented Lagrangian function is:

[0032]

[0033] Among them, ||·|| F is the Frobenius norm, Y is the Lagrange multiplier, and μ is a positive scalar.

[0034] According to one aspect of the present invention, there is provided an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes a remote sensing image cloud removal method for multi-temporal data as described in any one of the above technical solutions.

[0035] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, a remote sensing image cloud removal method for multi-temporal data as described in any one of the above technical solutions is implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention proposes a remote sensing image cloud removal method for multi-temporal data, which combines deep learning methods with traditional methods. The deep learning U-Net network is used to detect remote sensing images with clouds to form a binary cloud mask, and then morphological operations are performed as needed to ensure that the cloud area is fully detected and a large amount of noise is filtered out. The cloud area and cloud-free area are identified through the binary cloud mask, and cloud removal is achieved through the improved RPCA method, thereby greatly improving the cloud removal effect, reducing residual cloud amount, and reducing cloud and fog occlusion.

[0038] Furthermore, the use of a remote sensing image cloud removal method for multi-temporal data of the present invention can greatly improve the availability of remote sensing data, and has a positive impact on the acquisition of environmental or target information, ecological environment monitoring, weather forecasting, temperature inversion, prevention and rescue of meteorological disasters, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of a remote sensing image cloud removal method for multi-temporal data in an embodiment of the present invention is schematically shown;

[0041] Figure 2 The U-net neural network architecture constructed in step S2 in the embodiment of the present invention is schematically shown;

[0042] Figure 3 The training process of the constructed U-net neural network using the constructed data set in the embodiment of the present invention is schematically shown;

[0043] Figure 4 The results obtained by using the remote sensing image cloud removal method for multi-temporal data in an embodiment of the present invention are schematically shown. DETAILED DESCRIPTION

[0044] The description of the embodiments of this specification should be combined with the corresponding drawings, which should be considered as part of the complete specification. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated for simplification or convenience. Furthermore, the parts of each structure in the drawings will be described separately. It is worth noting that the elements not shown in the drawings or not described in words are in a form known to ordinary technicians in the relevant technical field.

[0045] The description of the embodiments herein and any reference to directions and orientations are only for the convenience of description and are not to be construed as any limitation on the scope of protection of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0046] like Figure 1 As shown, a remote sensing image cloud removal method for multi-temporal data of the present invention comprises the following steps:

[0047] Step S1: constructing a remote sensing cloud detection dataset, specifically including:

[0048] A cloud satellite image dataset is synthesized using a real cloud image dataset and a real cloud-free ground image dataset with the help of an atmospheric scattering model of satellite images;

[0049] Let S, G, C ∈ R n1×n2×1 , S, G, and C are the synthesized cloud satellite image, cloud-free ground image, and cloud image, respectively. The cloud image C is superimposed on the cloud-free ground image G to synthesize the cloud satellite image S, which is expressed as:

[0050]

[0051] in, represents the Hadamard product.

[0052] In this embodiment, the INRIA cloud-free satellite image dataset and the NOAA pure cloud dataset are used to select 288 INRIA cloud-free ground images, and for each test image, 7 cloud images are randomly selected and synthesized with them to obtain 288 groups of 7 frames of cloud satellite images to form a dataset.

[0053] Step S2, constructing a U-Net neural network;

[0054] like Figure 2 As shown in the figure, the U-Net neural network has 11 layers. The first 5 layers form a contraction path, which is also the encoding stage. The 6th to 10th layers form an expansion path, which is also the decoding stage. In the contraction path, convolution and downsampling are used to reduce the image size and extract shallow features. In the expansion path, transposed convolution and upsampling are used to obtain deep features. Figure 2 The connection operation shown combines the feature maps obtained in the encoding stage with the feature maps obtained in the decoding stage. The number of channels in each layer of the expansion path of U-Net is doubled to combine deep and shallow features. The 11th layer is a standard output layer with a convolution operation and a rectified linear unit (ReLU) activation function. In this embodiment, the convolution block consists of multi-channel 2D convolution and ReLU, the convolution kernel size is 3×3, and the step size is 2. The transposed convolution block consists of multi-channel 2D transposed convolution, batch normalization (BN) and ReLU, the convolution kernel size is 3×3, and the step size is 2. BN operation is performed between the transposed convolution and the ReLU activation function.

[0055] Step S3, using the data to train the U-Net neural network;

[0056] like Figure 3As shown in the figure, the remote sensing cloud detection dataset constructed in step S1 is used to train the U-Net neural network constructed in step S2. The single-frame satellite image S∈R with clouds is segmented into a frame of cloud image C∈R and a frame of background image G∈R. During the training process, the input is and In each iteration, a batch of images are sampled, and the ground image and cloud image are synthesized into a satellite image with clouds using formula (1). Then, the network weights are updated by the gradient descent method, and finally a trained network is obtained.

[0057] Step S4, inputting the cloud remote sensing image into the trained U-Net neural network to obtain a cloud image, and using a morphological method to obtain a binary cloud mask;

[0058] The remote sensing image with clouds is passed through a trained U-Net network to obtain a foreground, i.e., a cloud image, and a binary cloud mask is obtained by using a morphological method. In this embodiment, a morphological erosion operator and three morphological dilation operators of size 3 are used to implement the morphological operation.

[0059] Step S5: Based on the binary cloud mask, identify the cloud area and the cloud-free area, and set the balance value respectively;

[0060] Based on the obtained binary cloud mask, the cloud area and the cloud-free area are identified, and different balance values ​​are set for the cloud area and the cloud-free area, and a relatively low balance value λ is set for the cloud area. c , ensuring that all clouds and shadows are completely decomposed into an outlier matrix, setting a higher balance value λ for cloud-free areas nc , to ensure background

[0061] λ c =1 / max((m,n)) 1 / 2->0.1 / max((m,n)) 1 / 2

[0062] λ nc =1 / max((m,n)) 1 / 2 -> 1

[0063] Where D∈R m×n Represents a column-stacked image sequence, where m = rows × cols is the number of pixels in a single image and n is the total number of image sequences.

[0064] In this embodiment, the cloud area λ c =0.1 / max((m,n)) 1 / 2 , cloud-free area λ nc =1.

[0065] Step S6: Using RPCA to process the morphological cloud mask and the given remote sensing image with clouds, and completing the remote sensing image cloud removal after solving.

[0066] RPCA is used to process the morphological cloud mask and the given remote sensing image with clouds, which is expressed as:

[0067]

[0068] Wherein, Ω represents the binary cloud mask obtained in step S4, Ω - Indicates cloud-free areas.

[0069] The inaccurate augmented Lagrangian function is used to solve formula (2), and the augmented Lagrangian function is:

[0070]

[0071] Among them, ||·|| F is the Frobenius norm, Y is the Lagrange multiplier, and μ is a positive scalar.

[0072] The cloud removal method for remote sensing images of multi-temporal data of the present invention is used to obtain the result after cloud removal of remote sensing images as shown in the following figure: Figure 4 shown.

[0073] According to one aspect of the present invention, there is provided an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes a remote sensing image cloud removal method for multi-temporal data as described in any one of the above technical solutions.

[0074] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, a remote sensing image cloud removal method for multi-temporal data as described in any one of the above technical solutions is implemented.

[0075] Computer-readable storage media may include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments may be downloaded via computer networks such as the Internet, intranets, etc.

[0076] The invention discloses a remote sensing image cloud removal method for multi-temporal data, comprising the following steps: step S1, constructing a remote sensing cloud detection data set; step S2, constructing a U-Net neural network; step S3, training the U-Net neural network using data; step S4, inputting a remote sensing image with clouds into the trained U-Net neural network to obtain a cloud map, and using a morphological method to operate to obtain a binary cloud mask; step S5, based on the binary cloud mask, identifying a cloud area and a cloudless area, and setting a balance value respectively; step S6, using RPCA to process a morphological cloud mask and a given remote sensing image with clouds, and completing the remote sensing image cloud removal after solving. The present invention combines the deep learning method with the traditional method, uses the deep learning U-Net network to detect remote sensing images with clouds, forms a binary cloud mask, and then performs morphological operations as needed to ensure that the cloud area is fully detected and a large amount of noise is filtered out. The cloud area and the cloudless area are identified through the binary cloud mask, and then the cloud removal is achieved through the improved RPCA method, thereby greatly improving the cloud removal effect, reducing the residual cloud amount, reducing cloud and fog occlusion, and improving the availability of remote sensing data.

[0077] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0078] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] It should also be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0081] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A remote sensing image cloud removal method for multi-temporal data, characterized in that: The following steps are involved: Step S1, constructing a remote sensing cloud detection data set; Step S2, constructing a U-Net neural network; Step S3, using the data to train the U-Net neural network; Step S4, inputting the cloud remote sensing image into the trained U-Net neural network to obtain a cloud image, and using a morphological method to obtain a binary cloud mask; Step S5: Based on the binary cloud mask, identify the cloud area and the cloud-free area, and set the balance value respectively; Step S6: Using RPCA to process the morphological cloud mask and the given remote sensing image with clouds, and completing the remote sensing image cloud removal after solving.

2. The remote sensing image cloud removal method for multi-temporal data according to claim 1, characterized in that: The step S1 specifically includes: A cloud satellite image dataset is synthesized using a real cloud image dataset and a real cloud-free ground image dataset with the help of an atmospheric scattering model of satellite images; Let S, G, C ∈ R n1×n2×1 , S, G, and C are the synthesized satellite image with clouds, the cloud-free ground image, and the cloud image, respectively. The cloud image C is superimposed on the cloud-free ground image G to synthesize the cloud satellite image S, which is expressed as: S=(1-C)°G+C (1) in, Represents the Hadamard product.

3. The remote sensing image cloud removal method for multi-temporal data according to claim 2 is characterized in that: In step S2, the U-Net neural network has 11 layers, the first 5 layers form a contraction path, which is the encoding stage, and the 6th to 10th layers form an expansion path, which is the decoding stage; In the contraction path, the image size is reduced through convolution and downsampling to extract shallow features; In the expansion path, deep features are obtained through transposed convolution and upsampling.

4. The remote sensing image cloud removal method for multi-temporal data according to claim 3 is characterized in that: In step S3, a single-frame satellite image with clouds S∈R is segmented into a cloud image C∈R and a background image G∈R. During the training process, the input is and In each iteration, a batch of images are sampled, and the ground image and cloud image are synthesized into a satellite image with clouds using formula (1). The network weights are updated using the gradient descent method to obtain a trained U-Net neural network.

5. The remote sensing image cloud removal method for multi-temporal data according to claim 2, characterized in that: In step S5, a relatively low balance value λ is set for the cloud region. c , ensuring that all clouds and shadows are completely decomposed into an outlier matrix, then l c =1 / max((m,n)) 1 / 2 ->0.1 / max((m,n)) 1 / 2 ; Set a higher balance value λ for cloud-free areas nc , to ensure the background, we have: l nc =1 / max((m,n)) 1 / 2 ->1; Where D∈R m×n Represents a column-stacked image sequence, where m = rows × cols is the number of pixels in a single image and n is the total number of image sequences.

6. The remote sensing image cloud removal method for multi-temporal data according to claim 3, characterized in that: In step S6, RPCA is used to process the morphological cloud mask and the given remote sensing image with clouds, which is expressed as: Wherein, Ω represents the binary cloud mask obtained in step S4, Ω - Indicates cloud-free areas.

7. The remote sensing image cloud removal method for multi-temporal data according to claim 6, characterized in that: In step S6, an inaccurate augmented Lagrangian function is used to solve formula (2), and the augmented Lagrangian function is: Among them, ∥·∥ F is the Frobenius norm, Y is the Lagrange multiplier, and μ is a positive scalar.

8. An electronic device, characterized in that: include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the remote sensing image cloud removal method for multi-temporal data as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, implement the remote sensing image cloud removal method for multi-temporal data as described in any one of claims 1 to 7.

Images