A deep learning-based method for thick cloud removal in multi-temporal remote sensing images

Abstract

The present invention provides a method for removing thick clouds from multi-temporal remote sensing images based on deep learning, comprising the following steps: Step 1, constructing a simulated thick cloud occlusion remote sensing dataset; Step 2, constructing a first-stage global multi-temporal feature fusion network; Step 3, constructing a second-stage local single-temporal information recovery network; Step 4, constructing a total network model for removing thick clouds from multi-temporal remote sensing images using a two-stage network; Step 5, constructing a total loss function for the first and second-stage networks based on the L1 loss function; Step 6, training the multi-temporal remote sensing image thick cloud removal network model using a simulated remote sensing image dataset, and using the total loss function to drive the training of the network model and guide its optimization. The present invention combines CNN (convolutional neural network) and Transformer architectures to provide an effective solution to the challenging thick cloud removal problem in remote sensing image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method for removing thick clouds from multi-temporal remote sensing images based on deep learning. Background Art

[0002] Remote sensing imagery plays an increasingly important role in agriculture, forestry, urban planning, environmental protection, and national security. With its widespread application, the demand for image quality is becoming increasingly stringent. According to statistics, much of the world is obscured by clouds, which severely impacts the quality of optical remote sensing images. Large areas of cloud obscuring ground information significantly reduce the usability of observed remote sensing images. Therefore, cloud removal has become a critical step in remote sensing image preprocessing.

[0003] Methods for addressing cloud occlusion in remote sensing images are primarily categorized into multispectral, image inpainting, and multitemporal approaches. Multispectral methods leverage the correlation between data in various spectral bands to establish corresponding mathematical functions, which are then used to reconstruct scene information in cloud-obscured areas. Multispectral methods generally perform well in removing cloud in areas obscured by thin clouds, but have limitations in reconstructing areas obscured by thick clouds. Image inpainting methods treat cloud-obscured areas as missing pixels and, using mathematical physics or deep learning methods, use information from unobstructed areas as a reference to restore information obscured by thick clouds, thereby achieving cloud removal in remote sensing images. Multitemporal methods reconstruct cloud-obscured areas using information extracted from remote sensing images of the same area collected at different times. With the rapid development of deep learning, multitemporal deep learning cloud removal methods have been increasingly adopted. However, most of these methods are based on CNNs, which are less effective at extracting global feature information, resulting in low global information utilization and poor information recovery in areas obscured by thick clouds. Summary of the Invention

[0004] In view of this, the present invention combines the CNN structure with the Transformer structure to propose a deep learning-based method for removing thick clouds from multi-temporal remote sensing images. The two-stage network structure is used to achieve global multi-feature fusion and local single-temporal information restoration, solving the problem that the existing technology cannot make good use of global fusion information, resulting in unsatisfactory restoration of thick cloud-occluded areas. The specific solution is as follows:

[0005] A method for removing thick clouds from multi-temporal remote sensing images based on deep learning, comprising the following steps:

[0006] Step 1: Generate a remote sensing image with thick cloud cover and construct a simulated dataset of remote sensing images with thick cloud cover.

[0007] Step 2: Design a processing module using the CNN structure and the Transformer structure. The processing module performs global feature fusion on the multi-temporal cascade images to construct a global multi-temporal feature fusion network for the first stage of the model.

[0008] Step 3: Construct the local single-temporal information recovery network in the second stage of the model. By fusing the global multi-temporal information of the first stage with cross-stage features, the network can be used to supplement the information of the thick cloud-occluded areas of the remote sensing image during the cloud removal process.

[0009] Step 4: Construct a multi-temporal remote sensing image thick cloud removal network model through the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information restoration network;

[0010] Step 5: Based on the L1 loss function, a total loss function is constructed for the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information recovery network.

[0011] Step 6: training the multi-temporal remote sensing image thick cloud removal network model using the thick cloud occlusion remote sensing image simulation dataset, and guiding the optimization of the network model parameters using a total loss function during the training process.

[0012] Preferably, in step 1, the method for generating the thick cloud obstruction remote sensing image is:

[0013] First, collect real cloud-free remote sensing images and create cloud mask images;

[0014] Then, the cloud mask image is used to add cloud to the real cloud-free image to obtain the thick cloud occlusion remote sensing image;

[0015] Finally, image preprocessing is performed on the thick cloud occlusion remote sensing image to generate a thick cloud occlusion remote sensing image simulation data set.

[0016] Preferably, the cloud mask image is an image formed by separating clouds from a real cloud image, thereby forming an image containing only clouds.

[0017] Preferably, the method for constructing the global multi-temporal feature fusion network in the first stage is:

[0018] S1, design processing modules through CNN structure and Transformer structure respectively;

[0019] S2, performing feature coarse extraction, feature refinement, and feature fusion on the multi-phase cascade image through a processing module to generate a global feature map integrating multi-phase information. The present invention takes three phases as an example;

[0020] S3, forms a global multi-temporal feature fusion network model through skip connections;

[0021] The process formula of the global multi-temporal feature fusion network model construction method in the first stage is shown in Formula 1:

[0022]

[0023]

[0024]

[0025] Among them, Concat means cascading the three phases. is the multi-phase cascaded image, W represents the feature coarse extraction module, M represents the encoder-decoder module, To generate the multi-temporal global feature attention map, Split represents the slicing operation. It indicates that the first stage generates feature maps corresponding to three phase images.

[0026] Preferably, the processing modules in step 2 and step 3 include a feature coarse extraction module, an encoder-decoder module, and a feature refinement module:

[0027] The feature coarse extraction module extracts local features of the image, and the feature coarse extraction module is composed of a convolution structure;

[0028] The encoder-decoder module performs information fusion on the features extracted by the feature coarse extraction module to generate a global feature attention map based on channel fusion multi-temporal information. The encoder-decoder module includes an encoder, a decoder and an intermediate layer.

[0029] The encoders and decoders are symmetrically arranged with three of them based on the middle layer, and the decoders and encoders of the corresponding layers transmit information in a skip connection manner;

[0030] The encoder, decoder and intermediate layer are composed of a Transformer structure;

[0031] An upsampling layer and a downsampling layer are respectively provided before the encoder and the decoder;

[0032] The feature refinement module refines and integrates the extracted features, and the feature refinement module includes four Transformer structures.

[0033] Preferably, the Transformer structure includes channel attention and gating network, and the expression of the Transformer structure is formula 2:

[0034]

[0035]

[0036] Among them, Attention represents channel attention, Gating represents gating mechanism, Conv 1×1 represents a 1×1 convolution, T t-1 represents the output of the upper layer codec, Represents the output of the channel attention of this layer, T t Represents the output of the encoder and decoder of this layer;

[0037] The channel attention method restores the number of channels by expanding the number of channels and calculating the importance weight of each channel, thereby fusing the feature information between different channels based on the importance between channels, and realizing information interaction in the areas covered by thick clouds in each phase;

[0038] The gating network is used to control the outflow of unnecessary information, and the gating network includes a convolutional structure and a GELU activation function;

[0039] The convolution structure consists of two convolutional layers arranged in parallel, one of which is followed by a GELU activation function, and the other convolutional layer is a separate branch.

[0040] Preferably, when the feature coarse extraction module performs feature extraction in the global multi-temporal feature fusion network model in the first stage, the convolution is first grouped. At this time, the number of channels of the convolution is set to: the number of phases × 32; the groups parameter in the convolution is the number of phases, and the groups parameters in the first and second stages are 3 and 1 respectively. The structural expression of the convolution is shown in Formula 3:

[0041]

[0042] Among them, Conv 3×3 Represents a 3×3 ordinary convolution, Conv g represents a 3×3 grouped convolution, ReLU represents the ReLU activation function, X1 is the multi-phase cascade image input in the first stage, Represents the output images corresponding to images at different time phases;

[0043] When performing feature extraction in the second-stage local single-phase information recovery network model, convolution does not require grouping, and the groups parameter is set to 1. At this time, the number of channels in the convolution layer is 32, and features are directly extracted from the image. The convolution structure expression of the module is shown in Formula 4:

[0044]

[0045] X2 is the single-phase image of the second stage.

[0046] Preferably, in step 3, the construction process of the second-stage local single-phase information recovery network is:

[0047] S1, input single-phase image;

[0048] S2, the feature refinement module is used to further refine the extracted feature map;

[0049] S3, transferring the global feature information in the first-stage global multi-temporal feature fusion network to the second-stage local single-temporal information restoration network through a cross-stage feature fusion method, removing thick cloud occlusion in the remote sensing image through the second-stage local single-temporal information restoration network to restore the local ground texture and structure obscured by the thick clouds;

[0050] The process formula for constructing the local single-phase information recovery network in the second stage is shown in Formula 5:

[0051]

[0052] Among them, X M represents the information transferred across stages by the encoder-decoder module in the first stage, which is used to recover the local single-phase information in the second stage. k represents the kth time phase image to be declouded, represents the kth image where local thick cloud removal has been completed, W Representation feature rough extraction module , M Representation encoding Device decoding Device Module .

[0053] Preferably, the cross-stage feature fusion method is specifically:

[0054] First, the output of the encoder in the first-stage global multi-temporal feature fusion network model is concatenated with the upsampled output before the decoder at the corresponding level, thereby obtaining different fused feature information on the encoder and decoder at the same level.

[0055] Then, the obtained different fusion feature information is passed across stages to the encoder in the second stage local single-phase information recovery network;

[0056] Finally, the different fused feature information obtained across stages and the downsampled output cascade are used as the input of the encoder in this layer.

[0057] Preferably, the method for constructing the total loss function in step 4 is:

[0058] S1, the total loss function model is composed of the weighted sum of the loss function of the first-stage global multi-temporal feature fusion network model and the loss function of the second-stage local single-temporal information recovery network model. The expression of the total loss function Loss is shown in Formula 6:

[0059]

[0060] In formula 7, To balance the weight parameters of the two-stage loss, Loss1 and Loss2 represent the first-stage loss function and the second-stage loss function, respectively.

[0061] S2, the loss function Loss1 of the first-stage global multi-temporal feature fusion network and the loss function Loss2 of the second-stage local single-temporal information recovery network are composed of three parts: the global L1 loss function, the L1 loss function of the area not blocked by thick clouds, and the L1 loss function of the area blocked by thick clouds. The function expression is shown in Equation 7:

[0062]

[0063] Formula 7;

[0064] Where N, N repair , N non-repair Represent the total number of image pixels, the number of pixels blocked by thick clouds and the number of pixels not blocked, I k , represents the generated image and the real cloud-free image of the kth phase, It is used to balance global and local relationships.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] Aiming at the problem of multi-temporal thick cloud removal, the present invention first obtains real cloud-free images and cloud mask images; secondly, cloud processing is performed on the real cloud-free images using the cloud mask images to construct a simulated dataset of remote sensing images blocked by thick clouds; then, a remote sensing image thick cloud removal network with global and local stages is constructed using Transformer and CNN, and a loss function is designed based on the network; finally, the constructed multi-temporal remote sensing image network model is trained using the simulated dataset of remote sensing images blocked by thick clouds. During the training process, the total loss function is used to guide the optimization of the network model parameters, thereby solving the problem that the images cannot make good use of the global fusion information during multi-temporal thick cloud removal, resulting in unsatisfactory restoration effect of the thick cloud occlusion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0068] In the attached figure:

[0069] Figure 1 This is a flow chart of the method for removing thick clouds from remote sensing images of the present invention;

[0070] Figure 2This is a structural diagram of a network model for removing thick clouds from remote sensing images according to the present invention;

[0071] Figure 3 This is a structural diagram of the cross-stage feature fusion method of the present invention;

[0072] Figure 4 This is the Transformer structure diagram used in the present invention;

[0073] Figure 5 This is a structural diagram of the feature rough extraction module provided by the present invention;

[0074] Figure 6 This is a structural diagram of the encoder-decoder module provided by the present invention. DETAILED DESCRIPTION

[0075] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0076] According to the attached Figure 1-6 A method for removing thick clouds from multi-temporal remote sensing images based on deep learning is shown, characterized in that it includes the following steps:

[0077] Step 1: Generate a remote sensing image with thick cloud cover and construct a simulated dataset of remote sensing images with thick cloud cover.

[0078] Step 2: Design a processing module using the CNN structure and the Transformer structure. The processing module performs global feature fusion on the multi-temporal cascade images to construct a global multi-temporal feature fusion network for the first stage of the model.

[0079] Step 3: Construct the local single-temporal information recovery network in the second stage of the model. By fusing the global multi-temporal information of the first stage with cross-stage features, the network can be used to supplement the information of the thick cloud-occluded areas of the remote sensing image during the cloud removal process.

[0080] Step 4: Construct a multi-temporal remote sensing image thick cloud removal network model through the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information restoration network;

[0081] Step 5: Based on the L1 loss function, a total loss function is constructed for the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information recovery network.

[0082] Step 6: training the multi-temporal remote sensing image thick cloud removal network model using the thick cloud occlusion remote sensing image simulation dataset, and guiding the optimization of the network model parameters using a total loss function during the training process.

[0083] Furthermore, in step 1, the method for generating the thick cloud obstruction remote sensing image is:

[0084] First, collect real cloud-free remote sensing images and create cloud mask images;

[0085] Then, the cloud mask image is used to add cloud to the real cloud-free image to obtain the thick cloud occlusion remote sensing image;

[0086] Finally, image preprocessing is performed on the thick cloud occlusion remote sensing image to generate a thick cloud occlusion remote sensing image simulation data set.

[0087] Furthermore, the cloud mask image refers to an image formed by separating clouds from a real cloud image, thereby forming an image containing only clouds.

[0088] Furthermore, the method for constructing the global multi-temporal feature fusion network in the first stage is:

[0089] S1, design processing modules through CNN structure and Transformer structure respectively;

[0090] S2, performing feature coarse extraction, feature refinement, and feature fusion on the multi-phase cascade image through a processing module to generate a global feature map integrating multi-phase information. The present invention takes three phases as an example;

[0091] S3, forms a global multi-temporal feature fusion network model through skip connections;

[0092] The process formula of the global multi-temporal feature fusion network model construction method in the first stage is shown in Formula 1:

[0093]

[0094]

[0095]

[0096] Among them, Concat means cascading the three phases. is the multi-phase cascaded image, W represents the feature coarse extraction module, M represents the encoder-decoder module, To generate the multi-temporal global feature attention map, Split represents the slicing operation. It indicates that the first stage generates feature maps corresponding to three phase images.

[0097] Furthermore, the processing modules in step 2 and step 3 include a feature coarse extraction module, an encoder-decoder module, and a feature refinement module;

[0098] The feature coarse extraction module extracts local features of the image. The feature coarse extraction module is composed of a convolutional structure:

[0099] The encoder-decoder module performs information fusion on the features extracted by the feature coarse extraction module to generate a global feature attention map based on channel fusion multi-temporal information. The encoder-decoder module includes an encoder, a decoder and an intermediate layer.

[0100] The encoders and decoders are symmetrically arranged with three of them based on the middle layer, and the decoders and encoders of the corresponding layers transmit information in a skip connection manner;

[0101] The encoder, decoder and intermediate layer are composed of a Transformer structure;

[0102] An upsampling layer and a downsampling layer are respectively provided before the encoder and the decoder;

[0103] The feature refinement module refines and integrates the extracted features, and the feature refinement module includes four Transformer structures.

[0104] Furthermore, the Transformer structure includes channel attention and gating networks. The flow formula of the Transformer structure is as follows:

[0105]

[0106]

[0107] Among them, Attention represents channel attention, Gating represents gating mechanism, Conv 1×1 represents a 1×1 convolution, T t-1 represents the output of the upper layer codec, Represents the output of the channel attention of this layer, T t Represents the output of the encoder and decoder of this layer;

[0108] The channel attention method restores the number of channels by expanding the number of channels and calculating the importance weight of each channel, thereby fusing the feature information between different channels based on the importance between channels, and realizing information interaction in the areas covered by thick clouds in each phase;

[0109] The gating network is used to control the outflow of unnecessary information, and the gating network includes a convolutional structure and a GELU activation function;

[0110] The convolution structure consists of two convolutional layers arranged in parallel, one of which is followed by a GELU activation function, and the other convolutional layer is a separate branch.

[0111] Furthermore, when the feature coarse extraction module performs feature extraction in the global multi-temporal feature fusion network model in the first stage, the convolution is first grouped. At this time, the number of channels of the convolution is set to: the number of phases × 32; the groups parameter in the convolution is the number of phases, and the groups parameters in the first and second stages are 3 and 1 respectively. The process formula of the convolution structure is shown in Equation 3:

[0112]

[0113] Among them, Conv 3×3 Represents a 3×3 ordinary convolution, Conv g represents a 3×3 grouped convolution, ReLU represents the ReLU activation function, X1 is the multi-phase cascade image input in the first stage, Represents the output images corresponding to images at different time phases;

[0114] When performing feature extraction in the second-stage local single-phase information recovery network model, convolution does not require grouping, and the groups parameter is set to 1. At this time, the number of channels in the convolution layer is 32, and features are directly extracted from the image. The convolution structure expression of the module is shown in Formula 4:

[0115]

[0116] X2 is the single-phase image of the second stage.

[0117] Furthermore, in step 3, the construction process of the second-stage local single-phase information recovery network is as follows:

[0118] S1, input single-phase image;

[0119] S2, the feature refinement module is used to further refine the extracted feature map;

[0120] S3, transferring the global feature information in the first-stage global multi-temporal feature fusion network to the second-stage local single-temporal information restoration network through a cross-stage feature fusion method, removing thick cloud occlusion in the remote sensing image through the second-stage local single-temporal information restoration network to restore the local ground texture and structure obscured by the thick clouds;

[0121] The process formula for constructing the local single-phase information recovery network in the second stage is shown in Formula 5:

[0122]

[0123] Among them, X M represents the information transferred across stages by the encoder-decoder module in the first stage, which is used to recover the local single-phase information in the second stage. krepresents the kth time phase image to be declouded, represents the kth image where local thick cloud removal has been completed, W Representation feature rough extraction module , M Representation encoding Device decoding Device Module .

[0124] Furthermore, the cross-stage feature fusion method is specifically as follows:

[0125] First, the output of the encoder in the first-stage global multi-temporal feature fusion network model is concatenated with the upsampled output before the decoder at the corresponding level, thereby obtaining different fused feature information on the encoder and decoder at the same level.

[0126] Then, the obtained different fusion feature information is passed across stages to the encoder in the second stage local single-phase information recovery network;

[0127] Finally, the different fused feature information obtained across stages and the downsampled output cascade are used as the input of the encoder in this layer.

[0128] Furthermore, the total loss function in step 4 is constructed as follows:

[0129] S1, the total loss function model is composed of the weighted sum of the loss function of the first-stage global multi-temporal feature fusion network model and the loss function of the second-stage local single-temporal information recovery network model. The expression of the total loss function Loss is shown in Formula 6:

[0130]

[0131] In formula 7, To balance the weight parameters of the two-stage loss, Loss1 and Loss2 represent the first-stage loss function and the second-stage loss function, respectively.

[0132] S2, the loss function Loss1 of the first-stage global multi-temporal feature fusion network and the loss function Loss2 of the second-stage local single-temporal information recovery network are composed of three parts: the global L1 loss function, the L1 loss function of the area not blocked by thick clouds, and the L1 loss function of the area blocked by thick clouds. The function expression is shown in Equation 7:

[0133]

[0134] Where N, N repair , N non-repair Represent the total number of image pixels, the number of pixels blocked by thick clouds and the number of pixels not blocked, I k , represents the generated image and the real cloud-free image of the kth phase, It is used to balance global and local relationships.

[0135] It should be noted that:

[0136] The Transformer architecture in this paper is based on a new visual Transformer model proposed by Syed Waqas Zamir et al. in their paper "Restormer: Efficient Transformer for High-Resolution Image Restoration." This model first improves the Transformer's spatial self-attention to channel self-attention with depthwise separable convolutions; it then replaces the convolutional feedforward network with a gated network with gating and depthwise separable convolutions. This allows for the capture of long-range pixel interactions, enabling image processing tasks such as image denoising and deblurring. This paper applies the aforementioned Transformer to the field of multi-temporal remote sensing image cloud removal. This Transformer architecture can fuse feature information between different channels and perform information interaction on local areas obscured by thick clouds in each phase. Compared to traditional Transformer architectures, it is more suitable for extracting and fusing feature information between multiple temporal phases, making it a better choice for addressing the problem of multi-temporal remote sensing thick cloud removal.

[0137] The Transformer structure used in the encoder-decoder module of the present invention consists of channel attention and gating network.

[0138] In the present invention, the cross-stage feature fusion method is to cascade the output of the encoder in the first stage with the up-sampled output before the corresponding level decoder, and pass it to the encoder in the second stage for feature fusion, so as to ensure that the fused feature information of the first stage is passed to the second stage, thereby restoring the ground structure information obscured by thick clouds.

[0139] In the present invention, the multi-temporal remote sensing image network model is trained by constructing the thick cloud occlusion remote sensing image simulation dataset. The process is as follows: first, the remote sensing image simulation dataset with thick cloud occlusion is input, the image after cloud removal and restoration is output, and the total loss function is calculated between the output image and the real cloud-free image; then, the network parameters are automatically iteratively updated through back propagation, and the network model training is completed after multiple iterations of training until the set number of training times is reached; finally, the network model is tested.

[0140] In this paper, each codec is preceded by corresponding upsampling and downsampling layers. The Transformer architecture employed in this paper utilizes channel attention, which can better collect interactive information from features across different channels, thereby representing ground texture and structure information more accurately and in greater detail. Furthermore, compared to traditional Transformer architectures, the channel attention-based Transformer architecture is more suitable for extracting and fusing inter-channel feature information, making it a better choice for addressing the problem of thick cloud removal in multi-temporal remote sensing.

[0141] The present invention is described in detail below through specific examples.

[0142] Example 1

[0143] This embodiment uses remote sensing images acquired at three different time phases of the same area as an example to illustrate the present invention in detail. The three different time phases are the first time phase, the second time phase, and the third time phase. The first time phase, the second time phase, and the third time phase refer to the time intervals of the satellite's operation cycle, and are respectively on the first day of the first cycle, the first day of the second cycle, and the first day of the third cycle. For the remote sensing images acquired, the specific simulation processing method is as follows:

[0144] First, obtain the true cloud-free image and cloud mask image;

[0145] Secondly, the real cloud-free image and the cloud mask image are multiplied to obtain a remote sensing image that simulates thick cloud occlusion;

[0146] Then, the obtained three-phase simulated thick cloud occlusion remote sensing images are cascaded and used as the input of the first-stage global multi-temporal feature fusion network model. After passing through the feature coarse extraction module to obtain the local feature information of the thick cloud occlusion remote sensing images, they are input into the encoder-decoder module for information interaction from the channel dimension. After fusing the multi-temporal feature information, the extracted features are refined and integrated by the feature refinement module, and then separated and restored to the three-phase image for the loss function calculation of the first stage.

[0147] Finally, the second-stage local single-phase information restoration network model takes the single-phase image as input, performs feature extraction and fusion through the feature coarse extraction module, encoder-decoder module, and feature refinement module to obtain the restored image of the remote sensing image obscured by thick clouds.

[0148] It is worth noting that the global multi-temporal feature information of the first stage is fused across stages in the encoder-decoder module to recover thick cloud occlusion information.

[0149] The outputs of the first-stage global multi-temporal feature fusion network model and the second-stage local single-temporal information recovery network model are jointly used to calculate the loss function. After iterative training, the optimal network model parameters are obtained. The trained model is then fed into a test image with thick cloud occlusion, and the output is an image without thick clouds.

[0150] At the same time, the feature extraction methods in the first stage global multi-temporal feature fusion network model and the second stage local single-temporal information recovery network model are slightly different:

[0151] 1) In the first stage, the number of convolution channels in the global multi-temporal feature fusion network model is: the number of temporal phases × 32 (for example, for a three-temporal cascade image input, the number of channels is 3 × 32 = 96). The number of temporal phases is set as the groups parameter in the convolution. During convolution, the feature map corresponding to each temporal phase is first obtained, and then the feature information is fused.

[0152] 2) In the second stage, the number of channels in the local single-phase information recovery network model is set to 32, and the groups parameter is set to 1. No group convolution is required, and feature extraction is performed directly.

[0153] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for removing thick clouds from multi-temporal remote sensing images based on deep learning, characterized in that: The following steps are involved: Step 1: Generate a remote sensing image with thick cloud cover and construct a simulated dataset of remote sensing images with thick cloud cover. Step 2: Design a processing module using the CNN structure and the Transformer structure. The processing module performs global feature fusion on the multi-temporal cascade images to construct a global multi-temporal feature fusion network for the first stage of the model. Step 3: Construct a local single-temporal information recovery network in the second stage of the model. By fusing the global multi-temporal information of the first stage with cross-stage features, the network can be used to supplement the information of the thick cloud-occluded areas of the remote sensing image during the cloud removal process. Step 4: Construct a multi-temporal remote sensing image thick cloud removal network model through the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information restoration network; Step 5: Based on the L1 loss function, a total loss function is constructed for the first-stage global multi-temporal feature fusion network and the second-stage local single-temporal information recovery network. Step 6, training the multi-temporal remote sensing image thick cloud removal network model using the thick cloud occlusion remote sensing image simulation dataset, and guiding the optimization of the network model parameters using a total loss function during the training process; The construction method of the global multi-temporal feature fusion network in the first stage is: 1) Design processing modules using CNN and Transformer structures respectively; 2) The processing module performs feature coarse extraction, feature refinement, and feature fusion on the multi-phase cascade image to generate a global feature map that integrates multi-phase information. This invention takes three phases as an example; 3) Forming a global multi-temporal feature fusion network model through skip connections; The process formula of the global multi-temporal feature fusion network model construction method in the first stage is shown in Formula 1: Formula 1; in, Indicates cascading of the three phases. is the multi-temporal cascaded image, W represents the feature rough extraction module, M represents the encoder-decoder module, is the generated multi-temporal global feature attention map, Split Represents a slice operation, Indicates that the first stage generates feature maps corresponding to three phase images; In step 3, the construction process of the second-stage local single-phase information recovery network is as follows: S1, input single-phase image; S2, the feature refinement module is used to further refine the extracted feature map; S3, transferring the global feature information in the first-stage global multi-temporal feature fusion network to the second-stage local single-temporal information restoration network through a cross-stage feature fusion method, removing thick cloud occlusion in the remote sensing image through the second-stage local single-temporal information restoration network to restore the local ground texture and structure obscured by the thick clouds; The process formula for constructing the local single-phase information recovery network in the second stage is shown in Formula 5: Formula 5; in, X M represents the information transmitted across stages by the encoder-decoder module in the first stage, and is used to recover the local single-phase information in the second stage. X k Indicates the k A time-phase image to be declouded, Indicates the k An image with local thick cloud removal completed, W represents the feature rough extraction module, M Represents an encoder-decoder module.

2. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 1, characterized in that: In step 1, the method for generating the thick cloud obstruction remote sensing image is: First, collect real cloud-free remote sensing images and create cloud mask images; Then, the cloud mask image is used to add cloud to the real cloud-free image to obtain the thick cloud occlusion remote sensing image; Finally, image preprocessing is performed on the thick cloud occlusion remote sensing image to generate a thick cloud occlusion remote sensing image simulation data set.

3. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 2 is characterized in that: The cloud mask image is an image formed by separating clouds from a real cloud image, thereby forming an image containing only clouds.

4. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 1, characterized in that: The processing modules in steps 2 and 3 include a feature coarse extraction module, an encoder-decoder module, and a feature refinement module: The feature coarse extraction module extracts local features of the image, and the feature coarse extraction module is composed of a convolution structure; The encoder-decoder module performs information fusion on the features extracted by the feature coarse extraction module to generate a global feature attention map based on channel fusion multi-temporal information. The encoder-decoder module includes an encoder, a decoder and an intermediate layer. The encoders and decoders are symmetrically arranged with three of them based on the middle layer, and the decoders and encoders of the corresponding layers transmit information in a skip connection manner; The encoder, decoder and intermediate layer are composed of a Transformer structure; An upsampling layer and a downsampling layer are respectively provided before the encoder and the decoder; The feature refinement module refines and integrates the extracted features, and the feature refinement module includes four Transformer structures.

5. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 4, characterized in that: The Transformer structure includes channel attention and gating networks. The flow formula of the Transformer structure is as follows: Formula 2; in, Attention represents channel attention, Gating represents the gating mechanism, Conv 1×1 represents a 1×1 convolution, represents the output of the upper layer codec, Represents the output of the channel attention of this layer, Represents the output of the encoder and decoder of this layer; The channel attention method restores the number of channels by expanding the number of channels and calculating the importance weight of each channel, thereby fusing the feature information between different channels based on the importance between channels, and realizing information interaction in the areas covered by thick clouds in each phase; The gating network is used to control the outflow of unnecessary information, and the gating network includes a convolutional structure and a GELU activation function; The convolution structure consists of two convolutional layers arranged in parallel, one of which is followed by a GELU activation function, and the other convolutional layer is a separate branch.

6. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 5, characterized in that: When the feature coarse extraction module performs feature extraction in the global multi-temporal feature fusion network model in the first stage, the convolution is first grouped. At this time, the number of channels of the convolution is set to: number of phases × 32; the groups parameter in the convolution is the number of phases, and the groups parameters in the first and second stages are 3 and 1 respectively. The process formula of the convolution structure is shown in Equation 3: Formula 3; in, Conv 3×3 represents a 3×3 ordinary convolution, Conv g represents a 3×3 grouped convolution, ReLU express ReLU activation function, X 1 is the multi-temporal cascade image input to the first stage, Represents the output images corresponding to images at different time phases; When performing feature extraction in the second-stage local single-phase information recovery network model, convolution does not require grouping, and the groups parameter is set to 1. At this time, the number of channels in the convolution layer is 32, and features are directly extracted from the image. The convolution structure expression of the module is shown in Formula 4: Formula 4; X 2 This is the single-phase image of the second stage.

7. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 6, characterized in that: The cross-stage feature fusion method is specifically as follows: First, the output of the encoder in the first-stage global multi-temporal feature fusion network model is concatenated with the upsampled output before the decoder at the corresponding level, thereby obtaining different fused feature information of the encoder and decoder at the same level; Then, the obtained different fusion feature information is passed across stages to the encoder in the second stage local single-phase information recovery network; Finally, the different fused feature information obtained across stages and the downsampled output cascade are used as the input of the encoder in this layer.

8. The method for removing thick clouds from multi-temporal remote sensing images based on deep learning according to claim 1, characterized in that: The total loss function in step 4 is constructed as follows: S1, the total loss function model is composed of the weighted sum of the loss function of the first-stage global multi-temporal feature fusion network model and the loss function of the second-stage local single-temporal information recovery network model. Loss The expression of is shown in formula 6: Formula 6; In formula 7, To balance the weight parameters of the two-stage loss, Loss 1 and Loss 2 Represent the first stage loss function and the second stage loss function respectively; S2, loss function of the first stage global multi-temporal feature fusion network Loss 1 And the loss function of the second stage local single-phase information recovery network Loss 2 It consists of three parts: the global L1 loss function, the L1 loss function of the area not blocked by thick clouds, and the L1 loss function of the area blocked by thick clouds. The function expression is shown in Equation 7: Formula 7; in N , N repair , N non-repair Represent the total number of image pixels, the number of pixels blocked by thick clouds, and the number of pixels not blocked. Indicates the k The generated image and the real cloud-free image of the time phase, It is used to balance global and local relationships.

Images