A method and system for progressive removal of thin clouds in remote sensing images on a task-by-task basis.

Abstract

The present invention discloses a method and system for removing thin clouds from remote sensing images in a progressive task-by-task manner. The removal method comprises the following steps: a local information-guided cloud removal sub-step, a structural information restoration sub-step, and a high-frequency detail information restoration sub-step. The present invention decomposes the complex thin cloud removal problem into three sub-steps and uses a global information-guided cloud removal sub-network, a structural information restoration sub-network, and a high-frequency detail information restoration sub-network to solve the problem task by task respectively. The thin cloud removal, structural information restoration, and high-frequency detail information restoration sub-steps are gradually completed in a progressive manner, thereby achieving effective restoration of low-frequency structural information and high-frequency detail information in the remote sensing image while removing thin clouds from the remote sensing image, thereby solving the complex thin cloud removal problem in remote sensing images with high fidelity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for removing thin clouds from remote sensing images in a task-by-task progressive manner. Background Art

[0002] Remote sensing images, as a mainstream method for displaying Earth observation results, have gained widespread attention across various fields. However, the acquisition of remote sensing images involves various complex environments, such as rain, clouds, and fog. Among these, thin cloud cover is particularly common in remote sensing image acquisition, obscuring ground features and severely impacting subsequent interpretation, detection, segmentation, and other related applications. Therefore, the problem of thin cloud removal in remote sensing images has significant research value. Currently, most technical approaches to thin cloud removal in remote sensing images are based on deep learning methods, typically employing convolutional neural networks or Transformer network architectures to perform end-to-end thin cloud removal within a single network structure, achieving relatively good results.

[0003] For example, Chinese patent CN118314052A discloses a method for removing thin clouds from remote sensing images. This method includes: dividing the image to be processed into blocks to obtain multiple first sub-images; acquiring the first sub-images in a preset order; removing thin clouds from the first sub-images using a trained thin-cloud removal model to obtain a cloud-free sub-image; performing grayscale transformation on the cloud-free sub-image to obtain a second sub-image; and stitching all the second sub-images together to obtain a cloud-free image of the image to be processed. In other words, it removes thin clouds from images containing thin clouds using a conditional variational autoencoding-based thin-cloud removal model.

[0004] However, using only one network structure to solve the complex problem of thin cloud removal is inherently difficult to achieve good results in thin cloud removal, structural information restoration, and high-frequency detail restoration at the same time, and to restore remote sensing images with thin clouds with high fidelity. Summary of the Invention

[0005] The purpose of this invention is to provide a task-by-task progressive method for removing thin clouds from remote sensing images. This method effectively restores low-frequency structural information and high-frequency detail information in remote sensing images while removing thin clouds, thereby solving the complex problem of thin cloud removal in remote sensing images with high fidelity.

[0006] Furthermore, the present invention aims to provide a system for progressive remote sensing imagery of thin clouds.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] A task-by-task progressive method for removing thin clouds from remote sensing images includes the following steps:

[0009] 1) The global information-guided cloud removal sub-step includes extracting the shallow feature vector of the input remote sensing image containing thin clouds, then downsampling and reducing its dimensionality to obtain the first feature vector, obtaining the cloud removal guidance vector from the shallow feature vector, and inputting the cloud removal guidance vector and the first feature vector together into multiple consecutive cloud removal and restoration modules to remove the thin clouds, and outputting the first high-dimensional feature vector and the first result image after its fusion with the remote sensing image containing thin clouds;

[0010] 2) The structural information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, extracting features through a convolutional layer and downsampling to reduce the dimensionality to obtain the second feature vector, which is then input into multiple structural information restoration modules to iteratively repair the structure, and outputting the second high-dimensional feature vector and the second result image after fusing it with the first result image;

[0011] 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the second high-dimensional feature vector, extracting the third feature vector through a convolutional layer, and then inputting it into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The output feature vector after the high-frequency detail information restoration is optimized by a convolutional layer and then fused with the second result image to finally output the remote sensing image with thin clouds removed.

[0012] As one preferred embodiment, the method for obtaining the cloud removal guidance vector is as follows:

[0013] The shallow feature vectors are input into two average pooling layers with different parameters to extract guidance information at different scales.

[0014] The cloud removal guidance vector is obtained by processing guidance information at two different scales using attention and residual mechanisms.

[0015] As one preferred embodiment, the processing method of the cloud restoration module is as follows:

[0016] The initial cloud removal guidance feature vector is obtained by convolving the first input feature vector with the cloud removal guidance vector.

[0017] After expanding the input dimension of the initial cloud-guided feature vector, vector segmentation is performed in the channel dimension to obtain two feature vectors.

[0018] The two feature vectors after feature segmentation are processed by attention and residual mechanisms with the cloud removal guidance vector and the first input feature vector, respectively. After feature concatenation, the concatenated feature vectors are input into the convolutional layer to obtain the feature vector after thin cloud removal output by the cloud removal restoration module. The first input feature vector is either the first feature vector or the feature vector output by the previous cloud removal restoration module. The output feature vector of the last cloud removal restoration module is convolved and upsampled to obtain the first high-dimensional feature vector.

[0019] As one preferred embodiment, the processing method of the structural information restoration module is as follows:

[0020] After layer normalization, the second input feature vector is input into a series of deep convolutional layers and pointwise convolutional layers, and then the feature vector is restored by using residual operation to output the preliminary structural information.

[0021] After layer normalization, the feature vectors of the preliminary structural information reconstruction are input into continuous deep convolutional layers and pointwise convolutional layers to obtain feature vectors of further structural information reconstruction.

[0022] Attention and residual mechanisms are used to process the feature vectors for further structural information restoration to output the structural information restoration feature vector; wherein, the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module;

[0023] The output of the final-level structural information restoration module is convolved and upsampled to obtain the second high-dimensional feature vector.

[0024] As one preferred embodiment, the processing method of the high-frequency detail restoration module is as follows:

[0025] After normalizing the third input feature vector layer, a fast Fourier transform is performed to separate the amplitude component vector and the phase component vector;

[0026] The amplitude vector and phase vector are subjected to deep convolutional layer, pointwise convolutional layer, activation function, and convolutional layer operations respectively to restore high-frequency information in the frequency domain;

[0027] Perform an inverse fast Fourier transform on the amplitude and phase vectors after restoring the high-frequency detail information to transform the feature vectors after restoring the high-frequency information from the frequency domain to the image domain;

[0028] The feature vector is obtained by using the attention mechanism and residual connection to the feature vector of the image domain after high-frequency information restoration. The third input feature vector is the third feature vector or the feature vector output by the previous high-frequency detail restoration module.

[0029] The final high-frequency detail restoration module outputs a third feature vector after convolution and upsampling.

[0030] A task-by-task progressive system for removing thin clouds from remotely sensed images, including:

[0031] 1) A global information-guided declouding subnetwork is used to input a remote sensing image containing thin clouds and extract shallow feature vectors. Then, it is downsampled and dimensionality reduced to obtain the first feature vector. The declouding guidance vector is obtained from the shallow feature vector. The declouding guidance vector and the first feature vector are input together into multiple consecutive declouding restoration modules to remove thin clouds. The output is the first high-dimensional feature vector and the first result image after it is fused with the remote sensing image containing thin clouds.

[0032] 2) A structural information complex atom network is used to input a remote sensing image containing thin clouds and the first high-dimensional feature vector, and then perform convolutional layer to extract features and downsample to reduce the dimensionality to obtain a second feature vector. This second feature vector is then input to multiple structural information restoration modules to iteratively repair the structure and output the second high-dimensional feature vector and the second result image after it is fused with the first result image.

[0033] 3) A high-frequency detail information complex atom network is used to input a remote sensing image containing thin clouds and the second high-dimensional feature vector. The features are then concatenated and fused, and the third feature vector is extracted through a convolutional layer. This third feature vector is then input to multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final high-frequency detail information restored feature vector is then optimized by a convolutional layer and fused with the second result image to finally output a remote sensing image with thin clouds removed.

[0034] As one preferred approach, corresponding loss functions are established for the three sub-networks respectively, and then the loss function of overall high-fidelity restoration is added to constrain the overall image dimension. These are then combined into a total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention decomposes the complex problem of thin cloud removal into three sub-steps, corresponding to a global information-guided cloud removal sub-network, a structural information complex atom network, and a high-frequency detail information complex atom network, respectively. This achieves a task-by-task, step-by-step solution. By progressively completing the global information-guided cloud removal sub-step, the structural information complex atom step, and the high-frequency detail information complex atom step, it effectively restores low-frequency structural information and high-frequency detail information in remote sensing images while removing thin clouds, thus solving the complex problem of thin cloud removal in remote sensing images with high fidelity. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of the task-by-task progressive remote sensing image thin cloud system of the present invention.

[0038] Figure 2 A schematic diagram of a cloud subnetwork used to guide global information dissemination.

[0039] Figure 3 This is a schematic diagram of a complex atom network for structural information.

[0040] Figure 4 This is a schematic diagram of a complex atom network for high-frequency detail information.

[0041] Figure 5 This is a schematic diagram of the global information extraction unit structure.

[0042] Figure 6 This is a schematic diagram of the cloud restoration module structure.

[0043] Figure 7 This is a schematic diagram of the structural information restoration module.

[0044] Figure 8 A schematic diagram of the module structure for restoring high-frequency detail information.

[0045] Figures 9-11 This is a comparison image of a remote sensing image containing thin clouds and a remote sensing image after thin clouds have been removed using the method of this invention. Detailed Implementation

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] The method for removing thin clouds from remote sensing images in a task-by-task progressive manner according to the present invention includes the following steps:

[0048] 1) The global information-guided cloud removal sub-step includes convolving the input remote sensing image containing thin clouds to extract shallow feature vectors, downsampling and reducing the dimensionality of the shallow feature vectors to obtain a first feature vector, inputting the shallow feature vector into the global information extraction unit to obtain the cloud removal guidance vector, and then inputting it together with the first feature vector into multiple cloud removal and restoration modules (generally 3-6, preferably 4 consecutive) to remove the thin clouds, outputting a first high-dimensional feature vector and a first result image after its fusion with the remote sensing image containing thin clouds; wherein, the method for obtaining the cloud removal guidance vector is as follows...

[0049] The shallow feature vectors are input into two average pooling layers with different parameters to extract guidance information at different scales.

[0050] The cloud removal guidance vector is obtained by processing guidance information at two different scales using attention and residual mechanisms. The first average pooling layer uses a global pooling strategy to obtain overall image context information, while the second average pooling layer uses a regional pooling strategy to obtain local structural features, forming scale-differentiated guidance information. A parameter-differentiated dual pooling structure is used to achieve parallel extraction of multi-scale features. Global and local semantic information are dynamically fused through an attention mechanism. Residual connections are introduced to ensure lossless transmission of original feature information. This method can significantly improve the ability to perceive cloud-occluded areas at different scales in remote sensing image cloud removal processing and enhance the model's adaptability to complex scenes.

[0051] The processing method of the cloud restoration module is as follows:

[0052] The initial cloud removal-guided feature vector is obtained by convolving the first input feature vector with the cloud removal guidance vector.

[0053] After expanding the input dimension of the initial cloud-guided feature vector, vector segmentation is performed in the channel dimension to obtain two feature vectors.

[0054] After feature segmentation, the two feature vectors are processed with the cloud removal guidance vector and the first input feature vector using attention and residual mechanisms, respectively. These processed vectors are then concatenated and input into a convolutional layer to obtain the thin cloud-removed feature vector output by the cloud removal module. The input feature vector is either the first feature vector or the feature vector output by the previous cloud removal module, or the first high-dimensional feature vector output by the last cloud removal module. That is, except for the first one, the other cloud removal modules use the feature vectors mapped to the low-resolution vector domain to participate in the convolution.

[0055] 2) The structural information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, then performing convolutional layer feature extraction and downsampling to reduce dimensionality to obtain a second feature vector. Multiple feature vectors are then input, typically 3-6, preferably 4, and the structural information restoration modules iteratively repair the structure, outputting the second high-dimensional feature vector and the second result image fused with the first result image. The processing method of the structural information restoration module is as follows:

[0056] After layer normalization, the second input feature vector is input into a series of deep convolutional layers and pointwise convolutional layers, and then the feature vector is restored by using residual operation to output the preliminary structural information.

[0057] After layer normalization, the feature vectors of the preliminary structural information reconstruction are input into continuous deep convolutional layers and pointwise convolutional layers to obtain feature vectors of further structural information reconstruction.

[0058] Attention and residual mechanisms are used to process the feature vectors for further structural information restoration to output the feature vectors for structural information restoration in this module; wherein, the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module;

[0059] The feature vector output by the final-level structural information restoration module is the second high-dimensional feature vector.

[0060] 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with a second high-dimensional feature vector, extracting basic features through a convolutional layer to obtain a third feature vector, and iteratively restoring the high-frequency details of the image through multiple (usually 3-6, preferably 4) high-frequency detail restoration modules. The output feature vector is then optimized by a convolutional layer and fused with the second result image to finally output a remote sensing image with thin clouds removed. The processing method of the high-frequency detail restoration module is as follows:

[0061] After normalizing the third input feature vector layer, a fast Fourier transform is performed to separate the amplitude component vector and the phase component vector;

[0062] The amplitude vector and phase vector are subjected to deep convolutional layer, pointwise convolutional layer, activation function, and convolutional layer operations respectively to restore high-frequency information in the frequency domain;

[0063] Perform an inverse fast Fourier transform on the amplitude and phase vectors after restoring the high-frequency detail information to transform the feature vectors after restoring the high-frequency information from the frequency domain to the image domain;

[0064] The feature vector is obtained by using the attention mechanism and residual connection to restore the high-frequency information of the image domain. The third input feature vector is the third feature vector or the feature vector output by the previous high-frequency detail restoration module. The third feature vector is obtained by convolution and upsampling the output of the last high-frequency detail restoration module.

[0065] Meanwhile, this invention also discloses a task-by-task progressive remote sensing image thin cloud removal system, comprising,

[0066] The global information-guided declouding sub-network is used to extract shallow feature vectors from the input remote sensing image containing thin clouds through convolution, and obtain the first feature vector after downsampling and dimensionality reduction. The declouding guidance vector is obtained and input together with the first feature vector into multiple consecutive declouding restoration modules to remove thin clouds. The output is the first high-dimensional feature vector and the first result image after its fusion with the remote sensing image containing thin clouds.

[0067] The structural information complex atom network is used to concatenate and fuse a remote sensing image containing thin clouds with a first high-dimensional feature vector, then perform convolutional layer feature extraction and downsampling layer dimensionality reduction to obtain a second feature vector. Then, it is sequentially input into multiple structural information restoration modules to iteratively repair the structure, and outputs the second high-dimensional feature vector and the second result image after its fusion with the first result image.

[0068] A high-frequency detail information complex atom network is used to concatenate and fuse a remote sensing image containing thin clouds with a second high-dimensional feature vector. Then, a convolutional layer is used to extract basic features to obtain a third feature vector. After multiple high-frequency detail restoration modules, the high-frequency details of the image are restored iteratively. The obtained feature vector is optimized by a convolutional layer and then fused with the second result image to finally output a remote sensing image with thin clouds removed.

[0069] Each of the three sub-networks is given a corresponding loss function to constrain it. Then, an overall high-fidelity restoration loss function is added to constrain it in the overall image dimension. These are then combined into a total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

[0070] This invention receives a remote sensing image containing thin clouds as input and utilizes a global information-driven declouding algorithm to perform preliminary detection and restoration of the thin cloud region, aiming to remove thin cloud occlusion while preserving the basic content. The structural restoration sub-step focuses on restoring the integrity of the image structure (such as contours and main framework), restoring structural information through a feature reconstruction algorithm, and outputting a structurally restored image, achieving improved structural accuracy and clearer basic contours. The high-frequency detail information restoration sub-step receives the structurally restored image and simultaneously correlates it with high-frequency features of the original remote sensing image containing thin clouds for detail supplementation and verification, performing enhanced restoration on high-frequency details such as texture and edges, and finally outputting a complete remote sensing image with thin clouds removed.

[0071] The present invention will be further described below through specific embodiments.

[0072] The cloud removal method for remote sensing images based on a task-by-task progressive restoration network, as described in this invention, comprises three sub-steps (sub-networks). The specific method flow is as follows: Figure 1 As shown, the process will now be described in detail. First, a global information-guided cloud removal sub-step is performed on the remote sensing image containing thin clouds, laying the foundation for the subsequent two sub-steps (subtasks), as shown below:

[0073] (1)

[0074] in, These represent the first result image and the first high-dimensional feature vector output from the global information-guided declouding sub-step, respectively. Given a remote sensing image containing thin clouds as input, GRNet uses the proposed global information-guided declouding subnetwork, whose network structure is as follows: Figure 2 As shown, details will be described in subsequent steps.

[0075] Then, the global information guides the output of the first result image of the cloud removal sub-step. First high-dimensional feature vector Remote sensing images containing thin clouds The input is fed into the structural information complex atom network. Based on the global information guiding the declouding subnetwork to complete the declouding task, the structural information is restored, as shown below:

[0076] (2)

[0077] in, These are the second result image and the second high-dimensional feature vector output from the structural information complex atom step, respectively. TRNet is the proposed structural information complex atom network, and its network structure is as follows: Figure 3 As shown, details will be described in subsequent steps.

[0078] Finally, the second result image output by the structural information complex atom step is... Second high-dimensional feature vector Remote sensing images containing thin clouds The input is fed into the high-frequency detail information complex atom network, and the high-frequency detail information complex atom step is performed as follows:

[0079] (3)

[0080] in, To ultimately output a remote sensing image with thin clouds removed, HRNet is a proposed high-frequency detail information complex atom network, the structure of which is as follows: Figure 4 As shown, details will be described in subsequent steps.

[0081] The above process illustrates the network structure of the remote sensing image cloud removal method based on a task-by-task progressive restoration network. Furthermore, corresponding loss functions are established for each of the three sub-networks to constrain their respective sub-steps. Then, an overall high-fidelity restoration loss function is added to constrain the overall image dimension. These are then combined into a total loss function to train the task-by-task progressive remote sensing image thin cloud removal system, as shown below:

[0082] (4)

[0083] in, These are the loss functions constrained by the thin cloud removal subnetwork, the structural information complex atom network, and the high-frequency detail information complex atom network, respectively. The ground truth remote sensing images provided for the training dataset, It is an L1 norm.

[0084] Specifically, the loss functions for the three sub-networks mentioned above are as follows:

[0085] (5)

[0086] (6)

[0087] (7)

[0088] in, The LL subband after five-level wavelet decomposition. The LL subbands are the result of two-layer wavelet decomposition. The HH subband is obtained after wavelet decomposition of a single layer. Corresponding loss functions are established for each of the three subtasks to constrain their respective subtasks. Then, an overall high-fidelity restoration loss function is added to constrain the overall image dimension. These are combined as the total loss function to train the progressive remote sensing image thin cloud removal network. The trained network is then used to infer the final result of this method from remote sensing images with thin clouds.

[0089] Specifically, the following provides a further illustrative explanation of each sub-step.

[0090] First, 1) Global information guides the cloud removal sub-steps.

[0091] In this step, the Global Information Guided De-cloud sub-network GRNet is described in detail, and its network structure is as follows: Figure 2 As shown. Specifically, first, the remote sensing image containing thin clouds... The shallow feature vector is extracted from a convolutional layer, as shown below:

[0092] (8)

[0093] in, These are remote sensing images containing thin clouds and the extracted shallow feature vectors, with sizes of H×W × 3 and H×W × 64, respectively. conv1 is a 3×3 convolution operation, where H (Height) represents the image height (the number of pixels in the vertical direction), W (Width) represents the image width (the number of pixels in the horizontal direction), and the last number represents the number of image channels; that is, 3 input channels and 64 output channels.

[0094] Next, the extracted shallow feature vectors The vectors are respectively input to the downsampling layer and the global information extraction unit to perform low-resolution vector domain mapping and extract cloud removal guidance vectors, as shown below:

[0095] (9)

[0096] (10)

[0097] in, These are the first feature vector mapped to the low-resolution vector domain and the declouding guidance vector extracted by the global information extraction unit, respectively, both with a size of H / 8×W / 8×64. The function `nn.AvgPool2d`, with parameters `kernel_size=8`, `stride=8`, and `padding=0`, performs an 8x downsampling operation. `GIGB` is the Global Information Extraction Unit, and its structure information is as follows: Figure 5 As shown, the details will be elaborated later.

[0098] Then, the first feature vector mapped to the low-resolution vector domain and the declouding guidance vector extracted by the global information extraction unit are input into four consecutive declouding restoration modules to remove thin clouds from the feature vector containing thin clouds, as shown below:

[0099] (11)

[0100] (12)

[0101] (13)

[0102] (14)

[0103] in, These are the feature vectors after thin cloud removal output from four consecutive cloud removal and restoration modules, each with a size of H / 8 × W / 8 × 64. SCRB1, SCRB2, SCRB3, and SCRB4 are four consecutive cloud removal and restoration modules with the same network structure, such as... Figure 6 As shown, the details will be elaborated later.

[0104] For the feature vector after cloud removal The input is fed into the upsampling layer, which transforms it from a low-resolution vector domain into the original high-resolution vector domain, as shown below:

[0105] (15)

[0106] in, conv1 is an upsampling operation with an upsampling coefficient of 8, conv2 is a 3×3 convolution operation with 64 input channels and 64×8 output channels. The first high-dimensional feature vector of size H×W×64 after declouding is transformed to the high-resolution domain. It is output from the sub-network to the structural information complex atom network, providing basic support for the next stage of structural information complex atom steps.

[0107] Finally, the first high-dimensional feature vector of size H×W×64 after cloud removal is input into a convolutional layer and then combined with the input remote sensing image containing thin clouds to obtain the first result image of thin cloud removal output by this global information-guided cloud removal sub-network, as shown below:

[0108] (16)

[0109] in, These represent the first result image and the first high-dimensional feature vector of the cloud removal sub-step guided by global information, respectively. The input is a remote sensing image containing thin clouds. conv3 is a 3×3 convolution operation with 64 input channels and 3 output channels.

[0110] Next, the Global Information Extraction Unit (GIGB) described above will be elaborated in detail, and its corresponding structural information is as follows: Figure 5 As shown. Specifically, for the shallow feature vector input to this extraction unit... These are then fed into two average pooling layers with different parameters to extract guidance information at different scales, as shown below:

[0111] (17)

[0112] (18)

[0113] ATT1 and ATT2 are the guiding feature vectors extracted by two average pooling layers with different parameters, with sizes of H / 8×W / 8×64 and 1×1×64, respectively. and Both are functions of nn.AvgPool2d. The parameters used are kernel_size=8, stride=8, and padding=0. The parameters used are kernel_size=(H,W), stride=1, and padding=0.

[0114] After processing ATT1 and ATT2 using attention and residual mechanisms, the output of this Global Information Extraction Unit (GIGB) is shown below:

[0115] (19)

[0116] in, The cloud removal guidance vector extracted by the Global Information Extraction Unit (GIGB) has a size of H / 8×W / 8×64.

[0117] Next, the cloud recovery module SCRB described above will be explained in detail, and the corresponding structural information is as follows: Figure 6 As shown. A detailed explanation is given using SCRB1 as an example. Specifically, for the second feature vector input to this module, mapped to the low-resolution vector domain... Declouding guidance vector extracted by the global information extraction unit First of all Use after processing Preliminary guidance on moving away from the cloud is provided below:

[0118] (20)

[0119] in, The feature vectors are initially guided by cloud removal. LayerNorm is a layer normalization operation, and conv4 is a 3×3 convolution operation. Both the input and output channels are 64.

[0120] The initial cloud-guided feature vector The input is fed into a convolutional layer and activation function, which expands its dimensions to 128 and then performs vector segmentation along the channel dimension to lay the foundation for processing with different attention methods, as shown below:

[0121] (twenty one)

[0122] in, These are feature vectors after vector segmentation along the channel dimension, and their size is H / 8×W / 8×64. It's a vector splitting operation, dividing the vector into two vectors by taking half of each channel dimension. prelu is the torch.nn.PReLU activation function, and conv5 is a 3×3 convolution operation with 64 input channels and 128 output channels.

[0123] The feature vectors after feature segmentation are processed using attention and residual mechanisms, as shown below:

[0124] (twenty two)

[0125] (twenty three)

[0126] in, These are the feature vectors of some channels after processing by the attention mechanism and the residual mechanism, respectively. These are the second feature vector mapped to the low-resolution vector domain and the declouding guidance vector extracted by the global information extraction unit, respectively.

[0127] The processed feature vector After feature concatenation, the input is passed to a convolutional layer to obtain the feature vector after thin cloud removal output by the cloud removal and restoration module SCRB1, as shown below:

[0128] (twenty four)

[0129] in, The feature vector after thin cloud removal is output by the cloud removal and restoration module SCRB1. For feature concatenation operations performed along the channel dimension, conv6 is a 3×3 convolution operation with 128 input channels and 64 output channels, which is the feature vector after removing the thin clouds output by the cloud removal and restoration module SCRB1. By participating in the cloud removal and restoration module described above, the final feature vector can be obtained. .

[0130] 2) Structural Information Complex Atom Steps

[0131] In this step, the structural information complex atom network TRNett is described in detail, and its network structure is as follows: Figure 3 As shown. Specifically, the global information guides the output of the first high-dimensional feature vector in the cloud removal sub-step. Remote sensing images containing thin clouds After performing feature concatenation, the input is fed into a convolutional layer to fuse the first high-dimensional feature vector. The original remote sensing image information is shown below:

[0132] (25)

[0133] in, It is the feature vector after information fusion. For feature concatenation operations performed in the channel dimension, conv7 is a 3×3 convolution operation with 67 input channels and 64 output channels.

[0134] Feature vector after information fusion The input is fed to the downsampling layer to map it from the original high-resolution vector domain to the low-resolution vector domain for structural information restoration, as shown below:

[0135] (26)

[0136] in, The second feature vector is the low-resolution vector domain. AvgPool2d is the nn.AvgPool2d function with the parameters kernel_size=4, stride=4, and padding=0, which implements a 4x downsampling operation.

[0137] Next, the second eigenvector is mapped to the low-resolution vector domain. Four consecutive structural information restoration modules are input to perform iterative structural information restoration, as shown below:

[0138] (27)

[0139] in, The feature vectors are the structural information restored. STRB1, STRB2, STRB3, and STRB4 are four consecutive structural information restoration modules with the same network structure, such as... Figure 7 As shown, the details will be elaborated later.

[0140] Feature vectors after structural information restoration The input is fed into the upsampling layer, which transforms it from a low-resolution vector domain into the original high-resolution vector domain, as shown below:

[0141] (28)

[0142] in, The second high-dimensional feature vector, which is H×W×64 after structural information restoration in the high-resolution domain, is output from the sub-network to the high-frequency detail information complex atom network, providing basic support for the next stage of high-frequency detail information complex atom steps. conv8 is an upsampling operation with an upsampling coefficient of 4, and conv8 is a 3×3 convolution operation with 64 input channels and 64×8 output channels.

[0143] Finally, the second high-dimensional feature vector after restoring the structural information The first image output after the input is fed into a convolutional layer and global information is added to guide the cloud removal sub-step The remote sensing image obtained is the structure information restored from the output of the structural information complex atom network, i.e., the second result image. As shown below:

[0144] (29)

[0145] in, , These are the second result image of the complex atom step of structural information and the second high-dimensional feature vector output by the complex atom step of structural information, respectively. The first result image of the cloud removal subtask is shown. conv9 is a 3×3 convolution operation with 64 input channels and 3 output channels.

[0146] Next, the STRABS module for restoring structural information described above will be explained in detail. The corresponding structural information is as follows: Figure 7 As shown. Next, we will use STRB1 as an example to explain in detail. Specifically, for the second feature vector input to this module... After layer normalization, the input is fed into consecutive deep convolutional layers and pointwise convolutional layers. Then, residual operations are used to output the feature vector restored from the preliminary structural information, as shown below:

[0147] (30)

[0148] in, , These are the second feature vector input to this module and the feature vector for preliminary structural information reconstruction, respectively. LayerNorm is a layer normalization operation. pconv1 and dconv1 are a 3×3 depthwise convolutional layer and a pointwise convolutional operation, respectively, with 64 input and 64 output channels.

[0149] Feature vectors reconstructed from preliminary structural information After layer normalization, the input is fed into the subsequent depthwise convolutional layers and pointwise convolutional layers, as shown below:

[0150] (31)

[0151] in, These are feature vectors for further structural information reconstruction, and LayerNorm is a layer normalization operation. pconv2 and dconv2 are 3×3 depthwise convolutional layers and pointwise convolutional operations, respectively, with 64 input and 64 output channels.

[0152] Feature vectors for further structural information reconstruction using attention and residual mechanisms The final processing step is to output the feature vector for restoring the structural information of this module, as shown below:

[0153] (32)

[0154] in, This is the feature vector for restoring the structural information output by this module, and sigmoid is the torch.nn.Sigmod activation function.

[0155] 3) High-frequency detail information complex atom steps;

[0156] In this step, the High Frequency Detail Information Complex Atom Network (HRNet) is described in detail, and its network structure is as follows: Figure 4 As shown. Specifically, the second high-dimensional feature vector output by the complex atom step of the structural information is... Remote sensing images containing thin clouds After feature concatenation, the input is fed into a convolutional layer to fuse the second high-dimensional feature vector. The original remote sensing image information containing thin clouds is shown below:

[0157] (33)

[0158] in, It is the third feature vector obtained by fusing the structural information restoration feature vector and the original remote sensing image. For feature concatenation operations performed in the channel dimension, conv10 is a 3×3 convolution operation with 67 input channels and 64 output channels.

[0159] Next, the third feature vector after information fusion is... The high-frequency detail information is restored by inputting four consecutive high-frequency detail information restoration modules, as shown below:

[0160] (34)

[0161] in, The final feature vector after high-frequency detail information restoration is represented by HTRB1, HTRB2, HTRB3, and HTRB4, which are four consecutive high-frequency detail information restoration modules with identical network structures, such as... Figure 8 As shown, the details will be elaborated later.

[0162] Finally, the high-dimensional feature vector after high-frequency detail information restoration is input into a convolutional layer, and the resulting image after the structural information complex atom step is added to obtain the final thin cloud removal remote sensing image output by the high-frequency detail information complex atom network, as shown below:

[0163] (35)

[0164] in, , These are the final output images of the thin cloud removal remote sensing image and the second result image output by the structural information complex atom step, respectively. conv11 is a 3×3 convolution operation with 64 input channels and 3 output channels.

[0165] Next, the High Frequency Detail Information Reconstruction (HTRB) module described above will be explained in detail, and the corresponding structural information is as follows: Figure 8As shown. Next, we will use HTRB1 as an example to explain in detail. Specifically, for the third feature vector input to this module... After layer normalization, a Fast Fourier Transform is performed to separate the amplitude and phase vectors, laying the foundation for subsequent high-frequency information restoration in the frequency domain, as shown below:

[0166] (36)

[0167] in, , The amplitude and phase vectors are separated by Fast Fourier Transform, LayerNorm is the layer normalization operation, and FFT is the Fast Fourier Transform operation.

[0168] A series of deep convolutional layers, pointwise convolutional layers, activation functions, and convolutional layer operations are performed on the amplitude and phase vectors respectively to restore high-frequency information in the frequency domain, as shown below:

[0169] (37)

[0170] (38)

[0171] in, , These are the amplitude and phase vectors after high-frequency detail information restoration, respectively; conv12 and conv13 are both 3×3 convolution operations with 64 input channels and 64 output channels; prelu is the torch.nn.PReLU activation function; pconv3 and pconv5 are both 3×3 pointwise convolution operations with 64 input channels and 64 output channels; dconv4 and dconv6 are both 3×3 depthwise convolution operations with 64 input channels and 64 output channels.

[0172] The amplitude vector after restoring high-frequency detail information Phase part vector Perform an inverse fast Fourier transform to transform the feature vector, after restoring the high-frequency information, from the frequency domain to the image domain, as shown below:

[0173] (39)

[0174] in, The IFFT is the feature vector of the image domain after high-frequency information restoration, and it is the inverse fast Fourier transform operation.

[0175] The feature vectors of the image domain after high-frequency information restoration are processed using attention mechanisms and residual connections. The final step is to output the feature vector after restoring the high-frequency information of this module, as shown below:

[0176] (40)

[0177] in, The output of this module is the feature vector after high-frequency information restoration. The output of the last high-frequency detail information restoration module is the final feature vector after high-frequency information restoration. .

[0178] This invention decomposes the complex problem of thin cloud removal into three sub-steps, using global information to guide the cloud removal sub-network, structural information restoration sub-network, and high-frequency detail information restoration sub-network to solve the problem task by task. By progressively completing the thin cloud removal, structural information restoration, and high-frequency detail information restoration steps, it achieves the effective restoration of low-frequency structural information and high-frequency detail information in remote sensing images while removing thin clouds, thus solving the complex problem of thin cloud removal in remote sensing images with high fidelity.

Claims

1. A method for removing thin clouds from remote sensing images in a task-by-task progressive manner, characterized in that, Includes the following steps, 1) The global information-guided cloud removal sub-step includes extracting the shallow feature vector of the input remote sensing image containing thin clouds, then downsampling and reducing its dimensionality to obtain the first feature vector, obtaining the cloud removal guidance vector from the shallow feature vector, and inputting the cloud removal guidance vector and the first feature vector together into multiple consecutive cloud removal and restoration modules to remove the thin clouds, and outputting the first high-dimensional feature vector and the first result image after its fusion with the remote sensing image containing thin clouds; 2) The structural information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, extracting features through a convolutional layer and downsampling to reduce the dimensionality to obtain the second feature vector, which is then input into multiple structural information restoration modules to iteratively repair the structure, and outputting the second high-dimensional feature vector and the second result image after fusing it with the first result image; 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the second high-dimensional feature vector, extracting the third feature vector through a convolutional layer, and then inputting it into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The output feature vector after the high-frequency detail information restoration is optimized by a convolutional layer and then fused with the second result image to finally output the remote sensing image with thin clouds removed.

2. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as described in claim 1, characterized in that, The method for obtaining the cloud removal guidance vector is as follows: The shallow feature vectors are input into two average pooling layers with different parameters to extract guidance information at different scales. The cloud removal guidance vector is obtained by processing guidance information at two different scales using attention and residual mechanisms.

3. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as described in claim 2, characterized in that, The processing method of the cloud restoration module is as follows: The initial cloud removal guidance feature vector is obtained by convolving the first input feature vector with the cloud removal guidance vector. After expanding the input dimension of the initial cloud-guided feature vector, vector segmentation is performed in the channel dimension to obtain two feature vectors. The two feature vectors after feature segmentation are processed by attention and residual mechanisms with the cloud removal guidance vector and the first input feature vector, respectively. After feature concatenation, the concatenated feature vectors are input into the convolutional layer to obtain the feature vector after thin cloud removal output by the cloud removal restoration module. The first input feature vector is either the first feature vector or the feature vector output by the previous cloud removal restoration module. The output feature vector of the last cloud removal restoration module is convolved and upsampled to obtain the first high-dimensional feature vector.

4. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as described in claim 1, characterized in that, The processing method of the structural information restoration module is as follows: After layer normalization, the second input feature vector is input into a series of deep convolutional layers and pointwise convolutional layers, and then the feature vector is restored by using residual operation to output the preliminary structural information. After layer normalization, the feature vectors of the preliminary structural information reconstruction are input into continuous deep convolutional layers and pointwise convolutional layers to obtain feature vectors of further structural information reconstruction. Attention and residual mechanisms are used to process the feature vectors for further structural information restoration to output the structural information restoration feature vector; wherein, the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module; The output of the final-level structural information restoration module is convolved and upsampled to obtain the second high-dimensional feature vector.

5. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as described in claim 1, characterized in that, The processing method of the high-frequency detail restoration module is as follows: After normalizing the third input feature vector layer, a fast Fourier transform is performed to separate the amplitude component vector and the phase component vector; The amplitude vector and phase vector are subjected to deep convolutional layer, pointwise convolutional layer, activation function, and convolutional layer operations respectively to restore high-frequency information in the frequency domain; Perform an inverse fast Fourier transform on the amplitude and phase vectors after restoring the high-frequency detail information to transform the feature vectors after restoring the high-frequency information from the frequency domain to the image domain; The feature vector is obtained by using the attention mechanism and residual connection to the feature vector of the image domain after high-frequency information restoration. The third input feature vector is the third feature vector or the feature vector output by the previous high-frequency detail restoration module. The final high-frequency detail restoration module outputs a third feature vector after convolution and upsampling.

6. A task-by-task progressive system for removing thin clouds from remote sensing images, characterized in that: include, 1) A global information-guided declouding subnetwork is used to input a remote sensing image containing thin clouds and extract shallow feature vectors. Then, it is downsampled and dimensionality reduced to obtain the first feature vector. The declouding guidance vector is obtained from the shallow feature vector. The declouding guidance vector and the first feature vector are input together into multiple consecutive declouding restoration modules to remove thin clouds. The output is the first high-dimensional feature vector and the first result image after it is fused with the remote sensing image containing thin clouds. 2) A structural information complex atom network is used to input a remote sensing image containing thin clouds and the first high-dimensional feature vector, and then perform convolutional layer to extract features and downsample to reduce the dimensionality to obtain a second feature vector. This second feature vector is then input to multiple structural information restoration modules to iteratively repair the structure and output the second high-dimensional feature vector and the second result image after it is fused with the first result image. 3) A high-frequency detail information complex atom network is used to input a remote sensing image containing thin clouds and the second high-dimensional feature vector. The features are then concatenated and fused, and the third feature vector is extracted through a convolutional layer. This third feature vector is then input to multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final high-frequency detail information restored feature vector is then optimized by a convolutional layer and fused with the second result image to finally output a remote sensing image with thin clouds removed.

7. The task-by-task progressive remote sensing image thin cloud removal system as described in claim 6, characterized in that, We establish corresponding loss functions for the three sub-networks, and then add an overall high-fidelity restoration loss function to constrain the overall image dimension. We then combine them into a total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

Images