Task-by-task progressive remote sensing image thin cloud removal method and system

Abstract

The invention discloses a task-by-task progressive remote sensing image thin cloud removal method and system, and the method comprises the following steps: a bureau information guidance cloud removal sub-step, a structural information restoration sub-step, and a high-frequency detail information restoration sub-step. According to the method, a complex thin cloud removal problem is decomposed into three sub-steps to guide a cloud removal sub-network, a structure information recovery sub-network and a high-frequency detail information recovery sub-network to solve task by task by using global information, and the sub-steps of thin cloud removal, structure information recovery and high-frequency detail information recovery are completed step by step in a progressive mode. The low-frequency structure information and the high-frequency detail information in the remote sensing image are effectively restored while the thin cloud in the remote sensing image is removed, so that the problem of complex thin cloud removal of the remote sensing image is solved in a high-fidelity manner.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for removing thin clouds from remote sensing images in a task-by-task progressive manner. Background Art

[0002] As a mainstream method for displaying Earth observation results, remote sensing images have garnered widespread attention across various fields. However, the acquisition process of remote sensing images is subject to a variety of complex environments, such as rain, clouds, and fog. Among these, thin cloud occlusion is particularly common during remote sensing image acquisition, causing ground objects to be obscured by thin clouds, seriously impacting subsequent applications such as interpretation, detection, and segmentation. Therefore, the problem of thin cloud removal in remote sensing images is of considerable research value. Currently, most of the technical approaches adopted for thin cloud removal in remote sensing images are based on deep learning methods, typically using convolutional neural networks or Transformer network architectures to complete the task of thin cloud removal end-to-end within a single network structure, achieving good results.

[0003] For example, Chinese patent CN118314052A discloses a method for removing thin clouds from remote sensing images. The method includes: dividing an image to be processed into blocks to obtain multiple first sub-images; acquiring the first sub-images in a predetermined order; removing thin clouds from the first sub-images using a trained thinning cloud removal model to obtain a thin cloud-free sub-image; performing grayscale transformation on the thin cloud-free sub-image to obtain a second sub-image; and concatenating all the second sub-images to obtain a thin cloud-free image of the image to be processed. Specifically, the method removes thin clouds from a thin cloud-containing image using a thinning cloud removal model based on conditional variational autoencoding.

[0004] However, using only one network structure to solve the complex thin cloud removal problem is inherently difficult to simultaneously obtain good thin cloud removal, structural information restoration, and high-frequency detail restoration results, and to perform high-fidelity restoration of remote sensing images with thin clouds. Summary of the Invention

[0005] The purpose of the present invention is to provide a task-by-task progressive thin cloud removal method for remote sensing images, which can effectively restore the low-frequency structural information and high-frequency detail information in the remote sensing images while removing the thin clouds in the remote sensing images, thereby solving the complex problem of thin cloud removal in remote sensing images with high fidelity.

[0006] At the same time, the purpose of the present invention is to provide a system for thinning remote sensing images in a task-by-task progressive manner.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is: The method for removing thin clouds from remote sensing images in a task-by-task progressive manner includes the following steps: 1) A global information-guided cloud removal sub-step, comprising extracting shallow feature vectors from the input remote sensing image containing thin clouds, then downsampling and reducing the shallow feature vectors to obtain a first feature vector, obtaining a cloud removal guidance vector from the shallow feature vector, and inputting the cloud removal guidance vector and the first feature vector together into multiple consecutive cloud removal and restoration modules to remove thin clouds. The first high-dimensional feature vector and a first result image fused with the remote sensing image containing thin clouds are then output; 2) The structural information restoration step involves concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, performing convolutional layer feature extraction, and downsampling to reduce the dimensionality to obtain a second feature vector. This is then fed into multiple structural information restoration modules to iteratively repair the structure, outputting the second high-dimensional feature vector and a second result image fused with the first result image. 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the second high-dimensional feature vector, extracting the third feature vector through a convolutional layer, and then inputting it into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final feature vector after high-frequency detail information restoration is output and optimized by the convolutional layer. It is fused with the second result image to finally output the remote sensing image without thin clouds.

[0008] As one of the preferred solutions, the method for obtaining the cloud removal guidance vector is: The shallow feature vectors are respectively input into two average pooling layers with different parameters to extract guidance information of different scales; The de-clouding guidance vector is obtained by processing the guidance information of two different scales using the attention mechanism and the residual mechanism.

[0009] As one of the preferred solutions, the processing method of the cloud removal and restoration module is as follows: Performing feature convolution on the first input feature vector and the de-clouding guidance vector to obtain a preliminary de-clouding guidance feature vector; After expanding the input dimension of the feature vector for preliminary de-clouding guidance, perform vector segmentation in the channel dimension to obtain two feature vectors; The two feature vectors after feature segmentation are processed with the declouding guidance vector and the first input feature vector using the attention mechanism and the residual mechanism, and then feature splicing is performed and input into the convolution layer to obtain the feature vector after thin cloud removal output by the declouding and restoration module; wherein the first input feature vector is the first feature vector or the feature vector output by the previous declouding and restoration module, and the output feature vector of the last declouding and restoration module is convolved and up-sampled to obtain the first high-dimensional feature vector.

[0010] As one of the preferred solutions, the processing method of the structure information recovery module is: The second input feature vector is normalized and then input into the continuous depth convolution layer and point-by-point convolution layer, and then the residual operation is used to output the feature vector of the preliminary structural information restoration; The feature vectors of the initial structural information restoration are normalized and then input into the continuous depth convolution layer and point-by-point convolution layer to obtain the feature vectors of further structural information restoration; The feature vector of further structural information restoration is processed using the attention mechanism and the residual mechanism to output a feature vector of structural information restoration; wherein the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module; The output of the last-level structural information restoration module is convolved and up-sampled to obtain the second high-dimensional feature vector.

[0011] As one of the preferred solutions, the processing method of the high-frequency detail restoration module is: After normalizing the third input feature vector layer, perform fast Fourier transform to separate the amplitude part vector and the phase part vector; Performing depth convolution layer, point-by-point convolution layer, activation function, and convolution layer operations on the amplitude part vector and the phase part vector respectively to restore high-frequency information in the frequency domain; Performing an inverse fast Fourier transform on the amplitude component vector and the phase component vector after the high-frequency detail information is restored to convert the feature vector after the high-frequency information is restored from the frequency domain to the image domain; Using the attention mechanism and residual connection to restore the high-frequency information in the image domain, the feature vector is obtained, wherein the third input feature vector is the third feature vector or the feature vector output by the previous high-frequency detail restoration module; The output of the last-level high-frequency detail restoration module is convolved and up-sampled to obtain the third eigenvector.

[0012] The system for removing thin clouds from remote sensing images based on a task-by-task progressive approach includes: 1) A global information-guided cloud removal sub-network is used to input a remote sensing image containing thin clouds and extract shallow feature vectors. This is then downsampled and dimensionally reduced to obtain a first feature vector. A cloud removal guidance vector is derived from the shallow feature vector. This guidance vector and the first feature vector are then fed into multiple consecutive cloud removal and restoration modules to remove thin clouds. The network then outputs a first high-dimensional feature vector and a first result image fused with the remote sensing image containing thin clouds. 2) A structural information restoration atomic network, which takes as input the concatenated features of the remote sensing image containing thin clouds and the first high-dimensional feature vector, performs convolutional layer feature extraction and downsampling to reduce the dimensionality to obtain a second feature vector. This is then fed into multiple structural information restoration modules to iteratively repair the structure, and outputs a second high-dimensional feature vector and a second result image fused with the first result image. 3) High-frequency detail information restoration sub-network, which is used to input the concatenated fusion features of the remote sensing image containing thin clouds and the second high-dimensional feature vector and extract the third feature vector through the convolution layer. The third feature vector is then input into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final feature vector after high-frequency detail information restoration is output and optimized by the convolution layer. It is fused with the second result image to finally output the remote sensing image without thin clouds.

[0013] As one of the preferred solutions, corresponding loss functions are established for the three sub-networks respectively, and then the overall high-fidelity restoration loss function is added to constrain the overall dimension of the image. They are combined together as the total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention decomposes the complex thin cloud removal problem into three sub-steps, corresponding to the use of global information to guide the cloud removal sub-network, the structural information complex atom network, and the high-frequency detail information complex atom network, thereby achieving a task-by-task and step-by-step solution. The global information-guided cloud removal sub-step, the structural information complex atom step, and the high-frequency detail information complex atom step are gradually completed in a progressive manner, thereby achieving the effective restoration of low-frequency structural information and high-frequency detail information in the remote sensing image while removing thin clouds in the remote sensing image, thereby solving the complex thin cloud removal problem of remote sensing images with high fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a structural diagram of the task-by-task progressive remote sensing image thin cloud system of the present invention.

[0016] Figure 2 Schematic diagram of the cloud sub-network for global information guidance.

[0017] Figure 3 Schematic diagram of the structural information complex atomic network.

[0018] Figure 4 Schematic diagram of the complex atomic network for high-frequency detail information.

[0019] Figure 5 Schematic diagram of the global information extraction unit structure.

[0020] Figure 6 This is a schematic diagram of the cloud recovery module structure.

[0021] Figure 7 Schematic diagram of the structure information recovery module.

[0022] Figure 8 Schematic diagram of the high-frequency detail information restoration module structure.

[0023] Figures 9-11The figure is a comparison of a remote sensing image containing thin clouds and a remote sensing image after the thin clouds are removed using the method of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0025] The method for removing thin clouds from remote sensing images in a task-by-task progressive manner according to the present invention comprises the following steps: 1) A global information-guided cloud removal sub-step, comprising convolving the input remote sensing image containing thin clouds to extract a shallow feature vector, downsampling and reducing the dimension of the shallow feature vector to obtain a first feature vector, inputting the shallow feature vector into a global information extraction unit to obtain a cloud removal guidance vector, and then inputting the shallow feature vector and the first feature vector together into a plurality of generally 3-6, preferably 4, consecutive cloud removal and restoration modules to remove thin clouds, outputting a first high-dimensional feature vector and a first result image after fusion of the first feature vector and the remote sensing image containing thin clouds; wherein the cloud removal guidance vector is obtained by: The shallow feature vectors are respectively input into two average pooling layers with different parameters to extract guidance information of different scales; The declouding guidance vector is obtained by processing guidance information at two different scales using an attention mechanism and a residual mechanism. The first average pooling layer uses a global pooling strategy to obtain contextual information for the entire image, while the second average pooling layer uses a regional pooling strategy to obtain local structural features, forming scale-differentiated guidance information. A dual-pooling structure with differentiated parameters is used to achieve parallel extraction of multi-scale features. The attention mechanism dynamically fuses global and local semantic information, and residual connections are introduced to ensure lossless transmission of the original feature information. This method can significantly improve the ability to perceive cloud-occluded areas of different scales in remote sensing image declouding processing and enhance the model's adaptability to complex scenes.

[0026] The processing method of the cloud removal and restoration module is as follows: Convolving the first input feature vector with the de-clouding guidance vector to obtain a feature vector for preliminary de-clouding guidance; After expanding the input dimension of the feature vector for preliminary de-clouding guidance, perform vector segmentation in the channel dimension to obtain two feature vectors; The two feature vectors after feature segmentation are processed with the declouding guidance vector and the first input feature vector using the attention mechanism and residual mechanism, and then feature concatenation is performed before inputting into the convolution layer to obtain the feature vector after thin cloud removal output by the declouding and restoration module. The input feature vector is the first feature vector or the feature vector output by the previous declouding and restoration module, or the first high-dimensional feature vector output by the last declouding and restoration module. That is, the declouding and restoration modules other than the first one use the feature vector mapped to the low-resolution vector domain to participate in the convolution.

[0027] 2) The structural information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, performing convolutional layer feature extraction and downsampling to reduce the dimensionality to obtain a second feature vector, and then inputting multiple, generally 3-6, preferably 4, structural information restoration modules to iteratively repair the structure, outputting the second high-dimensional feature vector and a second result image after fusing it with the first result image; the processing method of the structural information restoration module is as follows: The second input feature vector is normalized and then input into the continuous depth convolution layer and point-by-point convolution layer, and then the residual operation is used to output the feature vector of the preliminary structural information restoration; The feature vectors of the initial structural information restoration are normalized and then input into the continuous depth convolution layer and point-by-point convolution layer to obtain the feature vectors of further structural information restoration; The feature vector of further structural information restoration is processed using the attention mechanism and the residual mechanism to output the feature vector of structural information restoration of this module; wherein the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module; The feature vector output by the last-level structural information restoration module is the second-highest-dimensional feature vector.

[0028] 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the second high-dimensional feature vector, extracting basic features through a convolution layer to obtain a third feature vector, iteratively restoring the high-frequency details of the image through multiple high-frequency detail restoration modules (generally 3-6, preferably 4), and then fusing the output feature vector with the second result image after feature optimization through a convolution layer to finally output a remote sensing image with thin clouds removed. The processing method of the high-frequency detail restoration module is as follows: After normalizing the third input feature vector layer, perform fast Fourier transform to separate the amplitude part vector and the phase part vector; Performing depth convolution layer, point-by-point convolution layer, activation function, and convolution layer operations on the amplitude part vector and the phase part vector respectively to restore high-frequency information in the frequency domain; Performing an inverse fast Fourier transform on the amplitude component vector and the phase component vector after the high-frequency detail information is restored to convert the feature vector after the high-frequency information is restored from the frequency domain to the image domain; The feature vector of the image domain after high-frequency information restoration is obtained using the attention mechanism and residual connection. The third input feature vector is the third feature vector or the feature vector output by the previous level high-frequency detail restoration module. The third feature vector is obtained after convolution and upsampling of the output of the last level high-frequency detail restoration module.

[0029] At the same time, the present invention also discloses a task-by-task progressive remote sensing image thin cloud removal system, comprising: A global information-guided cloud removal sub-network is used to perform convolution on the input remote sensing image containing thin clouds to extract shallow feature vectors, downsample and reduce the dimensionality to obtain a first feature vector, obtain a cloud removal guidance vector and input it and the first feature vector into multiple consecutive cloud removal and restoration modules to remove thin clouds, and output a first high-dimensional feature vector and a first result image after the first high-dimensional feature vector is fused with the remote sensing image containing thin clouds; A structural information restoration atomic network is used to concatenate and fuse the remote sensing image containing thin clouds with the first high-dimensional feature vector, perform feature extraction using a convolution layer, and perform dimensionality reduction using a downsampling layer to obtain a second feature vector. This second feature vector is then sequentially input into multiple structural information restoration modules to iteratively repair the structure, outputting the second high-dimensional feature vector and a second result image fused with the first result image. The high-frequency detail information restoration atomic network is used to splice and fuse the remote sensing image containing thin clouds with the second high-dimensional feature vector, extract the basic features through the convolution layer to obtain the third feature vector, and iteratively restore the high-frequency details of the image through multiple high-frequency detail restoration modules; the obtained feature vector is optimized by the convolution layer and fused with the second result image to finally output the remote sensing image without thin clouds.

[0030] Corresponding loss functions are established for the three sub-networks to constrain the corresponding sub-networks, and then the overall high-fidelity restoration loss function is added to constrain the overall dimension of the image. They are combined together as the total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

[0031] The present invention accepts a remote sensing image containing thin clouds as input and utilizes a global information-driven cloud removal algorithm to perform preliminary detection and restoration of thin cloud areas, aiming to remove cloud occlusion while preserving the underlying content. The structural restoration sub-step focuses on restoring the image's structural integrity (e.g., outlines and main framework). Using a feature reconstruction algorithm, it restores structural information and outputs a structurally repaired image, achieving improved structural accuracy and a clearer underlying outline. The frequency detail restoration sub-step receives the structurally repaired image and simultaneously associates high-frequency features of the original remote sensing image containing thin clouds for detail verification. It then performs enhanced restoration of high-frequency details such as texture and edges, ultimately outputting a complete remote sensing image with the thin clouds removed.

[0032] The present invention will be further described below through specific examples.

[0033] The remote sensing image declouding method based on the task-by-task progressive restoration network of the present invention includes three sub-steps (sub-networks). The specific method flow is as follows: Figure 1 The following describes the process. First, the global information-guided cloud removal sub-step is performed on the remote sensing image containing thin clouds, laying the foundation for the subsequent two sub-steps (subtasks), as shown below: (1) in, are the first result image and the first high-dimensional feature vector output by the global information-guided declouding sub-step, The input remote sensing image contains thin clouds, GRNet is the proposed global information guided de-clouding sub-network, and its network structure is as follows: Figure 2 The details will be described in the following steps.

[0034] Then, the global information is used to guide the first result image output by the de-clouding sub-step , the first high-dimensional eigenvector , remote sensing images containing thin clouds Input to the structure information recovery sub-network, and restore the structure information based on the global information guiding the de-clouding sub-network to complete the de-clouding task, as shown below: (2) in, are the second result image and the second high-dimensional feature vector output by the structural information complex atom step, respectively. TRNet is the proposed structural information complex atom network, and its network structure is as follows: Figure 3 The details will be described in the following steps.

[0035] Finally, the second result image output by the structural information atomization step is , the second highest dimensional feature vector , remote sensing images containing thin clouds Input to the high-frequency detail information complex atom network and perform the high-frequency detail information complex atom step as shown below: (3) in, To finally output the remote sensing image with thin clouds removed, HRNet is a high-frequency detail information complex atomic network proposed, and its network structure is as follows: Figure 4 The details will be described in the following steps.

[0036] The above process illustrates the network structure of the remote sensing image declouding method based on the task-by-task progressive restoration network. Furthermore, corresponding loss functions are established for each of the three sub-networks to constrain the corresponding sub-steps. The overall high-fidelity restoration loss function is then added to constrain the overall image dimension. These are combined together as the overall loss function to train the task-by-task progressive remote sensing image thin cloud removal system, as shown below: (4) in, They are the loss functions constrained by the thin cloud removal sub-network, the structural information complex atom network, and the high-frequency detail information complex atom network, The ground truth remote sensing images provided for the training dataset, is the L1 norm.

[0037] Specifically, the loss function for the three sub-networks mentioned above is as follows: (5) (6) (7) in, is the LL subband after the five-layer wavelet decomposition, is the LL subband after the two-layer wavelet decomposition, The HH subbands of the wavelet decomposition of a layer are constructed. Corresponding loss functions are established for each of the three subtasks to constrain the corresponding subtasks. A loss function for overall high-fidelity restoration is then added to constrain the overall image dimension. These functions are combined as the total loss function to train the task-by-task progressive remote sensing image thin cloud removal network. The trained task-by-task progressive remote sensing image thin cloud removal network is then used to infer remote sensing images with thin clouds to obtain the final result of this method.

[0038] Specifically, each sub-step is further explained below.

[0039] First, 1) global information guides the de-clouding sub-step, In this step, the global information guided de-clouding sub-network GRNet is described in detail. Its network structure is as follows: Figure 2 Specifically, firstly, the remote sensing image containing thin clouds Input into a convolutional layer to extract shallow feature vectors, as shown below: (8) in, The following are remote sensing images containing thin clouds and the extracted shallow feature vectors, with sizes of H × W × 3 and H × W × 64, respectively. conv1 is a 3 × 3 convolution operation, where H (Height) represents the image height, i.e., the number of pixels in the vertical direction, and W (Width) represents the image width, i.e., the number of pixels in the horizontal direction. The last number represents the number of channels in the image, i.e., 3 input channels and 64 output channels.

[0040] Next, the extracted shallow feature vector The data is input to the downsampling layer and the global information extraction unit respectively to map the low-resolution vector domain and extract the cloud removal guidance vector, as shown below: (9) (10) in, They are respectively the first eigenvector mapped to the low-resolution vector domain and the de-clouding guidance vector extracted by the global information extraction unit, and their sizes are both H / 8×W / 8×64. The corresponding parameters are kernel_size=8, stride=8, padding=0, which realizes 8-fold downsampling. GIGB is the global information extraction unit, and the corresponding structural information is as follows: Figure 5 The details will be described later.

[0041] Then, the first eigenvector mapped to the low-resolution vector domain and the cloud removal guidance vector extracted by the global information extraction unit are input into four consecutive cloud removal and restoration modules to remove thin clouds from the eigenvectors containing thin clouds, as shown below: (11) (12) (13) (14) in, They are the feature vectors after thin cloud removal output by four consecutive cloud removal and restoration modules, and their sizes are all H / 8 × W / 8 × 64. SCRB1, SCRB2, SCRB3, and SCRB4 are four consecutive cloud removal and restoration modules, and their network structures are the same, as shown in Figure 6 The details will be described later.

[0042] The eigenvector after declouding Input to the upsampling layer, which converts it from the low-resolution vector domain to the original high-resolution vector domain, as shown below: (15) in, It is an upsampling operation with an upsampling factor of 8, conv2 is a 3×3 convolution operation, the input channel is 64, and the output channel is 64×8. It is the first high-dimensional feature vector of size H×W×64 after declouding that is converted to the high-resolution domain. It is output from this sub-network to the structure information complex atom network, providing basic support for the structure information complex atom step in the next stage.

[0043] Finally, the first high-dimensional feature vector of size H×W×64 after declouding is input into a convolutional layer and then added to the input remote sensing image containing thin clouds. The first result image of thin cloud removal output by the declouding sub-network guided by this global information is obtained, as shown below: (16) in, are the first result image and the first high-dimensional feature vector of the de-clouding sub-step guided by global information, The input remote sensing image contains thin clouds, conv3 is a 3×3 convolution operation, the input channel is 64, and the output channel is 3.

[0044] Next, the global information extraction unit GIGB described above is elaborated in detail. The corresponding structural information is as follows: Figure 5 Specifically, for the shallow feature vector input to this extraction unit , which are input into two average pooling layers with different parameters to extract guidance information of different scales, as shown below: (17) (18) Among them, ATT1 and ATT2 are the guiding feature vectors extracted by the average pooling layer with two different parameters, and their sizes are H / 8×W / 8×64 and 1×1×64 respectively. and Both are nn.AvgPool2d functions. The parameters used are kernel_size=8, stride=8, padding=0, The parameters used are kernel_size=(H,W), stride=1, padding=0.

[0045] The attention mechanism and residual mechanism are used to process ATT1 and ATT2 as the output of the global information extraction unit GIGB, as shown below: (19) in, It is the de-clouding guidance vector extracted by the global information extraction unit GIGB, and its size is H / 8×W / 8×64.

[0046] Next, the above-described cloud recovery module SCRB is described in detail. The corresponding structural information is as follows: Figure 6 As shown. SCRB1 is used as an example to explain in detail. Specifically, for the second eigenvector mapped to the low-resolution vector domain input to this module, , the cloud removal guidance vector extracted by the global information extraction unit , first of all Use after processing Conduct preliminary cloud migration guidance as follows: (20) in, is the feature vector for preliminary cloud removal guidance, LayerNorm is the layer normalization operation, conv4 is a 3×3 convolution operation, and both the input and output channels are 64.

[0047] The feature vector of the initial cloud removal guidance The input is fed into a convolutional layer and activation function, and its dimension is expanded to 128 dimensions. After that, the vector is split in the channel dimension to lay the foundation for processing using different attentions, as shown below: (twenty one) in, It is the feature vector after vector segmentation in the channel dimension, and its size is H / 8×W / 8×64. This is a vector split operation, splitting the vector into two halves along the channel dimension. prelu is the torch.nn.PReLU activation function, conv5 is a 3×3 convolution operation, with 64 input channels and 128 output channels.

[0048] The feature vectors after feature segmentation are processed using the attention mechanism and the residual mechanism respectively, as shown below: (twenty two) (twenty three) in, are the feature vectors of some channels processed by the attention mechanism and the residual mechanism respectively. They are respectively the second eigenvector mapped to the low-resolution vector domain and the de-clouding guidance vector extracted by the global information extraction unit.

[0049] After processing, the feature vector After feature concatenation, the convolutional layer is input to obtain the feature vector after thin cloud removal output by the cloud removal and restoration module SCRB1, as shown below: (twenty four) in, is the feature vector after thin clouds are removed output by the cloud removal and restoration module SCRB1, It is a feature concatenation operation performed in the channel dimension. Conv6 is a 3×3 convolution operation with 128 input channels and 64 output channels. The feature vector after removing the thin clouds output by the cloud removal and restoration module SCRB1 is Participate in the above de-clouding and restoration module to calculate and get the final feature vector .

[0050] 2) Structural information recovery step In this step, the structural information complex atomic network TRNett is described in detail. Its network structure is as follows: Figure 3 Specifically, the global information is used to guide the first high-dimensional feature vector output by the de-clouding sub-step , remote sensing images containing thin clouds After the feature splicing operation, a convolution layer is input to fuse the first high-dimensional feature vector And the original remote sensing image information is as follows: (25) in, is the feature vector after information fusion, It is a feature concatenation operation performed in the channel dimension. conv7 is a 3×3 convolution operation with 67 input channels and 64 output channels.

[0051] The feature vector after information fusion The input is sent to the downsampling layer to map it from the original high-resolution vector domain to the low-resolution vector domain for structural information restoration, as shown below: (26) in, is the second eigenvector of the low-resolution vector domain, AvgPool2d is the nn.AvgPool2d function, and the corresponding parameters are kernel_size=4, stride=4, and padding=0, which implements a 4x downsampling operation.

[0052] Then the second eigenvector mapped to the low-resolution vector domain is Input four consecutive structural information recovery modules to perform iterative structural information recovery, as shown below: (27) in, is the feature vector after structural information restoration, STRB1, STRB2, STRB3, and STRB4 are four consecutive structural information restoration modules, and their network structures are the same, such as Figure 7 The details will be described later.

[0053] The eigenvector after restoring the structural information Input to the upsampling layer, which converts it from the low-resolution vector domain to the original high-resolution vector domain, as shown below: (28) in, It is the second highest dimensional feature vector of size H×W×64 after the structural information is restored after being converted to the high-resolution domain. It is output from this sub-network to the high-frequency detail information complex atom network, providing basic support for the high-frequency detail information complex atom step in the next stage. It is an upsampling operation with an upsampling factor of 4, conv8 is a 3×3 convolution operation, the input channel is 64, and the output channel is 64×8.

[0054] Finally, the second highest dimensional feature vector after structural information restoration The first result image output by the de-clouding sub-step after inputting it into a convolutional layer and adding global information to guide it , and obtain the remote sensing image of the structure information restored by the structure information restoration atomic network, that is, the second result image , as shown below: (29) in, 、 are the second result image of the structural information complex atom step and the second high-dimensional feature vector output by the structural information complex atom step, respectively. This is the first result image of the declouding subtask. Conv9 is a 3×3 convolution operation with 64 input channels and 3 output channels.

[0055] Next, the structure information recovery module STRB described above is described in detail. The corresponding structure information is as follows: Figure 7 Next, we will take STRB1 as an example to explain in detail. Specifically, for the second feature vector input to this module , after layer normalization, it is input into the successive depth convolution layer and point-by-point convolution layer, and then the residual operation is used to output the feature vector of the preliminary structural information restoration as shown below: (30) in, 、 The second eigenvector of the input module and the eigenvector of the initial structural information recovery are respectively. LayerNorm is the layer normalization operation. pconv1 and dconv1 are 3×3 depthwise convolution layers and point-by-point convolution operations, respectively. Their input and output channels are both 64.

[0056] Feature vectors restored from preliminary structural information After layer normalization, the input is fed into the successive depth convolutional layers and point-by-point convolutional layers, as shown below: (31) in, is the feature vector for further structural information recovery, and LayerNorm is the layer normalization operation. pconv2 and dconv2 are 3×3 depthwise convolution layers and pointwise convolution operations, respectively, with 64 input and output channels.

[0057] The feature vector of further structural information is restored using the attention mechanism and residual mechanism The final step is to output the feature vector of the restored structural information of this module, as shown below: (32) in, This is the feature vector restored from the structural information output by this module, and sigmoid is the torch.nn.Sigmod activation function.

[0058] 3) High-frequency detail information atomization step; In this step, the high-frequency detail information complex atomic network HRNet is described in detail. Its network structure is as follows: Figure 4 Specifically, the second high-dimensional feature vector output by the structural information complex atomic step is , remote sensing images containing thin clouds After the feature splicing operation, a convolution layer is input to fuse the second high-dimensional feature vector And the original remote sensing image information containing thin clouds is as follows: (33) in, It is the third eigenvector after the fusion of the structural information restoration eigenvector and the original remote sensing image. It is a feature concatenation operation performed in the channel dimension. Conv10 is a 3×3 convolution operation with 67 input channels and 64 output channels.

[0059] Then the third eigenvector after information fusion is Input four consecutive high-frequency detail information restoration modules to restore high-frequency detail information, as shown below: (34) in, is the feature vector after the final high-frequency detail information is restored. HTRB1, HTRB2, HTRB3, and HTRB4 are four consecutive high-frequency detail information restoration modules, and their network structures are the same, such as Figure 8 The details will be described later.

[0060] Finally, the high-dimensional feature vector after high-frequency detail information restoration is input into a convolutional layer and then added to the image resulting from the structural information atomization step. This yields the final remote sensing image with thin cloud removal output by the high-frequency detail information atomization network, as shown below: (35) in, 、 They are the final output remote sensing image of thin cloud removal and the second result image output by the structural information atomization step, respectively. Conv11 is a 3×3 convolution operation with 64 input channels and 3 output channels.

[0061] Next, the high-frequency detail information restoration module HTRB described above is elaborated in detail. The corresponding structural information is as follows: Figure 8 Next, we take HTRB1 as an example to explain in detail. Specifically, for the third eigenvector input to this module , perform layer normalization and then perform fast Fourier transform to separate the amplitude part vector and the phase part vector, laying the foundation for the subsequent restoration of high-frequency information in the frequency domain, as shown below: (36) in, 、 The fast Fourier transform is used to separate the amplitude part vector and the phase part vector, LayerNorm is the layer normalization operation, and FFT is the fast Fourier transform operation.

[0062] A series of depthwise convolutional layers, pointwise convolutional layers, activation functions, and convolutional layer operations are performed on the amplitude part vector and the phase part vector respectively to restore the high-frequency information in the frequency domain, as shown below: (37) (38) in, 、 They are the amplitude and phase part vectors after the high-frequency detail information is restored respectively; conv12 and conv13 are both 3×3 convolution operations, with 64 input channels and 64 output channels; prelu is the torch.nn.PReLU activation function, pconv3 and pconv5 are both 3×3 point-by-point convolution operations, with 64 input channels and 64 output channels; dconv4 and dconv6 are both 3×3 depth-wise convolution operations, with 64 input channels and 64 output channels.

[0063] The amplitude vector after restoring high-frequency detail information , phase part vector Perform inverse fast Fourier transform to convert the feature vector after high-frequency information restoration from the frequency domain to the image domain, as shown below: (39) in, is the feature vector of the image domain after high-frequency information is restored, and IFFT is the inverse fast Fourier transform operation.

[0064] Feature vector of the image domain after high-frequency information is restored using attention mechanism and residual connection Perform the final step and output the feature vector after the high-frequency information of this module is restored, as shown below: (40) in, This is the feature vector of the high-frequency information restored by this module. The feature vector output by the last high-frequency detail information restoration module is the feature vector of the final high-frequency information restoration. .

[0065] The present invention decomposes the complex thin cloud removal problem into three sub-steps and uses global information to guide the solution of each task: the cloud removal sub-network, the structural information restoration sub-network, and the high-frequency detail information restoration sub-network. The thin cloud removal, structural information restoration and high-frequency detail information restoration sub-steps are gradually completed in a progressive manner. It achieves the effective restoration of low-frequency structural information and high-frequency detail information in remote sensing images while removing thin clouds in remote sensing images, thereby solving the complex thin cloud removal problem in remote sensing images with high fidelity.

Claims

1. A task-by-task progressive remote sensing image thin cloud removal method, characterized in that: The following steps are included: 1) A global information-guided cloud removal sub-step, comprising extracting shallow feature vectors from the input remote sensing image containing thin clouds, then downsampling and reducing the shallow feature vectors to obtain a first feature vector, obtaining a cloud removal guidance vector from the shallow feature vector, and inputting the cloud removal guidance vector and the first feature vector together into multiple consecutive cloud removal and restoration modules to remove thin clouds. The first high-dimensional feature vector and a first result image fused with the remote sensing image containing thin clouds are then output; 2) The structural information restoration step involves concatenating and fusing the remote sensing image containing thin clouds with the first high-dimensional feature vector, performing convolutional layer feature extraction, and downsampling to reduce the dimensionality to obtain a second feature vector. This is then fed into multiple structural information restoration modules to iteratively repair the structure, outputting the second high-dimensional feature vector and a second result image fused with the first result image. 3) The high-frequency detail information restoration step includes concatenating and fusing the remote sensing image containing thin clouds with the second high-dimensional feature vector, extracting the third feature vector through a convolutional layer, and then inputting it into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final feature vector after high-frequency detail information restoration is output and optimized by the convolutional layer. It is fused with the second result image to finally output the remote sensing image without thin clouds.

2. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as claimed in claim 1, characterized in that: The method for obtaining the cloud removal guidance vector is as follows: The shallow feature vectors are respectively input into two average pooling layers with different parameters to extract guidance information of different scales; The de-clouding guidance vector is obtained by processing the guidance information of two different scales using the attention mechanism and the residual mechanism.

3. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as claimed in claim 2, wherein: The processing method of the cloud removal and restoration module is as follows: Performing feature convolution on the first input feature vector and the de-clouding guidance vector to obtain a preliminary de-clouding guidance feature vector; After expanding the input dimension of the feature vector for preliminary de-clouding guidance, perform vector segmentation in the channel dimension to obtain two feature vectors; The two feature vectors after feature segmentation are processed with the declouding guidance vector and the first input feature vector using the attention mechanism and the residual mechanism, and then feature splicing is performed and input into the convolution layer to obtain the feature vector after thin cloud removal output by the declouding and restoration module; wherein the first input feature vector is the first feature vector or the feature vector output by the previous declouding and restoration module, and the output feature vector of the last declouding and restoration module is convolved and up-sampled to obtain the first high-dimensional feature vector.

4. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as claimed in claim 1, wherein: The processing method of the structure information recovery module is: The second input feature vector is normalized and then input into the continuous depth convolution layer and point-by-point convolution layer, and then the residual operation is used to output the feature vector of the preliminary structural information restoration; The feature vectors of the initial structural information restoration are normalized and then input into the continuous depth convolution layer and point-by-point convolution layer to obtain the feature vectors of further structural information restoration; The feature vector of further structural information restoration is processed using the attention mechanism and the residual mechanism to output a feature vector of structural information restoration; wherein the second input feature vector is the second feature vector or the feature vector output by the previous structural information restoration module; The output of the last-level structural information restoration module is convolved and up-sampled to obtain the second high-dimensional feature vector.

5. The method for removing thin clouds from remote sensing images in a task-by-task progressive manner as claimed in claim 1, wherein: The processing method of the high-frequency detail restoration module is: After normalizing the third input feature vector layer, perform fast Fourier transform to separate the amplitude part vector and the phase part vector; Performing depth convolution layer, point-by-point convolution layer, activation function, and convolution layer operations on the amplitude part vector and the phase part vector respectively to restore high-frequency information in the frequency domain; Performing an inverse fast Fourier transform on the amplitude component vector and the phase component vector after the high-frequency detail information is restored to convert the feature vector after the high-frequency information is restored from the frequency domain to the image domain; Using the attention mechanism and residual connection to restore the high-frequency information in the image domain, the feature vector is obtained, wherein the third input feature vector is the third feature vector or the feature vector output by the previous high-frequency detail restoration module; The output of the last-level high-frequency detail restoration module is convolved and up-sampled to obtain the third eigenvector.

6. A task-by-task progressive remote sensing image thin cloud removal system, characterized by: include, 1) A global information-guided cloud removal sub-network is used to input a remote sensing image containing thin clouds and extract shallow feature vectors. This is then downsampled and dimensionally reduced to obtain a first feature vector. A cloud removal guidance vector is derived from the shallow feature vector. This guidance vector and the first feature vector are then fed into multiple consecutive cloud removal and restoration modules to remove thin clouds. The network then outputs a first high-dimensional feature vector and a first result image fused with the remote sensing image containing thin clouds. 2) A structural information restoration atomic network, which takes as input the concatenated features of the remote sensing image containing thin clouds and the first high-dimensional feature vector, performs convolutional layer feature extraction and downsampling to reduce the dimensionality to obtain a second feature vector. This is then fed into multiple structural information restoration modules to iteratively repair the structure, and outputs a second high-dimensional feature vector and a second result image fused with the first result image. 3) High-frequency detail information restoration sub-network, which is used to input the concatenated fusion features of the remote sensing image containing thin clouds and the second high-dimensional feature vector and extract the third feature vector through the convolution layer. The third feature vector is then input into multiple high-frequency detail restoration modules to iteratively restore the high-frequency details of the image. The final feature vector after high-frequency detail information restoration is output and optimized by the convolution layer. It is fused with the second result image to finally output the remote sensing image without thin clouds.

7. The task-by-task progressive remote sensing image thin cloud removal system according to claim 6, characterized in that: Corresponding loss functions are established for the three sub-networks respectively, and then the overall high-fidelity restoration loss function is added to constrain the overall dimension of the image. They are combined together as the total loss function to train the task-by-task progressive remote sensing image thin cloud removal system.

Images