Multi-temporal remote sensing image cloud removal method based on global perception selective fusion

Abstract

The invention discloses a multi-temporal remote sensing image cloud removal method based on global perception selective fusion, which belongs to the technical field of image cloud removal, is used for image cloud removal, and comprises the following steps: inputting a three-temporal cloud remote sensing image into a generative adversarial network framework, and inputting an output result into the generative adversarial network framework again after obtaining the output result; and repeating the cycle until a cloudless image meeting the requirement is obtained. The generative adversarial network framework comprises a generator and a discriminator, the generator comprises an encoder, an advanced feature extraction module and a decoder, the discriminator outputs a probability value to represent an image source, a multi-scale convolutional layer extracts image texture features, an adversarial loss function optimizes network parameters, and adversarial loss comprises pixel-level reconstruction loss and adversarial training loss. According to the method, the problem of space-time discontinuity in multi-temporal data fusion is effectively solved, earth surface detail retention and time sequence change rule capture are realized in cloud coverage area feature reconstruction, and a cloud-removed image with relatively good quality is obtained.

Description

Technical Field

[0001] The invention discloses a multi-temporal remote sensing image declouding method based on global perception selective fusion, and belongs to the technical field of image declouding. Background Art

[0002] Affected by atmospheric conditions, the loss of spectral information caused by cloud cover seriously destroys the continuity of the image, greatly restricting its effectiveness in subsequent analysis and application. With the rapid development of deep learning technology, data-driven cloud removal methods have gradually become a research hotspot, showing great application potential in key areas such as dynamic environmental monitoring and disaster emergency response. Especially in the context of the surge in high-resolution earth observation satellite data, how to break through the performance bottleneck of traditional methods and develop efficient and robust cloud removal algorithms has become an urgent need to improve the availability and timeliness of remote sensing data products.

[0003] The current mainstream remote sensing image declouding methods can be summarized into four categories: methods based on spatial restoration, methods based on spectral analysis, hybrid methods, and methods based on multi-temporal data. In recent years, multi-temporal declouding technology has attracted widespread attention due to its unique advantages. This method restores cloud-covered areas by integrating data from different temporal periods in the same geographical area. Compared with spatial domain declouding methods that are susceptible to complete cloud coverage limitations and the difficulty of heterogeneous data registration in SAR image fusion, multi-temporal methods can not only break through the bottleneck of continuous cloud cover, but also avoid the complexity of multi-modal data fusion, showing significant application value.

[0004] Early studies mostly adopted traditional multi-temporal image stitching strategies, such as the MODIS product method based on optimal pixel synthesis, but such methods are sensitive to inter-phase radiation differences and surface changes, which can easily lead to spatiotemporal discontinuities in the restoration results. With the development of deep learning, the cloud removal method based on neural networks has shown powerful data modeling capabilities, learning cloud distribution patterns through large-scale time series data, but its feature fusion mechanism lacks effective spatiotemporal correlation modeling. Subsequent studies have gradually improved feature selection and global modeling capabilities by introducing attention mechanisms and Swin Transformer architectures, but there are still two key bottlenecks. First, the existing attention modules mostly use independent calculation modes for channels and spatial weights, which breaks the inherent association between spatiotemporal features; second, the Transformer architecture is directly transplanted from general visual tasks, and has not been adaptively improved for multi-temporal data characteristics, which limits the effectiveness of global context modeling. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-temporal remote sensing image declouding method based on global perception selective fusion, so as to solve the problem in the prior art that the neural network-based declouding method breaks the inherent correlation of spatiotemporal features and limits the effectiveness of global context modeling.

[0006] A multi-temporal remote sensing image declouding method based on global perception selective fusion includes inputting three-temporal cloud remote sensing images into a generative adversarial network framework, re-inputting the output results into the generative adversarial network framework after obtaining them, and repeating the cycle until a cloud-free image that meets the requirements is obtained.

[0007] The generative adversarial network framework includes a generator and a discriminator. The generator includes an encoder, an advanced feature extraction module and a decoder. The discriminator outputs a probability value to represent the source of the image. The multi-scale convolutional layer extracts image texture features. The adversarial loss function optimizes network parameters. The adversarial loss includes pixel-level reconstruction loss and adversarial training loss.

[0008] The encoder includes a downsampling module, a triple weight selection module and a feature fusion layer;

[0009] The downsampling module includes three groups of parallel downsampling units, each group of downsampling units includes a convolution layer, a ReLU activation function and a batch normalization layer, and achieves size reduction of feature maps and multiplication of the number of channels through convolution kernels of different step sizes;

[0010] The triple weight selection module includes three parallel processing branches and a dynamic weight generator.

[0011] Input features of triple weight selection module The size is , is the number of channels, is the height, is the width;

[0012] Will Enter the first processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ;

[0013] Will Input to the second processing branch, passing through the spatial attention layer;

[0014] Will Enter the third processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ;

[0015] The results of the three processing branches are respectively processed through global pooling, average pooling, and convolutional layers to generate three weight maps, and the important parts of the feature maps of each processing branch are obtained. The original dimensions are restored after weighting the input features.

[0016] The dynamic weight generator includes a first feature transformation layer, a second feature compression layer and a weight prediction layer;

[0017] The first feature transformation layer uses a 3×3 convolution kernel with a step size of 1. The number of output channels is expanded to 32 times the input feature, and nonlinear mapping is performed through the ReLU activation function.

[0018] The second feature compression layer achieves spatial feature aggregation through a 3×3 convolution kernel, reducing the number of output channels to 16 times, and connecting a ReLU activation function to enhance feature sparsity;

[0019] The weight prediction layer uses a 1×1 convolution kernel to compress the number of channels to 3, and generates the three-dimensional weight coefficients of the original feature map after the Softmax activation function.

[0020] The feature fusion layer multiplies the three-dimensional weight coefficients with the corresponding weight maps respectively, and then sums them up to obtain the enhanced feature map.

[0021] The high-level feature extraction module adopts the UNet structure, which includes four residual downsampling blocks and six transposed convolution upsampling blocks;

[0022] The residual downsampling block extracts high-level semantic features through 3×3 convolution, and each block of the upsampling block contains a ReLU activation function, a deconvolution layer, and a batch normalization layer to gradually restore the feature map size.

[0023] The decoder consists of two ReSwin stages, each of which includes a ReLN normalization layer and a variable module;

[0024] The ReLN normalization layer includes three branches, and the three branches correspond to three outputs. The first branch includes a mean layer and a point convolution layer to form a first output. The second branch includes a variance layer and a point convolution layer to form a second output. The central branch includes layer normalization. The output of the layer normalization is combined with the first output, and then feature fused with the second output to obtain the central output.

[0025] The input features of the decoder are sequentially input to the ReLN normalization layer to generate three outputs. The central output enters the variable module, the second output is combined with the output of the variable module, and then feature fused with the first output and finally output.

[0026] The variable module of the first ReSwin stage is the windowed multi-head self-attention unit W-MSA, and the variable module of the second ReSwin stage is the multi-layer perceptron MLP.

[0027] Compared with the prior art, the present invention has the following beneficial effects: the present invention effectively solves the problem of spatiotemporal discontinuity in multi-temporal data fusion, realizes the retention of surface details and the capture of temporal change laws in the reconstruction of cloud-covered area features, and obtains a cloud-free image with better quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Module structure diagram for triple weight selection;

[0029] Figure 2 This is the ReSwin stage structure diagram;

[0030] Figure 3 This is the normalized layer structure diagram of ReLN. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] A multi-temporal remote sensing image declouding method based on global perception selective fusion includes inputting three-temporal cloud remote sensing images into a generative adversarial network framework, re-inputting the output results into the generative adversarial network framework after obtaining them, and repeating the cycle until a cloud-free image that meets the requirements is obtained.

[0033] The generative adversarial network framework includes a generator and a discriminator. The generator includes an encoder, an advanced feature extraction module and a decoder. The discriminator outputs a probability value to represent the source of the image. The multi-scale convolutional layer extracts image texture features. The adversarial loss function optimizes network parameters. The adversarial loss includes pixel-level reconstruction loss and adversarial training loss.

[0034] The encoder includes a downsampling module, a triple weight selection module and a feature fusion layer;

[0035] The downsampling module includes three groups of parallel downsampling units, each group of downsampling units includes a convolution layer, a ReLU activation function and a batch normalization layer, and achieves size reduction of feature maps and multiplication of the number of channels through convolution kernels of different step sizes;

[0036] The triple weight selection module includes three parallel processing branches and a dynamic weight generator.

[0037] Input features of triple weight selection module The size is , is the number of channels, is the height, is the width;

[0038] Will Enter the first processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ;

[0039] Will Input to the second processing branch, passing through the spatial attention layer;

[0040] Will Enter the third processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ;

[0041] The results of the three processing branches are respectively processed through global pooling, average pooling, and convolutional layers to generate three weight maps, and the important parts of the feature maps of each processing branch are obtained. The original dimensions are restored after weighting the input features.

[0042] The dynamic weight generator includes a first feature transformation layer, a second feature compression layer and a weight prediction layer;

[0043] The first feature transformation layer uses a 3×3 convolution kernel with a step size of 1. The number of output channels is expanded to 32 times the input feature, and nonlinear mapping is performed through the ReLU activation function.

[0044] The second feature compression layer achieves spatial feature aggregation through a 3×3 convolution kernel, reducing the number of output channels to 16 times, and connecting a ReLU activation function to enhance feature sparsity;

[0045] The weight prediction layer uses a 1×1 convolution kernel to compress the number of channels to 3, and generates the three-dimensional weight coefficients of the original feature map after the Softmax activation function.

[0046] The feature fusion layer multiplies the three-dimensional weight coefficients with the corresponding weight maps respectively, and then sums them up to obtain the enhanced feature map.

[0047] The high-level feature extraction module adopts the UNet structure, which includes four residual downsampling blocks and six transposed convolution upsampling blocks;

[0048] The residual downsampling block extracts high-level semantic features through 3×3 convolution, and each block of the upsampling block contains a ReLU activation function, a deconvolution layer, and a batch normalization layer to gradually restore the feature map size.

[0049] The decoder consists of two ReSwin stages, each of which includes a ReLN normalization layer and a variable module;

[0050] The ReLN normalization layer includes three branches, and the three branches correspond to three outputs. The first branch includes a mean layer and a point convolution layer to form a first output. The second branch includes a variance layer and a point convolution layer to form a second output. The central branch includes layer normalization. The output of the layer normalization is combined with the first output, and then feature fused with the second output to obtain the central output.

[0051] The input features of the decoder are sequentially input to the ReLN normalization layer to generate three outputs. The central output enters the variable module, the second output is combined with the output of the variable module, and then feature fused with the first output and finally output.

[0052] The variable module of the first ReSwin stage is the windowed multi-head self-attention unit W-MSA, and the variable module of the second ReSwin stage is the multi-layer perceptron MLP.

[0053] The triple weight selection module TWSM of the present invention is as follows Figure 1 As shown, it includes three parallel processing branches and a dynamic weight generator. Input features of the triple weight selection module The size is , is the number of channels, is the height, is the width; Enter the first processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ;Will Input the second processing branch, pass through the spatial attention layer; Enter the third processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ; The results of the three processing branches are respectively subjected to global pooling, average pooling, and convolution layers to generate three weight maps, and the important parts of the feature map of each processing branch are obtained. The input features are weighted and the original dimensions are restored. The dynamic weight generator includes the first feature transformation layer, the second feature compression layer, and the weight prediction layer; the first feature transformation layer uses a 3×3 convolution kernel with a step size of 1, and the number of output channels is expanded to 32 times the input feature, and nonlinear mapping is performed through the ReLU activation function; the second feature compression layer realizes spatial feature aggregation through a 3×3 convolution kernel, and the number of output channels is reduced to 16 times, and the ReLU activation function is connected to enhance feature sparsity; the weight prediction layer uses a 1×1 convolution kernel to compress the number of channels to 3, and generates the three-dimensional weight coefficients of the original feature map after the Softmax activation function. The feature fusion layer multiplies the three-dimensional weight coefficients with the corresponding weight maps, and then sums them up to obtain the enhanced feature map. .

[0054] ReSwin stage Figure 2 As shown in Figure 1, each ReSwin stage includes a ReLN normalization layer and a variable module; the input features of the decoder are sequentially input to the ReLN normalization layer to generate three outputs. The central output enters the variable module, the second output is combined with the output of the variable module, and then the feature fusion is performed with the first output to generate the output feature. The variable module of the first ReSwin stage is the windowed multi-head self-attention unit W-MSA, and the variable module of the second ReSwin stage is the multi-layer perceptron MLP.

[0055] ReLN normalization layer is as follows Figure 3 As shown, the ReLN normalization layer includes three branches, and the three branches correspond to three outputs. The first branch includes a mean layer and a point convolution layer to form a first output , the second branch includes a variance layer and a point convolution layer to form the second output , the central branch includes layer normalization, the output of layer normalization is combined with the first output, and then feature fused with the second output to obtain the central output .

[0056] The present invention adopts two data sets, namely the STGAN data set and the Sen2_MTC data set. The image acquisition period of the STGAN data set covers 2017-2021, constituting a continuous observation sequence across seasons and years. It is constructed by multi-phase remote sensing images of the Sentinel-2 satellite, with an average of one new image taken at the same geographical location every 6 days, covering a variety of climate zones and terrains around the world. The dataset contains 3130 cloud-free images and corresponding 9390 cloud-containing images (each cloud-free image corresponds to three cloud-containing images in different phases). The image size is 256×256 pixels, including four bands: red (R), green (G), blue (B) and near-infrared (NIR). The present invention only uses RGB band data.

[0057] The Sen2_MTC dataset is selected from Sentinel-2 images and covers real cloud distribution scenes. It contains 50 non-overlapping image blocks, each containing 70 continuous images with a size of 256×256 pixels, a time span of 14 months (2019-2020), and a time interval of 5 days. Focusing on cloud-prone areas, the 50 image blocks are distributed in the Southeast Asian monsoon region, the equatorial convective active belt, and the mid-latitude frontal cloud system area. The dataset contains complex illumination changes and dynamic cloud motion to verify the robustness of the model under dynamic cloud interference.

[0058] The effectiveness of the proposed method is verified from three parts: data set description, comparative experiment and ablation experiment. All experiments are based on the same hardware environment (V100) and hyperparameter settings (initial learning rate 0.0002, batch size 8) to ensure fairness of the comparison. In order to verify the effectiveness of the proposed network, the following representative methods are selected for comparison, including Pix2Pix, ResNet, SIF-GAN, and MSC-GAN.

[0059] Pix2Pix splices the RGB bands of three-phase images (t-1, t, t+1) into 9-channel inputs according to the time dimension, and realizes cross-phase feature interaction through the jump connection of the U-Net generator. The encoder uses 5 layers of downsampling (convolution with a step size of 2), and each layer superimposes a spatiotemporal attention mechanism to enhance the ability to capture cloud motion trajectories. The improved PatchGAN structure is used to expand the discrimination window to a spatiotemporal cube, and the spatial texture continuity and temporal dynamic consistency are evaluated simultaneously.

[0060] ResNet is based on the ResNet generator. Its input is the splicing features of multi-phase images. It relies on residual learning to alleviate the gradient vanishing. It builds a pyramid residual block, which includes a basic residual path, an expanded residual path, and a feature fusion layer. It processes the temporal correlation of time series images based on ResNet, and fuses the temporal features with the convolutional spatial features through gated residual connections.

[0061] STGAN is a multi-temporal declouding network that combines U-Net and ResNet, and introduces infrared bands to assist cloud detection and feature recovery. In the first stage, the U-Net cloud detection module is pre-trained with synthetic cloud data, and in the second stage, the encoder parameters are frozen and the ResNet feature recovery module is jointly trained to avoid feature conflicts caused by modality differences.

[0062] SIF-GAN solves the problem of cloud residue and ground object restoration distortion caused by the redundancy of feature fusion mechanism in multi-temporal remote sensing image declouding tasks, and realizes effective screening and fusion of multi-temporal features through dynamic weight allocation. It includes the design of dynamic feature screening mechanism, the introduction of improved channel attention module in the encoding stage, and the dynamic calculation of response weights of cloud pollution areas in RGB bands through parallel channel statistical analysis and spatial gradient perception. This module can suppress the interference characteristics of cloud-covered areas and enhance the texture details of cloud-free areas. At the same time, a dual-branch spatiotemporal feature interaction structure is designed: the main branch uses a residual contraction network to extract the current phase features; the auxiliary branch uses a gated recurrent unit (GRU) to model the temporal dependency relationship, generate a cross-phase feature weight matrix, and realize the dynamic weighted fusion of adjacent phase features. Triple constraint loss function, including adversarial loss, spectral fidelity loss and edge sharpening loss.

[0063] MSC-GAN uses a proposed multi-stream complementary coding architecture to address the dual challenges of insufficient cross-temporal feature interaction and semantic degradation of deep features in multi-temporal remote sensing image declouding tasks. A parallel three-stream coding network (current phase, previous phase, and subsequent phase) is constructed, and dynamic calibration of temporal features is achieved through a bidirectional gated attention module. Each coding stream contains 5 levels of downsampling layers. When generating feature maps at each level, a cross-stream feature distillation operation is introduced to retain cloud motion trajectories and surface stability features. Residual dense connections are used to replace traditional skip connections, and inter-layer feature reuse channels are established in the coding stage, so that shallow cloud detection features can directly participate in deep surface reconstruction, alleviating information attenuation caused by increased network depth. In addition, a group feature reweighting module is designed to divide high and low-level features into 4 semantic groups (terrain group, texture group, spectral group, and context group) according to the channel dimension. After extracting multi-scale features through parallel dilated convolutions (expansion rates = 2, 4, 6, 8), a dynamic weight allocator is used to achieve cross-group feature enhancement.

[0064] The present invention selects peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and cosine similarity (CS) as accuracy evaluation indicators, and conducts experiments on the STGAN dataset as shown in Table 1.

[0065] Table 1. STGAN dataset experiment (unit: dB)

[0066] ;

[0067] In Table 1, GSF represents the method of the present invention, and experiments are performed on the Sen2_MTC dataset as shown in Table 2.

[0068] Table 2. Sen2_MTC dataset experiment (unit: dB)

[0069] ;

[0070] In the STGAN and Sen2_MTC datasets, the present invention achieves the best results in terms of PSNR, SSIM and cosine similarity (CS). On the STGAN dataset, MSC-GAN achieves suboptimal results due to its multi-stream complementary architecture, but its performance on Sen2_MTC significantly decreases (SSIM decreases by 0.22), indicating that it is not adaptable enough to dynamic real clouds; while the network proposed in the present invention maintains stable performance in both datasets through the synergy of the temporal weighted selection module (TWSM) and the ReSwin global modeling.

[0071] In terms of thick cloud recovery, in the high cloud coverage area of ​​the STGAN dataset, the proposed network accurately recovers the boundaries of ground objects under the cloud by capturing multi-temporal global context, while ResNet and Pix2Pix lack temporal information fusion, resulting in ground object confusion, such as erroneous recovery of bare soil as vegetation. In terms of dynamic cloud processing, in the cirrus occlusion scene of Sen2_MTC, the proposed network effectively suppresses cloud artifacts through long-range dependency modeling of the ReSwin module, while SIF-GAN has poor recovery of ground details under the cloud due to biased local feature weight distribution.

[0072] To verify the contribution of each module, key components are gradually introduced with U-Net as the baseline. The experimental results are shown in Table 3.

[0073] Table 3. Ablation experiment (unit: dB)

[0074] ;

[0075] When TWSM is used alone, PSNR is only improved by 0.21dB. The reason is that although TWSM can enhance the representation of key regional features (such as cloud edges), it lacks a cross-temporal feature fusion mechanism and cannot fully utilize multi-temporal complementary information. When the ReSwin module is used alone, PSNR is improved by 0.95dB, but due to the lack of high-quality local feature support, global modeling is limited, resulting in blurred recovery results. When TWSM is used in conjunction with ReSwin, PSNR is improved by 1.32dB. The local key features extracted by TWSM complement the global temporal laws modeled by ReSwin.

[0076] In the present invention, the triple weight selection module includes a multi-dimensional attention branch, which establishes the correlation between the channel dimension and the spatiotemporal dimension through a tensor rotation operation; a dynamic weight generator, which generates the attention weights of each branch based on the spatial context of the feature map; an adaptive fusion layer, which performs weighted fusion on the features processed by the spatial attention, channel attention and spatiotemporal attention branches; in the four-stage encoding process, the resolution of the output feature map of each stage is gradually reduced (by ), the number of channels increases step by step ( ). Each decoding stage includes: a ReLN normalization layer, which maintains the relative difference between blocks through learnable scaling parameters and translation parameters; a windowed multi-head self-attention unit, which calculates the temporal-spatial joint attention in a local window and dynamically models the temporal laws such as vegetation growth and water body changes; a multi-layer perceptron, which performs nonlinear transformation on the attention output; in the decoding process, the feature map resolution is restored to the original size through two upsampling operations.

[0077] The key components of TWSM are BatchNorm, ReLU activation layer, convolution layer, spatial attention mechanism and dynamic weight generator. First, the input feature map is sent to the three-dimensional feature rotation unit, which contains three parallel processing branches: the first branch rotates the input feature map 90° counterclockwise along the channel-width plane to generate X_out1 with dimensions of H×C×W, capturing the interactive features of the channel and width dimensions; the second branch rotates the input feature map 90° counterclockwise along the channel-height plane to generate X_out2 with dimensions of W×C×H, modeling the association between the channel and height dimensions; the third branch keeps the original dimension of X_out3 of C×H×W, directly processing the local features of the spatial dimension. Then the three branches generate weight maps through global pooling, average pooling and convolution layers respectively, obtain the important parts of the feature map of each branch, and restore the original dimension after weighting the input feature map.

[0078] The dynamic weight generator is composed of three levels of convolutional neural networks connected in sequence. Its processing flow includes the first feature transformation layer, which uses a 3×3 convolution kernel, a step size of 1, and the number of output channels is expanded to 32 times the input feature, and nonlinear mapping is performed through the ReLU activation function; the second feature compression layer, which realizes spatial feature aggregation through a 3×3 convolution kernel, reduces the number of output channels to 16 times, and connects the ReLU activation function to enhance feature sparsity; the weight prediction layer, uses a 1×1 convolution kernel to compress the number of channels to 3, and generates the three-dimensional weight coefficients of the original feature map.

[0079] The core idea of ​​the ReSwin module design is to make the necessary improvements to make the Swin Transformer module more suitable for cloud removal tasks. The Swin Transformer's ability to capture global information enables the model to learn the temporal laws in multi-temporal images. Due to the slight differences in multi-temporal images, it is not possible to simply splice multiple images to get the final result. It is necessary to learn the internal temporal laws, such as changes in climate or weather, to make a more reasonable prediction of the final image. Therefore, the powerful ability of Swin Transformer to capture global information is used to enrich the model. However, some components of Swin Transformer are not suitable for low-level cloud removal tasks, so the module is improved for cloud removal tasks. First, ReLN normalization is designed to replace the original layer normalization. Since layer normalization will smooth out the relative differences in brightness (mean) and contrast (variance) between different image blocks, it makes it difficult for the model to distinguish between cloud-covered areas and clear surface areas. Therefore, the statistical mean and variance are reintroduced in ReLN normalization. After the introduced parameters pass through the convolution layer, an affine transformation is performed on the original output, and the mean and variance are output.

[0080] The spatiotemporal joint attention unit includes a window division module that divides the feature map into 8×8 non-overlapping windows; a cross-window shift mechanism that alternately uses (4, 4) and (-4, -4) window offsets in adjacent layers; multi-head attention calculations that are performed on the Q, K, and V matrices in each window. The windowed multi-head self-attention unit reduces the computational complexity and effectively models the global context information of the image, better understands the structure of the entire image and the distribution of objects, and thus more accurately infers the content of the cloud-occluded area. After another ReLN normalization, it enters the improved MLP module.

[0081] MLP includes convolutional layers and ReLU activation functions. The activation function in MLP in Swin is GeLU activation function. The non-monotonicity of GeLU may lead to gradient reversal problems, and nonlinearity also leads to its irreversibility. However, the essence of image declouding is the inverse process of restoring the original scene from the degraded input. Irreversibility will hinder the restoration of the image. The simple inverse transformation characteristics of ReLU are more conducive to image restoration. In addition, the number of convolutions is increased (twice before and after ReLU). After MLP, affine transformation is performed according to the mean and variance weights of ReLN output, and finally output.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-temporal remote sensing image cloud removal method based on global perception selective fusion, characterized in that: The method includes inputting three-temporal cloud remote sensing images into a generative adversarial network framework, re-inputting the output results into the generative adversarial network framework after obtaining them, and repeating the cycle until a cloud-free image that meets the requirements is obtained.

2. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 1 is characterized in that: The generative adversarial network framework includes a generator and a discriminator. The generator includes an encoder, an advanced feature extraction module and a decoder. The discriminator outputs a probability value to represent the source of the image. The multi-scale convolutional layer extracts image texture features. The adversarial loss function optimizes network parameters. The adversarial loss includes pixel-level reconstruction loss and adversarial training loss.

3. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 2 is characterized in that: The encoder includes a downsampling module, a triple weight selection module and a feature fusion layer; The downsampling module includes three groups of parallel downsampling units, each group of downsampling units includes a convolution layer, a ReLU activation function and a batch normalization layer, and achieves size reduction of feature maps and multiplication of the number of channels through convolution kernels of different step sizes; The triple weight selection module includes three parallel processing branches and a dynamic weight generator.

4. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 3 is characterized in that: Input features of triple weight selection module The size is , is the number of channels, is the height, is the width; Will Enter the first processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ; Will Input to the second processing branch, passing through the spatial attention layer; Will Enter the third processing branch, after passing through the first rotation layer, The size is , after passing through the spatial attention layer and then the second rotation layer, The size is ; The results of the three processing branches are respectively processed through global pooling, average pooling, and convolutional layers to generate three weight maps, and the important parts of the feature maps of each processing branch are obtained. The original dimensions are restored after weighting the input features.

5. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 4 is characterized in that: The dynamic weight generator includes a first feature transformation layer, a second feature compression layer and a weight prediction layer; The first feature transformation layer uses a 3×3 convolution kernel with a step size of 1. The number of output channels is expanded to 32 times the input feature, and nonlinear mapping is performed through the ReLU activation function. The second feature compression layer achieves spatial feature aggregation through a 3×3 convolution kernel, reducing the number of output channels to 16 times, and connecting a ReLU activation function to enhance feature sparsity; The weight prediction layer uses a 1×1 convolution kernel to compress the number of channels to 3, and generates the three-dimensional weight coefficients of the original feature map after the Softmax activation function.

6. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 5 is characterized in that: The feature fusion layer multiplies the three-dimensional weight coefficients with the corresponding weight maps respectively, and then sums them up to obtain the enhanced feature map.

7. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 6 is characterized in that: The high-level feature extraction module adopts the UNet structure, which includes four residual downsampling blocks and six transposed convolution upsampling blocks; The residual downsampling block extracts high-level semantic features through 3×3 convolution, and each block of the upsampling block contains a ReLU activation function, a deconvolution layer, and a batch normalization layer to gradually restore the feature map size.

8. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 7 is characterized in that: The decoder consists of two ReSwin stages, each of which includes a ReLN normalization layer and a variable module; The ReLN normalization layer includes three branches, and the three branches correspond to three outputs. The first branch includes a mean layer and a point convolution layer to form a first output. The second branch includes a variance layer and a point convolution layer to form a second output. The central branch includes layer normalization. The output of the layer normalization is combined with the first output, and then feature fused with the second output to obtain the central output.

9. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 8, characterized in that: The input features of the decoder are sequentially input to the ReLN normalization layer to generate three outputs. The central output enters the variable module, the second output is combined with the output of the variable module, and then feature fused with the first output and finally output.

10. The multi-temporal remote sensing image cloud removal method based on global perception selective fusion according to claim 9, characterized in that: The variable module of the first ReSwin stage is the windowed multi-head self-attention unit W-MSA, and the variable module of the second ReSwin stage is the multi-layer perceptron MLP.

Images