A cloud detection method and system for high-resolution remote sensing images based on a multi-scale cross self-attention Transformer network

Abstract

This invention discloses a cloud detection method and system for Gaofen-1 remote sensing images based on a multi-scale cross-self-attention Transformer network, belonging to the field of computer vision and remote sensing image processing technology. The method includes: acquiring and preprocessing multispectral images from the Gaofen-1 satellite; and inputting the processed images into a trained TransUNet deep learning model. The model employs an encoder-decoder architecture: the encoder is built based on an improved hybrid vision Transformer to extract multi-scale features; the core multi-scale cross-self-attention module performs multi-scale pooling on deep features, generating multiple sets of key-value pairs, which are then used as queries for cross-attention calculation, effectively fusing global and local contextual information; the decoder, through progressive upsampling and skip connections, fuses the encoder's multi-scale features to gradually recover spatial details. The model outputs pixel-level four-class probability maps (clear sky, thick clouds, thin clouds, and cloud shadows) and generates a final cloud mask. This invention fully utilizes the global modeling capability and multi-scale feature fusion of the Transformer, significantly improving the accuracy and robustness of cloud detection for thin clouds, fragmented clouds, and complex scenes in Gaofen-1 images.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically involving a deep learning model for semantic segmentation of remote sensing images, which is particularly suitable for automated and high-precision detection of thick clouds, thin clouds, cloud shadows, and clear-sky ground features in multispectral images of Gaofen-1 (GF-1) satellite. Background Technology

[0002] Optical remote sensing imagery, as a crucial data source for Earth observation, is widely used in fields such as geographic mapping, environmental monitoring, and disaster assessment. However, cloud cover is one of the main factors affecting the usability of optical remote sensing imagery. Statistics show that approximately 66% of the Earth's surface is covered by clouds, severely restricting the effective utilization of remote sensing data.

[0003] Traditional cloud detection methods often rely on threshold segmentation, spectral indexing, or shallow machine learning models, which have limited robustness and generalization capabilities, making them ill-suited for complex surface backgrounds and diverse cloud formations. In recent years, deep learning methods based on convolutional neural networks (CNNs) have significantly improved cloud detection accuracy. However, the inherent local receptive field of CNNs makes it difficult to effectively model global contextual dependencies in remote sensing imagery, resulting in insufficient ability to identify thin clouds, fragmented clouds, and cloud shadow boundaries, and susceptibility to interference from high-albedo features.

[0004] The Transformer architecture, with its self-attention mechanism, has demonstrated powerful global modeling capabilities in natural language processing and computer vision. However, directly applying it to remote sensing image cloud detection still faces challenges: remote sensing images are characterized by multi-band, high resolution, and complex ground structures, requiring the design of an appropriate encoder to extract multi-scale features and to effectively fuse local details with global semantic information.

[0005] To address this, a cloud detection method based on the multi-scale cross-attention Transformer network (TransUNet) for Gaofen-1 remote sensing images is proposed. This method aims to solve the problems of insufficient global context modeling and inadequate utilization of multi-scale features in existing technologies, and to achieve refined identification of clouds, thin clouds, and cloud shadows. Summary of the Invention

[0006] The purpose of this invention is to provide a high-precision and robust remote sensing image cloud detection method and system, which is particularly suitable for Gaofen-1 satellite imagery and can achieve pixel-level automatic classification of clouds, thin clouds, cloud shadows and clear sky.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A cloud detection method for Gaofen-1 remote sensing images based on a multi-scale cross-attention Transformer network includes the following steps:

[0009] S1: Acquire multispectral remote sensing images from the Gaofen-1 satellite, perform preprocessing including radiometric and geometric correction, and crop the images into fixed-size tiles;

[0010] S2: Input the preprocessed image patches into the trained TransUNet deep learning model;

[0011] S3: The TransUNet model extracts multi-scale hierarchical features of the input image through its hybrid feature encoder;

[0012] S4: Input the deepest feature map output by the encoder into the multi-scale cross self-attention module to fuse global contextual information with multi-scale local features;

[0013] S5: The progressive feature decoder is used to upsample the fused features and fuse them with the features of the corresponding level of the encoder through skip connections to gradually restore the spatial resolution.

[0014] S6: Output a probability map of each pixel belonging to one of the four categories: "clear sky", "thick clouds", "thin clouds" and "cloud shadows", and generate the final cloud detection mask through threshold segmentation or Argmax operation.

[0015] The TransUNet model includes:

[0016] Hybrid Feature Encoder: The encoder is built on an improved Visual Transformer (ViT) architecture, with its first-layer convolutional network adapted to the blue, green, red, and near-infrared bands of the Gaofen-1 image. The encoder contains four downsampling stages, each consisting of multiple stacked Transformer blocks for extracting multi-level features. The core of each Transformer block is a Multi-Head Self-Attention (MHSA) mechanism, used to model long-range dependencies in the feature maps. Let the input feature sequence within each stage be... Where N is the sequence length and D is the feature dimension, the query (Q), key (K), and value (V) matrix is ​​obtained through linear mapping:

[0017]

[0018] in For learnable weight matrix, The dimension is defined for each attention head. The calculation for single-head attention is as follows:

[0019]

[0020] The outputs of multiple attention heads are concatenated and then linearly projected to obtain the output of MHSA:

[0021]

[0022] Where h is the number of attention heads. The output projection matrix is ​​denoted as . The multi-scale feature maps output by the encoder in the four stages are denoted as . ,in The spatial resolution decreases step by step, while the semantic information is enhanced step by step.

[0023] Multi-scale cross-self-attention module: This module is the core innovation of this invention, aiming to fuse global context and multi-scale local context information. It uses the deepest feature map output by the encoder. The primary input is used. First, multiple average pooling operations of different scales are performed in parallel. Downsampling is performed to generate multiple key-value (KV) pairs while preserving the original resolution. This is the query (Q). Subsequently, cross-attention is calculated, enabling the global query to retrieve relevant information from contextual features at different scales. This process can be represented as:

[0024]

[0025] Here, 's' represents different downsampling scales. Attention outputs from different scales are fused (e.g., concatenated or weighted summation) to obtain enhanced features rich in multi-scale global contextual information. .

[0026] Progressive Feature Decoder: The decoder consists of multiple decoding blocks, responsible for progressively restoring spatial resolution and performing fine segmentation. Each decoding block upsamples the features from the previous layer and then connects them to the feature maps at the corresponding scales of the encoder via skip connections. The fusion process begins by combining the encoder features. The features are concatenated with the upsampled decoder features along the channel dimension, and then integrated through a convolutional layer. Let the input features of the j-th decoded block be... The corresponding encoder features are The fusion process can then be represented as:

[0027]

[0028] By fusing multi-scale features step by step, the decoder can simultaneously utilize deep semantic information and shallow detail information (such as edges) to ultimately restore the feature map to the size of the input image.

[0029] Classification output head and model training: A classification head is set in the final output layer of the decoder and intermediate layers (for deep supervision). It is usually composed of a 1×1 convolutional layer and a softmax activation function, which maps the features to the probability that each pixel belongs to one of the four categories: "clear sky", "thick clouds", "thin clouds", and "cloud shadow". The model training uses the cross-entropy loss function:

[0030]

[0031] Where N is the total number of pixels, and C=4 is the number of categories. One-hot encoding of the real label. Predict probabilities for the model.

[0032] Evaluation metrics: Model performance is evaluated using metrics such as overall accuracy (OA), mean intersection-over-union ratio (mIoU), and F1 score. These metrics are calculated based on the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in the confusion matrix.

[0033] Overall accuracy (OA):

[0034]

[0035] Mean Intersection over Union (mIoU):

[0036]

[0037] F1 score:

[0038]

[0039] Precision and Recall are calculated as follows:

[0040] ,

[0041] Beneficial effects

[0042] The beneficial effects of this invention are as follows:

[0043] 1. High-precision cloud detection: By effectively fusing global and local contextual information through a multi-scale cross-attention module, the detection accuracy of thin clouds, fragmented clouds and cloud shadows is significantly improved, especially in complex terrain backgrounds.

[0044] 2. Multi-scale feature utilization: The multi-scale features extracted by the hybrid feature encoder are fused with the decoder through skip connections, which not only preserves deep semantic information but also restores shallow detail information, thus improving the accuracy of segmentation boundaries.

[0045] 3. Strong robustness: Transformer's global modeling capability makes the model more adaptable to different cloud types and land surface types, reducing misclassification of high albedo features.

[0046] 4. High degree of automation: The end-to-end deep learning model realizes the automatic conversion from raw images to cloud detection masks without manual intervention.

[0047] 5. Good scalability: The model architecture can be easily extended to other satellite data sources and other semantic segmentation tasks. Attached Figure Description

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Figure 1. Schematic diagram of the overall network structure of the TransUNet model;

[0050] Figure 2. Detailed structural diagram of the multi-scale channel spatial attention module;

[0051] Figure 3. Schematic diagram of the hybrid feature encoder structure;

[0052] Figure 4. Schematic diagram of progressive feature decoding block structure;

[0053] Figure 5. Training process and data preprocessing flowchart; Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] 1. Data Preparation and Preprocessing

[0056] Acquire multispectral images (blue, green, red, and near-infrared bands) from the Gaofen-1 (GF-1) satellite and pixel-level manually annotated ground truth values ​​(including four categories: clear sky, thick clouds, thin clouds, and cloud shadows). For example... Figure 5 As shown, after preprocessing the imagery including radiometric calibration and atmospheric correction, it is cropped into 512×512 pixel patches. Data augmentation techniques such as random flipping and rotation are then applied online to the training set patches.

[0057] 2. Detailed Explanation of Model Architecture

[0058] like Figure 1 As shown, the TransUNet model adopts an encoder-decoder structure.

[0059] 2.1 Hybrid Feature Encoder (corresponding to) Figure 3 )

[0060] The encoder is built based on an improved Mix Vision Transformer (MiT). The input is a 512×512×4 image patch, which is first passed through a 7×7 convolutional layer (stride 2) for preliminary feature extraction and downsampling, resulting in a 256×256×64 feature map.

[0061] Subsequently, the feature maps undergo four encoding stages (Stage 1-Stage 4). Each stage contains multiple improved Transformer blocks, the core of which is an efficient multi-head self-attention (EMHA) and a hybrid feedforward network (Mix-FFN).

[0062] Overlapping Patch Embedding: At the beginning of each stage, the feature map is segmented into a sequence of patches using overlapping convolutions. Let the input be... If the kernel size is K and the stride is S, then this operation can be represented as:

[0063]

[0064] Efficient Multi-Head Attention (EMHA): To reduce computational cost, spatial downsampling (scaling factor R) is performed on the keys (K) and values ​​(V). Attention calculation follows the formula:

[0065]

[0066] Mix-FFN: Uses 3×3 depthwise convolutions to provide location information, as shown in the formula:

[0067]

[0068] The feature maps output from the four stages are labeled C1 (128×128×64), C2 (64×64×128), C3 (32×32×320), and C4 (16×16×512), with decreasing resolution and enhanced semantic information.

[0069] 2.2 Multi-scale cross-self-attention module (corresponding to) Figure 2 )

[0070] This module is the core of the model, and its structure is as follows: Figure 2 As shown. The feature map C4 of the deepest layer of the encoder is used as input.

[0071] Multi-scale context generation: C4 is input into three parallel average pooling layers with pooling kernel sizes (downsampling factors) of 1×1 (identity), 4×4, and 8×8, respectively, to generate three sets of features at different scales as keys (K_s) and values ​​(V_s).

[0072] Cross-attention calculation: The original resolution C4 is used as the query (Q). The attention between Q and each group K_s is calculated separately and then aggregated with the corresponding V_s, using the following formula:

[0073]

[0074] Feature fusion: The three sets of outputs at different scales are upsampled to the size of C4, summed, and then integrated through a 1×1 convolution. Finally, a residual connection is made with the input C4 to output the enhanced features. This integrates global and multi-scale local contexts.

[0075] 2.3 Progressive Feature Decoder (corresponding to) Figure 4 )

[0076] The decoder consists of four decoding blocks (structured as follows) Figure 4 This process, which involves gradually restoring spatial resolution, is called "( )".

[0077] Upsampling and skip connections: Each decoding block first upsamples the features of the previous layer by 2 times (bilinear interpolation), and then concatenates them with the feature maps (C3, C2, C1) of the encoder at the corresponding scale through skip connections in the channel dimension.

[0078] Feature integration: The concatenated features are fused and refined through two 3×3 convolutional layers (containing BatchNorm and ReLU activation in between). The formula can be simply described as follows:

[0079]

[0080] Deep supervision: During the training phase, an auxiliary classification head is introduced into the intermediate layer of the decoder (e.g., the output corresponding to C3 and C2 scales) to calculate the loss together with the final output, thereby accelerating convergence.

[0081] 3. Model Training and Optimization

[0082] Loss function: A combination of weighted cross-entropy loss (WCE) and Dice loss is used to address the pixel imbalance issue in categories such as cloud and clear sky. The total loss is:

[0083]

[0084] , where λ is the balancing weight.

[0085] Optimization details: The AdamW optimizer was used with an initial learning rate of 6e-5, employing a cosine annealing learning rate scheduling strategy. The batch size was set to 8, and training was performed on four NVIDIA L20 GPUs for approximately 80,000 iterations.

[0086] 4. Model Lightweighting and Deployment

[0087] To facilitate practical engineering deployment, this invention also employs knowledge distillation technology, using the aforementioned TransUNet model as a teacher model to guide the learning of a more structurally simplified student model. While maintaining an accuracy loss of no more than 2%, the student model reduces the number of parameters by 60% and increases inference speed by approximately 3 times, making it more suitable for deployment in resource-constrained environments.

Claims

1. A cloud detection method for Gaofen-1 remote sensing images based on a multi-scale cross-attention Transformer network, characterized in that, Includes the following steps: S1: Acquire multispectral remote sensing images from the Gaofen-1 satellite, preprocess them, and crop them into fixed-size tiles; S2: Input the preprocessed image patches into the trained TransUNet model; S3: The TransUNet model extracts multi-scale hierarchical features of the input image through its hybrid feature encoder; S4: Input the deepest feature map output by the encoder into the multi-scale cross self-attention module to fuse global and multi-scale contextual information; S5: The progressive feature decoder is used to upsample the fused features and fuse them with the features of the corresponding level of the encoder through skip connections to gradually restore the spatial resolution. S6: Output a probability map of each pixel belonging to one of the four categories: "clear sky", "thick clouds", "thin clouds" and "cloud shadows", and generate the final cloud detection mask through threshold segmentation or Argmax operation.

2. The method according to claim 1, characterized in that, The preprocessing in step S1 includes radiometric correction and geometric correction.

3. The method according to claim 1, characterized in that, The fixed-size block in step S1 is 512×512 pixels.

4. The method according to claim 1, characterized in that, The hybrid feature encoder of the TransUNet model is built on an improved visual Transformer (ViT) architecture. The first layer of the convolutional network is adapted to the blue, green, red and near-infrared four band inputs of the Gaofen-1 image and includes four downsampling stages, each of which is composed of multiple stacked Transformer blocks.

5. The method according to claim 1, characterized in that, The multi-scale cross-attention module downsamples the deepest feature map by performing multiple parallel average pooling operations at different scales, generating multiple sets of key-value (KV) pairs, while retaining the original resolution feature map as a query (Q) for cross-attention calculation.

6. The method according to claim 1, characterized in that, The progressive feature decoder consists of multiple decoding blocks. Each decoding block upsamples the features of the previous layer and then fuses them with the feature map of the encoder at the corresponding scale through skip connections. The fusion process first concatenates the encoder features and the upsampled decoder features in the channel dimension, and then integrates the features through a convolutional layer.

7. The method according to claim 1, characterized in that, The classification output head in step S6 consists of a 1×1 convolutional layer and a Softmax activation function, which maps the features to the probability that each pixel belongs to one of the four categories.

8. The method according to claim 1, characterized in that, The model training uses the cross-entropy loss function, combined with a combination of weighted cross-entropy loss (WCE) and Dice loss, to address the problem of imbalanced number of pixels in each class.

9. A cloud detection system for Gaofen-1 remote sensing images based on a multi-scale cross-attention Transformer network, characterized in that, include: Data acquisition module: used to acquire multispectral remote sensing images from the Gaofen-1 satellite; Preprocessing module: Used to perform radiometric and geometric corrections on the image, and to crop it into fixed-size blocks; Model inference module: contains a trained TransUNet model for performing the method described in any one of claims 1-8; Results output module: Used to output cloud detection mask and classification probability map.

10. The system according to claim 9, characterized in that, It also includes a model training module for training the TransUNet model, using the AdamW optimizer with an initial learning rate of 6e-5 and a cosine annealing learning rate scheduling strategy.

Images