A three-branch method for cloud and cloud shadow segmentation in remote sensing images based on CNN, transformer and GCN

Abstract

This invention addresses the problems of insufficient local feature and global information modeling, difficulty in modeling long-distance pixel dependencies, and difficulties in model fusion and multi-scale feature integration in cloud and cloud shadow image segmentation in the field of remote sensing. It discloses a three-branch remote sensing image cloud and cloud shadow segmentation method based on CNN, Transformer, and GCN, relating to the field of remote sensing image processing. The method includes the following steps: S1, acquiring a dataset of clouds and cloud shadows composed of remote sensing images and preprocessing the dataset; S2, establishing a three-branch network model, which includes a GCN branch, a CNN branch, and a Transformer branch; S3, inputting the preprocessed cloud and cloud shadow dataset into the three-branch network model for model training; S4, using the trained three-branch network model to predict and generate mask images of clouds and cloud shadows. This invention has broad cross-domain application potential and can efficiently handle various image segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, specifically to a method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN. Background Technology

[0002] Cloud segmentation and cloud shadow segmentation are crucial in remote sensing, meteorology, and environmental monitoring. Cloud changes not only affect weather forecasts but also reflect climate change dynamics. Accurate detection can improve forecast reliability and assist in extreme weather warnings and environmental monitoring. Approximately 67% of the Earth's surface is covered by clouds, impacting natural disaster monitoring; accurate segmentation helps identify disaster-stricken areas and guide rescue efforts. Cloud shadow segmentation also plays an important role in land cover assessment and solar energy utilization. Traditional methods such as threshold segmentation and manual feature extraction, while easily optimized, are greatly affected by lighting conditions and have limited effectiveness in complex environments.

[0003] In recent years, Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in computer vision, achieving end-to-end training and improving prediction performance through deep learning. Long et al. proposed Fully Convolutional Networks (FCNs) for pixel-level classification, while Ronneberger et al.'s U-Net achieved accurate segmentation through symmetric structures and skip connections. Chen et al.'s DeepLab enhanced boundary recognition using dilated convolutions, Zhao et al.'s PSPNet integrated multi-scale features through pyramid pooling modules, Yu et al.'s BiseNet employed a bilateral architecture to extract details and semantic information, and Long et al.'s D-Unet broadened dilated convolution modules. Despite the excellent performance of these methods in many applications, CNNs use fixed-size filters for convolution operations, thus limiting their ability to model the correlations between distant pixels and making it difficult to capture long-range semantic information. This leads the network to focus more on local information in the input data while neglecting global contextual information, thereby affecting the model's overall understanding ability.

[0004] The Transformer has made significant progress in computer vision in recent years. Initially used in natural language processing, the Transformer effectively integrates global information through multi-head attention mechanisms, capturing relationships between pixels and enhancing the flexibility and efficiency of applications in complex scenes. Recent research has extended the Transformer to obtain global features. Wang et al. proposed the Pyramid-based VIT (PVT), introducing a pyramid structure; the Swin Transformer employs a hierarchical design and sliding window to improve efficiency and multi-scale processing capabilities; Wu et al.'s Convolutional Vision Transformer (CvT) improves performance by introducing convolution; and Hu et al. replaced the ViT in the dual-branch network with EdgeViT to improve inference speed. However, while Transformer models excel at capturing global contextual information, they lack effective perception of local features when processing image data, making it difficult to accurately capture subtle features and local information in images.

[0005] Existing methods typically combine CNNs with Transformers or Transformers with Generative Networks (GCNs) to overcome receptive field limitations and extract multi-scale information. However, CNN-based networks, due to their fixed receptive field size, struggle to adapt to features at different scales, impacting segmentation performance. While CNN-Transformer combinations can extract both local and global features, they primarily target Euclidean space data, making it difficult to directly utilize graph structures in remote sensing images. Furthermore, simple feature concatenation or weighted averaging can lead to information redundancy or loss, limiting performance improvement. CNN-GCN combinations, limited by their fixed neighborhood, struggle to capture long-distance dependencies, and feature fusion requires additional data transformation, increasing training costs. While Transformer-GCN combinations leverage the advantages of global dependency modeling and graph structure modeling, they neglect local details, resulting in blurred edges and category confusion, reducing segmentation accuracy. Summary of the Invention

[0006] To address the challenges in cloud and cloud shadow image segmentation in remote sensing, such as insufficient modeling of local features and global information, difficulties in modeling long-distance pixel dependencies, and challenges in model fusion and multi-scale feature integration, this invention aims to provide a three-branch remote sensing image cloud and cloud shadow segmentation method based on CNN, Transformer, and GCN. By integrating the advantages of different networks, this method can provide more powerful feature representation and better performance when processing cloud and cloud shadow segmentation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN includes the following steps:

[0009] S1. Obtain the cloud and cloud shadow dataset composed of remote sensing images, and preprocess the dataset;

[0010] S2. Establish a three-branch network model, which includes a CNN branch, a Transformer branch, and a GCN branch;

[0011] S3. Input the preprocessed cloud and cloud shadow dataset into the three-branch network model for model training;

[0012] S4. Use the trained three-branch network model to predict and generate mask images of clouds and cloud shadows.

[0013] To optimize the above technical solution, the specific measures also include:

[0014] Furthermore, S1 specifically includes:

[0015] S1.1, Acquire remote sensing images of clouds and cloud shadows from satellites;

[0016] S1.2, uniformly crop the original remote sensing images to the same size, label the remote sensing images, and divide all the cropped remote sensing images into training set and test set.

[0017] Furthermore, in S2, the three-branch network model also includes: a graph feature encoder, a dual-path pyramid stitching module, a graph feature decoder, and a detail enhancement upsampling module;

[0018] The graph feature encoder performs superpixel segmentation on the cloud and cloud shadow datasets, generating a node feature matrix Q and an adjacency matrix E, providing graph features and adjacency information for GCN branches;

[0019] The GCN branch extracts graph features based on the node feature matrix Q and the adjacency matrix E;

[0020] The cloud and cloud shadow datasets are processed through a hybrid backbone network containing CNN branches and Transformer branches to extract fine features and capture long-range dependencies. The CNN branches are used to extract shallow features, and the Transformer branches are used to extract deep features.

[0021] The CNN branch includes multiple network stages, and the Transformer branch also includes multiple network stages; the dual-path pyramid splicing module fuses the shallow features from each stage of the CNN branch and the deep features from each stage of the Transformer branch to obtain the fused features of each stage of the hybrid backbone network.

[0022] The graph feature decoder fuses the graph features extracted from the GCN branches with the fusion features from each stage of the hybrid backbone network;

[0023] The detail enhancement upsampling module takes the fusion features of each stage of the hybrid backbone network as input, and after upsampling and detail fusion processing, outputs a feature map with gradually restored resolution; the convolutional classification layer of the last detail enhancement upsampling module generates a mask image of clouds and cloud shadows.

[0024] Furthermore, the specific process by which the graph feature encoder performs superpixel segmentation on the cloud and cloud shadow dataset to generate the node feature matrix Q and adjacency matrix E is as follows:

[0025] The SLIC algorithm is used to divide the preprocessed cloud and cloud shadow images into superpixels based on color and texture information. Each superpixel consists of pixels with similar features. The cloud and cloud shadow images are of size H×W, where H and W are the height and width of the cloud and cloud shadow images, respectively. The result of superpixel segmentation is an H×W label matrix S, where each element represents the superpixel label S(x,y) to which the corresponding pixel belongs. A graph structure is constructed using the label matrix S, where each node represents a superpixel. The features of a superpixel include its color, texture, shape, and position information. The feature vector of a superpixel is represented as:

[0026] f i =[c i ,t i ,s i ,p i ]

[0027] In the formula, f i Let c be the feature vector of the i-th superpixel. i It is the color feature of the i-th superpixel, t i It is the texture feature of the i-th superpixel, s i p is the shape feature of the i-th superpixel. i It is the positional feature of the i-th superpixel;

[0028] The feature vectors of all superpixels constitute a node feature matrix Q, denoted as:

[0029] Q = [f1] T f2 T , ..., f i T , ..., f N T ] T

[0030] In the formula, f N Let be the feature vector of the Nth superpixel, with the superscript T indicating transpose;

[0031] The edges of the graph are constructed using the adjacency relationships between superpixels. If two superpixels S... i and S j If they are spatially adjacent, then an edge e is created in the graph. ij ;

[0032] Construct the adjacency matrix E of the graph, and the elements E in the adjacency matrix E. ij This indicates whether nodes i and j are connected.

[0033] Furthermore, the specific process by which the GCN branch extracts graph features based on the node feature matrix Q and the edge index matrix E is as follows:

[0034]

[0035]

[0036] in, Let E represent the output feature of node j in the first layer, ELU represent the activation function, V is the set of all nodes in the graph, and E represents the output feature of node j in the first layer. ij These are elements of the adjacency matrix E, indicating whether nodes i and j are connected. ij (1) W represents the attention weights of the first layer, indicating the importance of node j to node i. (1) f represents the first-layer learnable weight matrix, used for linear transformation of the input features. j Let be the j-th vector in the node feature matrix Q, representing the feature vector of the j-th superpixel. It is the feature representation of node i in the second layer, a ij (2) W represents the attention weights of the second layer, indicating the importance of node j to node i. (2) This represents the learnable weight matrix of the second layer, used to linearly transform the output features of the first layer;

[0037] The feature representations of all nodes in the second layer constitute the graph feature matrix F extracted by GCN. gcn .

[0038] Furthermore, the CNN branch has a total of 5 network layers. Each network layer contains a double convolution operation and a max pooling operation. The i-th network layer is expressed by the formula:

[0039] Output conv i =ReLU(BatchNorm2D(Conv2D) 3×3 (ReLU(BatchNorm2D(Conv2D 3×3 (X i ))))))

[0040] Output i =MaxPool2D(Output) conv i (kernel_size=2)

[0041] In the formula, Output conv i This is the output of a two-layer convolution operation. ReLU represents the ReLU activation function, BatchNorm2D represents batch normalization, used to standardize the convolution result, and Conv2D... 3×3 This indicates a two-dimensional convolution with a kernel size of 3×3, X i The input features are those of the i-th layer of the network structure; Output i This represents the output of the i-th layer of the network structure. MaxPool2D represents the max pooling operation, which is used for downsampling. kernel_size = 2 indicates that the size of the pooling kernel is 2×2.

[0042] Furthermore, the Transformer branch has a total of 4 network layers, and the expression of the i-th layer network structure is as follows:

[0043] Output' conv i =LayerNorm(Rearrange(Conv2D) i (X′ i )))

[0044] X i =LayerNorm(Output′) conv i )

[0045] Q i =to_q(X i ),K i =to_k(X i ),V i =to_v(X i )

[0046]

[0047] Attention_Output i =Attention_Weights i ·V i

[0048] Attention_Output i =Linear(Attention_Output) i ′)+X i

[0049] FFN(Attention_Outputi )=Linear_2(GELU(Linear_1(Attention_Output i )))+Attention_Output i

[0050] Transformer_Output i =FFN(Attention_Output) i )

[0051] In the formula, Output′ conv i This represents the feature map after convolution, rearrangement, and normalization operations in the i-th layer. `LayerNorm` indicates that layer normalization is performed on the rearranged output to ensure consistent mean and variance for each feature. `Rearrange` represents the rearrangement operation. `Conv2D` i Let X′ represent the convolution operation in the i-th layer. i X represents the input feature map of the i-th layer; i This represents the features after pre-normalization; Q i K i and V i These represent the query vector, key vector, and value vector, respectively, while to_q, to_k, and to_v represent the linear transformations of the corresponding query, key, and value, respectively. It is the scaling factor, softmax represents the softmax activation function, and Attention_Weights i Attention_Output represents the weight matrix in the i-th layer of the attention mechanism. i ′ represents the initial attention output of the i-th layer, Attention_Output i This represents the attention obtained by linearly projecting the initial attention output of the i-th layer and adding a residual connection. `Linear` indicates a linear transformation, and `FFN(Attention_Output)` represents the attention. i () represents a feedforward network, Linear_1 and Linear_2 represent the linear transformations of the first and second layers, respectively, GELU represents the activation function, and Transformer_Output represents the output. i This represents the final output of the i-th layer Transformer.

[0052] Furthermore, the specific process by which the dual-path pyramid stitching module fuses shallow features from each stage of the CNN branch and deep features from each stage of the Transformer branch is as follows:

[0053] shallow = ReLU(BN(Conv)3×3 (X shallow )))

[0054] deep = ReLU(BN(Conv) 3×3 (SPP(X deep ))))

[0055] fused = concat(shallow, deep)

[0056] out = ReLU(BN(Conv) 3×3 (fused)))

[0057] In the formula, shallow represents the shallow feature map after feature extraction, ReLU represents the activation function, BN represents the normalization layer, and Conv represents the normalization layer. 3×3 This represents a convolution with a kernel of 3, X shallow The shallow feature map represents the feature map from the CNN branch, deep represents the deep feature map after feature extraction, SPP represents the spatial pyramid pooling operation, used to capture multi-scale features, and X represents the shallow feature map from the CNN branch. deep This represents the deep feature map from the Transformer branch. fused represents the feature map after concatenating shallow and deep. concat represents the concatenation of two feature maps in the channel dimension. out represents the feature map output after processing by the activation function, which is the feature map obtained by fusing the two-path pyramid concatenation module.

[0058] Furthermore, the specific implementation process of the graph feature decoder is as follows:

[0059] The graph feature matrix extracted by GCN is F gcn ∈R N×F Where N is the number of superpixels, F is the feature dimension of each superpixel, and R... N×F It is an N×F dimensional real number field. The features extracted by GCN are mapped back to the image space using the label matrix of superpixels, resulting in the mapped spatialized GCN feature map F'(x,y,·); the symbol "·" represents all elements, that is, F'(x,y,·) represents the complete feature vector at the superpixel position (x,y);

[0060] Adjusting the dimensions of the mapped spatialized GCN feature map yields the dimension-adjusted GCN feature map F. reshape ;

[0061] The dimension-adjusted GCN feature map F is then concatenated using a stitching operation. reshape The feature map 'out' obtained by fusing with the dual-path pyramid stitching module is stitched along the channel dimension to obtain the final multi-channel feature map, as expressed by the following formula:

[0062] F fused =Concat(out,F) reshape )

[0063] In the formula, Concat represents the concatenation operation, and F fused This is the final multi-channel feature map.

[0064] Furthermore, the implementation process of the detail enhancement upsampling module is as follows:

[0065] Y'=Conv 3x3 (Y)

[0066] Y″ = PiexlShuffle(Y′)

[0067] Y″′=DW_Conv(Y″)

[0068] y = GlobalAvgPool(Y″′)

[0069] y′=Sigmoid(FC2(ReLU(FC1(y))))

[0070] Y″″=Y″′×y′

[0071] Y out =ReLU(Y″″)

[0072] In the formula, Y represents the input feature map, Y' represents the feature map after convolution, and Conv 3x3 Y″ represents a convolution with a kernel of 3, Y″ represents the feature map after pixel rearrangement, PixelShuffle represents the pixel rearrangement operation, Y″′ represents the feature map after depthwise separable convolution, DW_Conv represents depthwise separable convolution, y represents the feature map after global average pooling, GlobalAvgPool represents global average pooling, y' represents the feature map after fully connected layers and activation functions, Sigmoid represents the activation function, FC2 and FC1 represent the second and first fully connected layers respectively, and Y”” represents the feature map after applying attention weights. out This represents the final output feature map, and ReLU represents the activation function.

[0073] The beneficial effects of this invention are:

[0074] This invention offers significant advantages in processing the unique multi-scale and multi-shape features of remote sensing images, as well as preserving segmentation details and accuracy at high resolution. In particular, this method effectively addresses challenges in remote sensing images, such as high-resolution segmentation of complex features and full utilization of multi-scale information. Furthermore, thanks to its modular and flexible architecture, this method possesses broad cross-domain application potential and can efficiently handle various image segmentation tasks. Experimental results validate the effectiveness of this method; compared to existing technologies, the model shows significant improvements in both accuracy and the ability to handle complex scenes.

[0075] The three-branch network model in this invention uses CNN and Transformer branches for multi-scale feature extraction, capturing local texture information and long-range global dependencies of the image, respectively. The GCN branch enhances the graph structure information representation capability of features by modeling the structured relationships within the image and the dependencies between nodes. To address the incompatibility issue between CNN and GCN data structures, this invention designs a graph decoder-encoder architecture. First, superpixel segmentation is used to divide the image into regions, generating graph features and adjacency information, facilitating GCN extraction of superpixel-level features. Then, these features are mapped back to the pixel space and fused with the pixel-level features from the encoder stage to achieve collaborative feature representation. Furthermore, to strengthen the multi-scale feature fusion of CNN and Transformer, this invention proposes a dual-path pyramid stitching module. This module acquires multi-scale information through pyramid pooling and fuses shallow and deep features, significantly improving the network's multi-scale perception capability while also considering image details and semantic information. Furthermore, this invention incorporates a detail enhancement upsampling module to address the detail loss and edge blurring issues inherent in traditional upsampling methods. This module effectively enhances key feature representation while restoring image resolution, further improving the accuracy and quality of image reconstruction. Validated on datasets such as Biome 8, HRC-WHU, and SPARCS, this invention demonstrates superior performance compared to existing technologies. Attached Figure Description

[0076] Figure 1 This is a flowchart of the cloud and cloud shadow segmentation method according to an embodiment of the present invention.

[0077] Figure 2 This is the overall architecture diagram of the three-branch network model according to an embodiment of the present invention.

[0078] Figure 3 This is a CNN branch model diagram according to an embodiment of the present invention.

[0079] Figure 4 This is a Transformer branch model diagram according to an embodiment of the present invention.

[0080] Figure 5This is a graph feature encoder-graph feature decoder model diagram according to an embodiment of the present invention.

[0081] Figure 6 This is a GCN branch model diagram according to an embodiment of the present invention.

[0082] Figure 7 This is a diagram of the dual-path pyramid splicing module according to an embodiment of the present invention.

[0083] Figure 8 This is a diagram of the detail enhancement upsampling module according to an embodiment of the present invention. Detailed Implementation

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0085] Example 1

[0086] This invention proposes a three-branch remote sensing image cloud and cloud shadow segmentation method based on CNN, Transformer, and GCN. The overall process of this method is as follows: Figure 1 As shown, it includes the following steps:

[0087] S1. Obtain the cloud and cloud shadow dataset composed of remote sensing images, and preprocess the dataset; S1 specifically includes:

[0088] S1.1 Acquire remote sensing images of clouds and cloud shadows from Landsat-8, Sentinel-1, and Sentinel-2 satellites; To comprehensively evaluate the performance of the algorithm, the collected images cover a variety of different scenes, including urban areas, rural areas, mountainous areas, water bodies, and wastelands, ensuring that diverse cloud shadow images are included.

[0089] S1.2, due to the limited GPU memory capacity, the images must be cropped into smaller blocks to facilitate subsequent training and prediction. The original remote sensing images were uniformly cropped to a size of 256×256 pixels, and the cropped images were manually labeled. The Labelme tool was used to label each cropped image, with labels categorized into three types: clouds, cloud shadows, and background. The labeled pixels for clouds were [255, 255, 255], for thin clouds [200, 200, 200], for cloud shadows [0, 0, 0], and for the background [128, 128, 128]. The labeling process was crucial to ensure the accuracy of the labels and thus the predictive performance of the model. After filtering out unqualified images, 5900 labeled images were obtained. All the cropped remote sensing images were then divided into training and test sets in an 8:2 ratio.

[0090] S2. Establish a three-branch network model, such as Figure 2 As shown, the three-branch network model includes a GCN branch, a CNN branch, a Transformer branch, a graph feature encoder, a dual-path pyramid stitching module, a graph feature decoder, and a detail enhancement upsampling module;

[0091] Graph feature encoder-graph feature decoder model, such as Figure 5 As shown.

[0092] The graph feature encoder performs superpixel segmentation on the cloud and cloud shadow datasets, generating a node feature matrix Q and an adjacency matrix E, providing graph features and adjacency information for GCN branches. The specific process of the graph feature encoder performing superpixel segmentation on the cloud and cloud shadow datasets and generating the node feature matrix Q and adjacency matrix E is as follows:

[0093] The SLIC (Simple Linear Iterative Clustering) algorithm is used to divide the preprocessed cloud and cloud shadow images into superpixels based on color and texture information. Each superpixel consists of pixels with similar features. The cloud and cloud shadow images are of size H×W, where H and W are the height and width of the cloud and cloud shadow images, respectively. The result of superpixel segmentation is an H×W label matrix S, where each element represents the superpixel label S(x,y) to which the corresponding pixel belongs. A graph structure is constructed using the label matrix S, where each node represents a superpixel. The features of a superpixel include its color, texture, shape, and position information. The feature vector of a superpixel is represented as:

[0094] f i =[c i ,t i ,s i ,p i ]

[0095] In the formula, f iLet c be the feature vector of the i-th superpixel. i It is the color feature of the i-th superpixel, t i It is the texture feature of the i-th superpixel, s i p is the shape feature of the i-th superpixel. i It is the positional feature of the i-th superpixel;

[0096] The feature vectors of all superpixels constitute a node feature matrix Q, denoted as:

[0097] Q = [f1] T f2 T , ..., f i T , ..., f N T ] T

[0098] In the formula, f N Let be the feature vector of the Nth superpixel, with the superscript T indicating transpose;

[0099] The edges of the graph are constructed using the adjacency relationships between superpixels. If two superpixels S... i and S j If they are spatially adjacent, then an edge e is created in the graph. ij ;

[0100] Construct the adjacency matrix E of the graph, and the elements E in the adjacency matrix E. ij This indicates whether nodes i and j are connected.

[0101] GCN branches extract graph features based on the node feature matrix Q and the adjacency matrix E; the GCN branch model is as follows: Figure 6 As shown, the specific process of GCN branches extracting graph features based on the node feature matrix Q and the edge index matrix E is as follows:

[0102]

[0103] in, Let E represent the output feature of node j in the first layer, ELU represent the activation function, V is the set of all nodes in the graph, and E represents the output feature of node j in the first layer. ij These are elements of the adjacency matrix E, indicating whether nodes i and j are connected. ij (1) W represents the attention weights of the first layer, indicating the importance of node j to node i. (1) f represents the first-layer learnable weight matrix, used for linear transformation of the input features. j Let be the j-th vector in the node feature matrix Q, representing the feature vector of the j-th superpixel. It is the feature representation of node i in the second layer, a ij(2) W represents the attention weights of the second layer, indicating the importance of node j to node i. (2) This represents the learnable weight matrix of the second layer, used to linearly transform the output features of the first layer;

[0104] The feature representations of all nodes in the second layer constitute the graph feature matrix F extracted by GCN. gcn .

[0105] The cloud and cloud shadow datasets are processed through a hybrid backbone network containing CNN branches and Transformer branches to extract fine features and capture long-range dependencies. The CNN branches are used to extract shallow features, and the Transformer branches are used to extract deep features. The CNN branches include multiple network stages, and the Transformer branches also include multiple network stages.

[0106] The CNN branch has a total of 5 network layers. Each layer contains a double convolution operation and a max pooling operation. The i-th layer is as follows: Figure 3 As shown, this can be expressed by the formula:

[0107] Output conv i =ReLU(BatchNorm2D(Conv2D) 3×3 (ReLU(BatchNorm2D(Conv2D 3×3 (X i ))))))

[0108] Output i =MaxPool2D(Output) conv i (kernel_size=2)

[0109] In the formula, Output conv i This is the output of a two-layer convolution operation. ReLU represents the ReLU activation function, BatchNorm2D represents batch normalization, used to standardize the convolution result, and Conv2D... 3×3 This indicates a two-dimensional convolution with a kernel size of 3×3, X i The input features are those of the i-th layer of the network structure; Output i This represents the output of the i-th layer of the network structure. MaxPool2D represents the max pooling operation, which is used for downsampling. kernel_size = 2 indicates that the size of the pooling kernel is 2×2.

[0110] The Transformer branch has a total of 4 layers of network structure. The i-th layer network structure is as follows: Figure 4 As shown, the expression is as follows:

[0111] Output'conv i =LayerNorm(Rearrange(Conv2D) i (X′ i )))

[0112] X i =LayerNorm(Output′) conv i )

[0113] Q i =to_q(X i ),K i =to_k(X i ),V i =to_v(X i )

[0114]

[0115] Attention_Output i =Attention_Weights i ·V i

[0116] Attention_Output i =Linear(Attention_Output) i ′)+X i

[0117] FFN(Attention_Output i )=Linear_2(GELU(Linear_1(Attention_Output i )))+Attention_Output i

[0118] Transformer_Output i =FFN(Attention_Output) i )

[0119] In the formula, Output′ conv i This represents the feature map after convolution, rearrangement, and normalization operations in the i-th layer. `LayerNorm` indicates that layer normalization is performed on the rearranged output to ensure consistent mean and variance for each feature. `Rearrange` represents the rearrangement operation. `Conv2D` i Let X′ represent the convolution operation in the i-th layer. i X represents the input feature map of the i-th layer; i This represents the features after pre-normalization; Q i Ki and V i These represent the query vector, key vector, and value vector, respectively, while to_q, to_k, and to_v represent the linear transformations of the corresponding query, key, and value, respectively. It is the scaling factor, softmax represents the softmax activation function, and Attention_Weights i Attention_Output represents the weight matrix in the i-th layer of the attention mechanism. i ′ represents the initial attention output of the i-th layer, Attention_Output i This represents the attention obtained by linearly projecting the initial attention output of the i-th layer and adding a residual connection. `Linear` indicates a linear transformation, and `FFN(Attention_Output)` represents the attention. i () represents a feedforward network, Linear_1 and Linear_2 represent the linear transformations of the first and second layers, respectively, GELU represents the activation function, and Transformer_Output represents the output. i This represents the final output of the i-th layer Transformer.

[0120] The dual-path pyramid concatenation module fuses shallow features from each stage of the CNN branches and deep features from each stage of the Transformer branches to obtain fused features from each stage of the hybrid backbone network; the dual-path pyramid concatenation module is as follows: Figure 7 As shown, the specific process by which the dual-path pyramid stitching module fuses shallow features from each stage of the CNN branch and deep features from each stage of the Transformer branch is as follows:

[0121] shallow = ReLU(BN(Conv) 3×3 (X shallow )))

[0122] deep = ReLU(BN(Conv) 3×3 (SPP(X deep ))))

[0123] fused = concat(shallow, deep)

[0124] out = ReLU(BN(Conv) 3×3 (fused)))

[0125] In the formula, shallow represents the shallow feature map after feature extraction, ReLU represents the activation function, BN represents the normalization layer, and Conv represents the normalization layer. 3×3 This represents a convolution with a kernel of 3, X shallowThe shallow feature map represents the feature map from the CNN branch, deep represents the deep feature map after feature extraction, SPP represents the spatial pyramid pooling operation, used to capture multi-scale features, and X represents the shallow feature map from the CNN branch. deep This represents the deep feature map from the Transformer branch. fused represents the feature map after concatenating shallow and deep. concat represents the concatenation of two feature maps in the channel dimension. out represents the feature map output after processing by the activation function, which is the feature map obtained by fusing the two-path pyramid concatenation module.

[0126] The graph feature decoder fuses the graph features extracted from the GCN branches with the fused features from each stage of the hybrid backbone network; the specific implementation process of the graph feature decoder is as follows:

[0127] The graph feature matrix extracted by GCN is F gcn ∈R N×F Where N is the number of superpixels, F is the feature dimension of each superpixel, and R... N×F It is an N×F dimensional real number field. The features extracted by GCN are mapped back to the image space using the superpixel label matrix, resulting in the mapped spatialized GCN feature map F'(x,y,·). The symbol "·" represents all elements, i.e., F'(x,y,·) represents the complete feature vector at the superpixel position (x,y). The specific process is as follows:

[0128] The superpixel label matrix is ​​S, with a size of H×W, where H and W are the height and width of the cloud and cloud shadow images, respectively. The target feature map has a size of H. i ×W i H i and W i These are the height and width of the target feature map, respectively. The size of the label matrix is ​​adjusted using nearest neighbor interpolation.

[0129] S' = Resize(S,(H) i W i ))

[0130] Where S' represents the adjusted label matrix, Resize represents the nearest neighbor interpolation method.

[0131] Based on the adjusted label matrix S', assign a corresponding GCN feature to each pixel:

[0132] F'(x,y,·)=F gcn [S'(x,y),·]

[0133] Where F'(x,y,·) represents the mapped spatialized GCN feature map with a size of H. i ×W i×F, where the symbol · represents "all elements", that is, F'(x,y,·) represents the complete feature vector (of dimension F) at the superpixel position (x,y). gcn [S'(x,y),·] represents selecting the corresponding row (i.e. the feature vector of the superpixel) from the graph feature matrix extracted from GCN based on the adjusted superpixel label S'.

[0134] Adjusting the dimensions of the mapped spatialized GCN feature map yields the dimension-adjusted GCN feature map F. reshape The mapped spatialized GCN feature map F'(x,y,·) is transferred from H... i ×W i ×F is converted to channel priority format F×H i ×W i :

[0135] F reshape =Reshape(F'(x,y,·),(2,0,1))

[0136] Where Reshape represents the dimension adjustment operation, F reshape ∈N F×H′×W′ This represents the GCN feature map after dimension adjustment.

[0137] The dimension-adjusted GCN feature map F is then concatenated using a stitching operation. reshape The feature map 'out' obtained by fusing with the dual-path pyramid stitching module is stitched along the channel dimension to obtain the final multi-channel feature map, as expressed by the following formula:

[0138] F fused =Concat(out,F) reshape )

[0139] In the formula, Concat represents the concatenation operation, and F fused This is the final multi-channel feature map.

[0140] The detail enhancement upsampling module takes the fused features from each stage of the hybrid backbone network as input, and after upsampling and detail fusion processing, outputs a feature map with progressively restored resolution. The final detail enhancement upsampling module's convolutional classification layer generates mask images of clouds and cloud shadows. The detail enhancement upsampling module is as follows: Figure 8 As shown, the implementation process of the detail enhancement upsampling module is as follows:

[0141] Y'=Conv 3x3 (Y)

[0142] Y″ = PiexlShuffle(Y′)

[0143] Y″′=DW_Conv(Y″)

[0144] y = GlobalAvgPool(Y″′)

[0145] y′=Sigmoid(FC2(ReLU(FC1(y))))

[0146] Y″″=Y″′×y′

[0147] Y out =ReLU(Y″″)

[0148] In the formula, Y represents the input feature map, Y' represents the feature map after convolution, and Conv 3x3 Y″ represents a convolution with a kernel of 3, Y″ represents the feature map after pixel rearrangement, PixelShuffle represents the pixel rearrangement operation, Y″′ represents the feature map after depthwise separable convolution, DW_Conv represents depthwise separable convolution, y represents the feature map after global average pooling, GlobalAvgPool represents global average pooling, y' represents the feature map after fully connected layers and activation functions, Sigmoid represents the activation function, FC2 and FC1 represent the second and first fully connected layers respectively, and Y”” represents the feature map after applying attention weights. out This represents the final output feature map, and ReLU represents the activation function.

[0149] S3. Input the preprocessed cloud and cloud shadow dataset into the three-branch network model for model training;

[0150] The network training process is as follows:

[0151] (1) Set training parameters

[0152] During training, 12 samples are selected each time, with a maximum of 300 iterations. A poly LR strategy is used, with an initial learning rate of 0.001 and a power of 2.

[0153] (2) Forward Propagation and Loss Calculation

[0154] Forward Propagation: At the start of training, the original remote sensing images are input into the neural network for forward propagation. During this process, the input data is processed through multiple network layers, ultimately outputting the model's predictions. The network's output images are compared with manually labeled images to calculate the loss.

[0155] Loss function: The cross-entropy loss function is used to calculate the error between the output image and the manually labeled image. The cross-entropy loss function is very effective in multi-class classification tasks, measuring the difference between the predicted value and the true label. The formula is as follows:

[0156]

[0157] Where L represents the cross-entropy loss, y i It is the target tag. It is the output of the network prediction.

[0158] (3) Backpropagation and parameter update

[0159] Backpropagation: After calculating the loss, the chain rule (backpropagation) is used to propagate the loss back from the output layer to each subsequent layer. The gradient of each parameter is calculated based on its contribution to the total loss. Through this backpropagation process, the network adjusts its parameters, reducing output error.

[0160] Optimizer: Parameter updates are performed using the SDG optimizer. SGD is a common optimization algorithm that updates model parameters by calculating the gradient in each iteration. To accelerate model convergence, SGD employs a momentum strategy with a momentum coefficient set to 0.9. By incorporating momentum from previous updates, oscillations during parameter updates are avoided, promoting a smoother optimization process.

[0161] (4) Training process

[0162] In each iteration, the network will perform the above steps, including forward propagation, loss calculation, backpropagation, and parameter updates, until the maximum number of iterations (300) is reached. During training, the model performance can be tested periodically on the validation set to check whether the loss value gradually converges and whether the model's accuracy has reached the expected level.

[0163] (5) Save the optimal model and evaluation

[0164] Model saving: After training is complete, save the optimal model for subsequent testing and applications. Typically, the optimal model is saved based on the minimum loss or highest accuracy on the training and validation sets.

[0165] Testing and Evaluation: The trained model is evaluated using a test set, and the final test error, accuracy, and other metrics, such as miou (mean intersection over union), are calculated.

[0166] S4. Use the trained three-branch network model to predict and generate mask images of clouds and cloud shadows.

[0167] To comprehensively evaluate the performance of our method in cloud and cloud shadow segmentation tasks, we selected a series of widely accepted evaluation metrics, including precision (P), recall (R), F1 score, overall precision (OA), pixel precision (PA), average pixel precision (MPA), average crosslinking (MIoU), and frequency-weighted union-intersection (FWIOU). The formulas for each evaluation metric are as follows:

[0168]

[0169] In this context, true positives (TP) represent the number of correctly predicted cloud (cloud shadow) pixels, false positives (FP) represent the number of incorrectly predicted cloud (cloud shadow) pixels, and true negatives (TN) represent correctly classified non-cloud (non-cloud) pixels. False negatives (FN) represent incorrectly classified cloud (cloud shadow) pixels, k represents the number of categories (excluding background), and p... ii p represents the number of i. ij This represents the number of items belonging to category i but incorrectly predicted as category j.

[0170] To demonstrate the feasibility of the algorithm, it was compared with numerous cutting-edge semantic segmentation algorithms such as DeepLabv3+, PVT, CMTF, DBNet, and Unet, showing superior performance. Table 1 compares the evaluation metrics of different methods across various categories on the same dataset. This invention achieves the highest precision (P), recall (R%), and F1 score in thin cloud detection and cloud detection. In cloud shadow detection, this invention achieves the highest recall (R%) and F1 score, reaching 86.21% and 87.84% respectively. While its precision (P%) is not the highest, it is only slightly behind the best model. Table 2 compares the overall evaluation metrics of each model on the same dataset. As can be seen from the table, this method has the highest precision (P), F1 score, recall (R), average pixel precision (MPA), frequency-weighted intersection-over-union ratio (FWIOU), and average intersection-over-union ratio (MIoU) compared to other methods, reaching 91.21%, 91.09%, 91.11%, 89.83%, 84.01%, and 82.32%, respectively.

[0171] In summary, this invention achieves better results than other models in remote sensing image cloud and cloud shadow segmentation tasks.

[0172] Table 1 Comparison of evaluation metrics for each model on the dataset.

[0173]

[0174] Table 2 Comparison of overall evaluation metrics for each model on the dataset.

[0175]

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A CNN, Transformer and GCN-based three-branch remote sensing image cloud and cloud shadow segmentation method, characterized in that, Includes the following steps: S1. Obtain the cloud and cloud shadow dataset composed of remote sensing images, and preprocess the dataset; S2. Establish a three-branch network model, which includes a CNN branch, a Transformer branch, and a GCN branch; In S2, the three-branch network model further includes: a graph feature encoder, a dual-path pyramid stitching module, a graph feature decoder, and a detail enhancement upsampling module; The graph feature encoder performs superpixel segmentation on the cloud and cloud shadow datasets, generating a node feature matrix Q and an adjacency matrix E, providing graph features and adjacency information for GCN branches; The GCN branch extracts graph features based on the node feature matrix Q and the adjacency matrix E; The cloud and cloud shadow datasets are processed through a hybrid backbone network containing CNN branches and Transformer branches to extract fine features and capture long-range dependencies. The CNN branches are used to extract shallow features, and the Transformer branches are used to extract deep features. The CNN branch includes multiple network stages, and the Transformer branch also includes multiple network stages; the dual-path pyramid splicing module fuses the shallow features from each stage of the CNN branch and the deep features from each stage of the Transformer branch to obtain the fused features of each stage of the hybrid backbone network. The graph feature decoder fuses the graph features extracted from the GCN branches with the fusion features from each stage of the hybrid backbone network; The detail enhancement upsampling module takes the fusion features of each stage of the hybrid backbone network as input, and after upsampling and detail fusion processing, outputs a feature map with gradually restored resolution; the convolutional classification layer of the last detail enhancement upsampling module generates a mask image of clouds and cloud shadows. S3. Input the preprocessed cloud and cloud shadow dataset into the three-branch network model for model training; S4. Use the trained three-branch network model to predict and generate mask images of clouds and cloud shadows.

2. The CNN, Transformer and GCN-based three-branch remote sensing image cloud and cloud shadow segmentation method of claim 1, wherein, S1 specifically includes: S1.1, Acquire remote sensing images of clouds and cloud shadows from satellites; S1.2, uniformly crop the original remote sensing images to the same size, label the remote sensing images, and divide all the cropped remote sensing images into training set and test set.

3. The CNN, Transformer and GCN-based three-branch remote sensing image cloud and cloud shadow segmentation method of claim 1, wherein, The graph feature encoder performs superpixel segmentation on the cloud and cloud shadow datasets to generate the node feature matrix Q and adjacency matrix E. The specific process is as follows: The SLIC algorithm is used to divide the preprocessed cloud and cloud shadow images into superpixels based on color and texture information. Each superpixel consists of pixels with similar features. The cloud and cloud shadow images are H×W in size, where H and W are the height and width of the cloud and cloud shadow images, respectively. The result of superpixel segmentation is an H×W label matrix S, where each element represents the superpixel label to which the corresponding pixel belongs. A graph structure is constructed using a label matrix S, where each node represents a superpixel. The features of a superpixel include its color, texture, shape, and location information. The feature vector of a superpixel is represented as: In the formula, Let i be the feature vector of the i-th superpixel. It is the first i Color features of each superpixel It is the first i Texture features of a superpixel It is the first i The shape features of each superpixel It is the first i The location features of each superpixel; The feature vectors of all superpixels constitute a node feature matrix Q, denoted as: In the formula, Let be the feature vector of the Nth superpixel, with the superscript T indicating transpose; The edges of the graph are constructed using the adjacency relationships between superpixels. If two superpixels... and If they are spatially adjacent, then an edge is created in the graph. ; Constructing the adjacency matrix of the graph E Adjacency matrix E elements in E ij Represents a node and Whether they are connected.

4. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The specific process by which the GCN branch extracts graph features based on the node feature matrix Q and the edge index matrix E is as follows: in, Represents a node The output features of the first layer, This represents the activation function. V It is the set of all nodes in the graph. E ij It is an adjacency matrix E The element represents a node. and Are they connected? This represents the attention weights of the first layer, and represents the nodes. right The importance of This represents the first-layer learnable weight matrix, used for linearly transforming the input features. Let be the j-th vector in the node feature matrix Q, representing the th . j The feature vector of each superpixel It is a node In the feature representation of the second layer, This represents the attention weights of the second layer, and represents the nodes. right The importance of This represents the learnable weight matrix of the second layer, used to linearly transform the output features of the first layer; The feature representations of all nodes in the second layer constitute the graph feature matrix extracted by the GCN. .

5. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The CNN branch has a total of 5 network layers. Each layer contains a double convolution operation and a max pooling operation. The layered network structure can be expressed by the following formula: In the formula, It is the output of a double convolution operation, and ReLU represents the ReLU activation function. This indicates the batch normalization operation, used to standardize the convolution results. This indicates a two-dimensional convolution with a kernel size of . , It is the first Input characteristics of layered network structures; Indicates the first i The output of the layered network structure, This indicates a max pooling operation, used for downsampling. The size of the pooling kernel is represented as .

6. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The Transformer branch has a total of 4 layers of network structure. The expression of the i-th layer network structure is as follows: In the formula, Indicates the first The feature map of the layer after convolution, rearrangement, and normalization operations. This indicates that layer normalization is performed on the rearranged output to ensure that the mean and variance of each feature are consistent. This indicates a rearrangement operation. Indicates the first Convolution operations of layers, Indicates the first The input feature map of the layer; This indicates features that have undergone pre-normalization. , and These represent the query vector, key vector, and value vector, respectively. , and These represent linear transformations of the corresponding query, key, and value, respectively. It represents the scaling factor, and softmax represents the softmax activation function. Indicates the first The weight matrix in the layer attention mechanism Indicates the first Initial attention output of the layer, Indicates the first The layer linearly projects the initial attention output and adds residual connections to obtain the attention. "Linear" represents the linear transformation. Indicates a feedforward network. and These represent the linear transformations of the first and second layers, respectively, and GELU represents the activation function. Indicates the first The final output of the Transformer layer.

7. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The specific process by which the dual-path pyramid stitching module fuses shallow features from each stage of the CNN branch and deep features from each stage of the Transformer branch is as follows: In the formula, shallow represents the shallow feature map after feature extraction, ReLU represents the activation function, and BN represents the normalization layer. This indicates a convolution with a kernel of 3. denoted as shallow feature map from CNN branch, 'deep' as deep feature map after feature extraction, and 'SPP' as spatial pyramid pooling operation used to capture multi-scale features. This represents the deep feature map from the Transformer branch. fused represents the feature map after concatenating shallow and deep. concat represents the concatenation of two feature maps in the channel dimension. out represents the feature map output after processing by the activation function, which is the feature map obtained by fusing the two-path pyramid concatenation module.

8. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The specific implementation process of the graph feature decoder is as follows: The graph feature matrix extracted by GCN is ,in, It is the number of superpixels. It is the feature dimension of each superpixel. It is an N×F dimensional real number field. The features extracted by GCN are mapped back to the image space using the label matrix of superpixels, resulting in the mapped spatialized GCN feature map. ;symbol" " represents all elements, that is Indicates superpixel position The complete feature vector at the location; Adjusting the dimensions of the mapped spatialized GCN feature map yields the dimension-adjusted GCN feature map. ; The dimension-adjusted GCN feature map is then concatenated. The feature map 'out' obtained by fusing with the dual-path pyramid stitching module is stitched along the channel dimension to obtain the final multi-channel feature map, as expressed by the following formula: In the formula, Concat represents the concatenation operation. This is the final multi-channel feature map.

9. The method for segmenting clouds and cloud shadows in three-branch remote sensing images based on CNN, Transformer, and GCN as described in claim 1, characterized in that, The implementation process of the detail enhancement upsampling module is as follows: In the formula, Indicates the input feature map, This represents the feature map after convolution. This indicates a convolution with a kernel of 3. This represents the feature map after pixel rearrangement processing; PixelShuffle indicates the pixel rearrangement operation. This represents the feature map after depthwise separable convolution processing. This represents depthwise separable convolution. This represents the feature map after global average pooling. `GlobalAvgPool` indicates global average pooling. This represents the feature map processed by the fully connected layer and activation function. Sigmoid represents the activation function, and FC2 and FC1 represent the second and first fully connected layers, respectively. This represents the feature map after attention weights have been applied. This represents the final output feature map, and ReLU represents the activation function.

Images