Converter-based high-resolution remote sensing image cloud detection method

Abstract

The invention relates to a high-resolution remote sensing image cloud detection method based on a converter, and the method combines a pyramid visual converter PVT with a circulation refinement architecture, and is specially used for a cloud detection task. The unique combination utilizes the capacity of capturing multi-scale features by PVT, and the resolution and quality of the features are improved through cyclic refinement. The method obviously overcomes the problem of detail degradation in cloud segmentation. The innovative method highlights the potential of combining the PVT and the cyclic refinement architecture, especially in the aspects of high-resolution feature extraction and detailed segmentation.

Description

Technical Field

[0001] The invention relates to a cloud detection method, and in particular to a high-resolution remote sensing image cloud detection method based on a converter. Background Art

[0002] In remote sensing, cloud detection is crucial for accurate analysis of satellite and aerial imagery. Literature suggests that clouds cover approximately 67% of the Earth's surface at any given time. Such pervasive cloud cover significantly degrades the quality of data extracted from these images, primarily by obscuring the Earth's surface, leading to potential errors in measurements or interpretations. Therefore, developing and implementing effective cloud detection techniques is essential to maintain data integrity and mitigate the effects of clouds obscuring surface features. This is particularly important in applications that require accurate surface observations, such as land cover classification, agricultural monitoring, and urban planning.

[0003] Cloud detection methods include various techniques. Threshold-based methods use specific spectral bands, especially visible and infrared, to set thresholds. Since clouds are usually brighter in these bands than the Earth's surface, pixels exceeding the predefined brightness value can be accurately identified as clouds. In addition, spectral indices like the Normalized Difference Snow Index (NDSI) and the Normalized Difference Vegetation Index (NDVI) provide an alternative strategy. These indices use the unique spectral characteristics of clouds, snow, and vegetation to effectively distinguish them.

[0004] In recent years, the application of machine learning and deep learning (DL) algorithms to cloud detection has attracted great attention due to their ability to autonomously learn from training datasets and accurately distinguish between cloudy and clear pixels. Among various techniques, methods based on convolutional neural networks (CNNs) have proven to be particularly effective. Their proficiency in capturing spatial hierarchical structures in images demonstrates the strong ability of CNNs to identify and learn spatial features of cloud images with extraordinary accuracy. However, despite their effectiveness in identifying spatial features, CNNs often fall short in understanding global image characteristics. This limitation highlights an important area of ​​research and emphasizes the need for more advanced algorithms to provide a more comprehensive understanding of image data. Summary of the invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: the technical problem of detail degradation in cloud segmentation.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: a high-resolution remote sensing image cloud detection method based on a converter, comprising the following steps:

[0007] S1: construct an image cloud detection model, which includes a pyramid visual transformer PVT, a channel attention module CAB, a graph fusion module and a decoder module;

[0008] Pyramid Visual Transformer PVT is used as the backbone network architecture. The image X is input into PVT, and PVT divides the image X into a multi-scale feature map FM. i ;

[0009] S2: Using channel attention module CAB for FM i Adjust FD t =CAB(FM i );

[0010] In the first iteration, the initial fusion feature map FD is obtained 0 =CAB(FM1);

[0011] S3: For the current number of cycles t, the current feature map FM i And the feature map FD output after the previous iteration t-1 Splice to get FM i ′;

[0012] S4: The output of S3 is further processed by the channel attention module Then generate The features are input into the convolution module to obtain FD t , after each round, output the current feature FD t Continue to update the prediction as input for the next round of loop;

[0013] S5: The final feature map FD is passed through the decoder module T Decoded into the final cloud detection result P i , then calculate the corresponding loss in They represent the weighted intersection loss of the i-th prediction result and the weighted binary cross entropy loss of the i-th prediction result respectively;

[0014] S6: The final four-round prediction results are processed by the graph fusion processing module to generate the final cloud detection result P that integrates all the prediction information. final , then calculate the total loss in in, and Represent the fusion results P final The weighted intersection loss and weighted binary cross entropy loss of Calculate the gradient, and then use the AdamW optimizer to update the parameters of the image cloud detection model according to the gradient. When the number of loops reaches the total number of loops T, the loop ends and the optimal image cloud detection model is obtained.

[0015] S7: For a new image, input it into the optimal image cloud detection model and output the detection result.

[0016] Furthermore, in S1, PVT is used as a backbone network architecture for feature extraction, and feature maps of different scales are generated in each stage by extracting multi-scale feature maps from the input image X. Each stage includes a core structure, which includes overlapping patch embedding and a Transformer encoder;

[0017] Each stage of PVT calculates feature maps of different scales through the formula, the formula is as follows:

[0018] FM i =PVT i (X),i=1,2,3,4.

[0019] Output feature map FM i The size of is gradually reduced with the depth of the network.

[0020] Furthermore, in S2 and S4, the specific process of processing features by the channel attention module is as follows: it aims to enhance the feature response of important channels.

[0021] First, for FM i First use Global Average Pooling (GAP) to generate channel importance scores:

[0022] g i =GAP(F)

[0023] Where F represents FM in S2 i or FM in S4 i ′;

[0024] Secondly, through a fully connected layer, FM i g i Converted to the channel weight W for each channel i :

[0025] W i =σ(W i ′·g i )

[0026] Where σ represents the activation function Sigmoid.

[0027] Finally, use Wi Weight each channel of F to get

[0028] Furthermore, in S3, the goal of fusion is to combine the feature graph of the current cycle with the result of the previous cycle so that the model can be gradually optimized and the prediction can be refined. The specific formula is:

[0029] FM i ′=cat(FM i ,FD t-1 )

[0030] Where cat(·) represents the concatenation operation in the channel dimension, t∈[1,T], where T represents the total number of loops, which is a preset hyperparameter.

[0031] Furthermore, the S4 convolution module operation is defined as:

[0032]

[0033] Where Conv(·) represents a standard convolution operation.

[0034] Furthermore, the workflow of the S5 decoder module is as follows:

[0035] First, an upsampling operation is performed: the feature map is gradually upsampled to the same resolution as the input image.

[0036] FM up =Upsample(FD T ), where Upsample represents an upsampling operation.

[0037] Secondly, through convolution decoding, the multi-channel feature map is decoded into a binary cloud detection map P:

[0038] P = σ(Conv(FM up ));

[0039] Where σ represents the activation function Sigmoid.

[0040] Furthermore, for the weighted intersection loss in S5 It is used to measure the degree of regional overlap between the predicted results and the true labels, focusing on areas that are difficult to segment (such as edges and small objects). The calculation is as follows:

[0041]

[0042] Where T j and P j denote the true label and predicted value of pixel j, respectively, and w jrepresents the weight of pixel j.

[0043] For the weighted binary cross entropy loss in S5 It measures the difference between the predicted probability distribution and the true label, helping the model to perform accurate classification at the pixel level. The calculation is as follows:

[0044]

[0045] Furthermore, the workflow of the S6 graph fusion processing module is as follows:

[0046] 1) Flatten the prediction output results: In the S5 stage, four different prediction results P1, P2, P3, and P4 are generated. Each result P i , i = 1, 2, 3, 4 is a two-dimensional feature map, which represents the prediction values ​​of different pixels. The prediction results of these different stages are flattened into node representations. For each pixel, the prediction values ​​of the four stages are combined to form a feature vector. Assuming that there are N pixels in total, the feature vector (node ​​feature) of each pixel is expressed as:

[0047] h n =[P1(n),P2(n),P3(n),P4(n)] T ,n=1,2,…N

[0048] Where P i (n) represents the prediction result of pixel n in the i-th stage;

[0049] Finally, we get a node feature matrix H with a shape of N×4, which means that H has N nodes and each node has 4 features:

[0050] H=[h1,h2,…,h N ] T

[0051] where h N It represents the predicted feature vector of the Nth pixel in multiple rounds of refinement iterations.

[0052] 2) Calculation of adjacency matrix: Construct adjacency matrix A to represent the similarity between nodes;

[0053]

[0054] Among them, the adjacency matrix A is an N×N matrix, in which the elements A ij Represents the similarity between node i and node j;

[0055] 3) Graph convolution operation: The node features are updated and fused through the graph convolution network GCN. The formula of graph convolution is as follows:

[0056] H′=σ(D -1 AHW)

[0057] Among them, H′ is the updated node feature matrix with a shape of N×d, d represents the feature dimension after convolution, A represents the adjacency matrix, D is the degree matrix representing the degree of each node, which is used to normalize the adjacency matrix, W is the trainable weight matrix used to adjust the node features, and σ is the ReLU activation function;

[0058] 4) Multi-layer graph convolution: Assume that after L layers of graph convolution operations, the final node feature matrix is: H L =σ(D - 1 AH L-1 W L-1 )

[0059] Among them, H L-1 , W L-1 They represent the node feature matrix and weight matrix of the graph convolutional network at layer L-1 respectively;

[0060] 5) Final fusion prediction results:

[0061] After completing the graph convolution operation, the node feature matrix H L The prediction result needs to be restored to a two-dimensional form; each node feature corresponds to the final prediction result of a pixel point; the final output is:

[0062] P final = reshape(H L )

[0063] where reshape is the flattened node feature matrix H L Restore the shape to the original image.

[0064] Compared with the prior art, the present invention has at least the following advantages:

[0065] This paper introduces a novel approach combining the Pyramid Visual Transformer (PVT) with a recurrent refinement architecture specifically for the cloud detection task. This unique combination exploits the ability of PVT to capture multi-scale features and improves the resolution and quality of these features through recurrent refinement. This approach significantly overcomes the problem of detail degradation in cloud segmentation. This innovative approach makes a significant contribution to the field of cloud detection, both theoretically and methodologically. It highlights the potential of combining PVT with a recurrent refinement architecture, especially for high-resolution feature extraction and detailed segmentation.

[0066] In order to overcome the detail degradation problem in cloud segmentation, the present invention designs the following five mechanisms to solve it:

[0067] 1. Multi-round refinement cycle mechanism: Through multiple rounds of iteration, features and predictions are gradually refined, especially in detail areas, prediction errors are gradually corrected to ensure that details are not lost in the first prediction.

[0068] 2. Channel Attention Mechanism: CAB enables the model to dynamically adjust the weights of different channels, enhance attention to key details, and prevent the model from ignoring the details of the cloud.

[0069] 3. Graph fusion module: By integrating multiple prediction results through graph convolution, the model can make full use of the information in multiple rounds of predictions for correction and optimization, ensuring that details are preserved and enhanced in multiple rounds of fusion.

[0070] 4. Multi-scale feature fusion: Feature fusion at different scales helps the model process large-scale clouds and small cloud structures at the same time, ensuring that the model captures detailed information.

[0071] 5. Feedback and error correction mechanism: Each round of feedback mechanism helps the model continuously optimize the processing of complex cloud boundaries and small cloud clusters to prevent detail degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is the overall framework of the present invention.

[0073] Figure 2 For qualitative comparison of experimental results. DETAILED DESCRIPTION

[0074] The present invention is described in further detail below.

[0075] In this work, we aim to address persistent challenges in this area by introducing a novel Transformer-based approach. Our approach employs the Pyramid Visual Transformer (PVT) as the primary feature extraction network, specifically chosen to improve the ability to extract high-resolution features - a known limitation in traditional CNN architectures. The choice of PVT over other Transformer architectures is based on its ability to efficiently balance computational demands when processing high-resolution inputs. This efficiency is achieved through the unique design of PVT, which features a progressively shrinking pyramid and a spatially reduced attention mechanism. Together, these elements enhance the computational model, enabling the extraction of high-quality features without the significant computational burden typically associated with processing high-resolution data.

[0076] We introduce the Recurrent Refinement Architecture (CRA) to extract multi-scale features while simultaneously generating multiple predictions using feature maps of different scales. Furthermore, this framework supports the development and refinement of both high- and low-resolution features through an iterative feedback loop, thus providing a comprehensive solution to the recognition challenge without compromising the ability to learn high-resolution features. To optimize performance and results, we end the optimization process by computing the loss for each of the five predictions generated by the CRA module and summing these individual losses to form the final loss function.

[0077] To achieve a high standard for cloud detection, our method adopts a unique strategy that leverages both low-resolution and high-resolution image features. This strategy is achieved by introducing a pyramid visual transformer (PVT) and a recurrent refinement architecture. The PVT is designed to extract multi-scale features, while the recurrent refinement architecture facilitates the generation of multiple predictions using feature maps at various scales. By combining these predictions, we compute a final loss that improves the accuracy of cloud detection. The PVT, recurrent refinement architecture, and final optimization method are discussed in detail below.

[0078] This paper highlights the significant performance advantages of the Transformer architecture while addressing a major limitation: high memory consumption. This issue becomes particularly critical in resource-constrained environments and may hinder the wider application of Transformers. To address this issue, we adopt the Pyramid Visual Transformer (PVT) as our main backbone network architecture, which aims to overcome the memory limitations of traditional Transformer models without sacrificing performance.

[0079] After completing the recurrent refinement architecture, we obtain four prediction results from different refinement cycles, as well as the final prediction processed by the graph fusion module. Each prediction demonstrates the precise refinement of feature extraction and fusion at each stage, ensuring high accuracy and robustness for the task. This innovative approach not only enhances the interaction between features at different scales, but also significantly improves the prediction accuracy in edge regions.

[0080] The training process of the model is as follows:

[0081] We used the PyTorch framework. The computational experiments were conducted on a single NVIDIA GTX 4090 GPU, covering both the training and testing phases. During the data input phase, we kept the original resolution of the dataset images at 1920 × 720 pixels to demonstrate the ability of our model to process high-resolution images.

[0082] For optimization, we chose AdamW, a variant of the Adam optimizer that introduces a separate weight decay component to enhance the ability to handle typical optimization challenges of Transformer-based models. We initialized the learning rate to 0.0001 and implemented a dynamic learning rate scheduling scheme. Specifically, our schedule was to reduce the learning rate by a factor of 0.1 every 20 epochs to gradually reduce the learning rate and stabilize the training. Our training schedule lasted for 150 epochs with a batch size of 8, ensuring that the model was exposed to diverse data samples and effectively learned to generalize from the training dataset.

[0083] Loss function:

[0084] In order to improve the accuracy of the predicted images generated by the Pyramid Visual Transformer (PVT) and the Cycle Refinement architecture, it is crucial to design an effective loss function. The loss function formula we use is as follows:

[0085]

[0086] in Respectively represent the weighted intersection loss of the i-th prediction result and the weighted binary cross entropy loss of the i-th prediction result. When it is less than the set threshold.

[0087] By using the Pyramid Visual Transformer (PVT) and iterative refinement architecture, we have obtained five predictions with different levels of detail. Therefore, for these five predictions, we need to calculate It represents the final loss function:

[0088] in

[0089] A high-resolution remote sensing image cloud detection method based on a transformer comprises the following steps:

[0090] S1: Construct an image cloud detection model, which includes a pyramid visual transformer PVT, a channel attention module CAB, a graph fusion module and a decoder module.

[0091] Pyramid Visual Transformer PVT is used as the backbone network architecture. The image X is input into PVT, and PVT divides the image X into a multi-scale feature map FM. i .

[0092] S2: Using channel attention module CAB for FM i Adjust FD t =CAB(FM i ).

[0093] In the first iteration, the initial fusion feature map FD is obtained 0 =CAB(FM1).

[0094] S3: For the current number of cycles t, the current feature map FM i And the feature map FD output after the previous iteration t-1 Splice to get FM i ′.

[0095] S4: The output of S3 is further processed by the channel attention module Then generate The features are input into the convolution module to obtain FD t , after each round, output the current feature FD t Continue to update the prediction as input for the next round of loop.

[0096] S5: The final feature map FD is passed through the decoder module T Decoded into the final cloud detection result P i , then calculate the corresponding loss in They represent the weighted intersection loss of the i-th prediction result and the weighted binary cross entropy loss of the i-th prediction result respectively.

[0097] S6: The final four-round prediction results are processed by the graph fusion processing module to generate the final cloud detection result P that integrates all the prediction information. final , then calculate the total loss in in, and Represent the fusion results P final The weighted intersection loss and weighted binary cross entropy loss of The gradient is calculated, and then the parameters of the image cloud detection model are updated according to the gradient according to the AdamW optimizer. When the number of loops reaches the total number of loops T, the loop ends and the optimal image cloud detection model is obtained.

[0098] S7: For a new image, input it into the optimal image cloud detection model and output the detection result.

[0099] Specifically, in S1, PVT is used as the backbone network architecture for feature extraction, and each stage generates feature maps of different scales by extracting multi-scale feature maps from the input image X. Each stage includes a core structure, which includes overlapping patch embedding and a Transformer encoder;

[0100] Each stage of PVT calculates feature maps of different scales through the formula, the formula is as follows:

[0101] FM i =PVT i (X),i=1,2,3,4.

[0102] Output feature map FM i The size of is gradually reduced with the depth of the network.

[0103] Specifically, in S2 and S4, the specific process of the channel attention module processing features is as follows: it aims to enhance the feature response of important channels.

[0104] First, for FM i First use Global Average Pooling (GAP) to generate channel importance scores:

[0105] g i =GAP(F)

[0106] Where F represents FM in S2 i or FM in S4 i ′;

[0107] Secondly, through a fully connected layer, FM i g i Converted to the channel weight W for each channel i :

[0108] W i =σ(W i ′·g i )

[0109] Where σ represents the activation function Sigmoid.

[0110] Finally, use W i Weight each channel of F to get

[0111] Specifically, in S3, the goal of fusion is to combine the feature graph of the current cycle with the result of the previous cycle so that the model can be gradually optimized and the prediction can be refined. The specific formula is:

[0112] FM i ′=cat(FM i ,FD t-1 )

[0113] Where cat(·) represents the concatenation operation in the channel dimension, t∈[1,T], where T represents the total number of loops, which is a preset hyperparameter.

[0114] Specifically, the S4 convolution module operation is defined as:

[0115]

[0116] Where Conv(·) represents a standard convolution operation.

[0117] Specifically, the workflow of the S5 decoder module is as follows:

[0118] First, an upsampling operation is performed: the feature map is gradually upsampled to the same resolution as the input image.

[0119] FM up =Upsample(FD T ), where Upsample represents an upsampling operation.

[0120] Secondly, through convolution decoding, the multi-channel feature map is decoded into a binary cloud detection map P:

[0121] P = σ(Conv(FM up ));

[0122] Where σ represents the activation function Sigmoid.

[0123] Specifically, for the weighted intersection loss in S5 It is used to measure the degree of regional overlap between the predicted results and the true labels, focusing on areas that are difficult to segment (such as edges and small objects). The calculation is as follows:

[0124]

[0125] Where T j and P j denote the true label and predicted value of pixel j, respectively, and w j represents the weight of pixel j.

[0126] For the weighted binary cross entropy loss in S5 It measures the difference between the predicted probability distribution and the true label, helping the model to perform accurate classification at the pixel level. The calculation is as follows:

[0127]

[0128] Specifically, the workflow of the S6 graph fusion processing module is as follows:

[0129] 1) Flatten the prediction output results: In the S5 stage, four different prediction results P1, P2, P3, and P4 are generated. Each result P i, i = 1, 2, 3, 4 is a two-dimensional feature map, which represents the prediction values ​​of different pixels. The prediction results of these different stages are flattened into node representations. For each pixel, the prediction values ​​of the four stages are combined to form a feature vector. Assuming that there are N pixels in total, the feature vector (node ​​feature) of each pixel is expressed as:

[0130] h n =[P1(n),P2(n),P3(n),P4(n)] T ,n=1,2,…N

[0131] Where P i (n) represents the prediction result of pixel n in the i-th stage;

[0132] Finally, we get a node feature matrix H with a shape of N×4, which means that H has N nodes and each node has 4 features:

[0133] H=[h1,h2,…,h N ] T

[0134] where h N It represents the predicted feature vector of the Nth pixel in multiple rounds of refinement iterations.

[0135] 2) Calculation of adjacency matrix: Construct adjacency matrix A to represent the similarity between nodes;

[0136]

[0137] Among them, the adjacency matrix A is an N×N matrix, in which the elements A ij Represents the similarity between node i and node j;

[0138] 3) Graph convolution operation: The node features are updated and fused through the graph convolution network GCN. The formula of graph convolution is as follows:

[0139] H′=σ(D -1 AHW)

[0140] Among them, H′ is the updated node feature matrix with a shape of N×d, d represents the feature dimension after convolution, A represents the adjacency matrix, D is the degree matrix representing the degree of each node, which is used to normalize the adjacency matrix, W is the trainable weight matrix used to adjust the node features, and σ is the ReLU activation function;

[0141] 4) Multi-layer graph convolution: Assume that after L layers of graph convolution operations, the final node feature matrix is: H L =σ(D - 1 AH L-1 WL-1 )

[0142] Among them, H L-1 , W L-1 They represent the node feature matrix and weight matrix of the graph convolutional network at layer L-1 respectively;

[0143] 5) Final fusion prediction results:

[0144] After completing the graph convolution operation, the node feature matrix H L The prediction result needs to be restored to a two-dimensional form; each node feature corresponds to the final prediction result of a pixel point; the final output is:

[0145] P final = reshape(H L )

[0146] where reshape is the flattened node feature matrix H L Restore the shape to the original image.

[0147] Evaluation method:

[0148] To evaluate the performance of our model on the HRC_WHU dataset, we used five standard metrics commonly used in the field of image segmentation: mean absolute error (MAE), mean intersection over union (mIoU), pixel accuracy (PA), Dice coefficient, and F1 score. Each metric provides a unique perspective on accuracy and effectiveness, making the comparison with other models more balanced. MAE quantifies the average prediction error, mIoU evaluates the accuracy of the overlap between predicted and actual segments, and PA measures the proportion of correctly classified pixels. In addition, the Dice coefficient and F1 score evaluate the precision and recall of the model, respectively. Together, these metrics facilitate a comprehensive evaluation and allow for a concise comparison of the effectiveness of our models.

[0149] The dataset selected for the invention uses the High Resolution Cloud Dataset (HRC_WHU). This dataset is well-known in the field of cloud detection for its diversity and high resolution. The dataset is sourced from Google Earth and includes a wide range of landscapes, such as water bodies, vegetation areas, urban environments, snow / ice areas, and barren areas. The high resolution and diverse scene categories enable a comprehensive evaluation of the adaptability and performance of the algorithm in different environmental contexts. Each image in the dataset is accompanied by a ground truth (GT) cloud mask carefully annotated by remote sensing experts from Wuhan University. These expert annotations provide a high-quality, reliable comparison standard to ensure that the evaluation of cloud detection algorithms is accurate and credible. Importantly, the public availability of the HRC_WHU dataset promotes its widespread adoption and benchmarking in the research community.

[0150] Table 1 provides a detailed overview of the HRC_WHU dataset, which includes 150 high-resolution remote sensing images, each of which includes three channels (RGB) and is presented in TIFF format. This format was chosen at the initial collection stage to maintain data quality. The original resolution of these images is 1280×720 pixels, and the average file size is about 2.55MB. The dataset ensures a balanced distribution of effective training and testing, with each class being equally represented by 30 images. In terms of dataset partitioning, we follow the default configuration, allocating 120 images for training and 30 for testing.

[0151] Table 1

[0152]

[0153] To evaluate the performance of our model on the HRC_WHU dataset, we used five standard metrics commonly used in the field of image segmentation: mean absolute error (MAE), mean intersection over union (mIoU), pixel accuracy (PA), Dice coefficient, and F1 score. Each metric provides a unique perspective on accuracy and effectiveness, making the comparison with other models more balanced. MAE quantifies the average prediction error, mIoU evaluates the accuracy of the overlap between predicted and actual segments, and PA measures the proportion of correctly classified pixels. In addition, the Dice coefficient and F1 score evaluate the precision and recall of the model, respectively. Together, these metrics facilitate a comprehensive evaluation and allow for a concise comparison of the effectiveness of our models.

[0154] To further verify the effectiveness of the present invention, a comparative analysis is conducted with five other widely used models in the field. The neural network architectures evaluated in our experiments include Res2Net, ResNet, ResNeXt, Pyramid Visual Transformer (PVT), DeepLabV3, and our proposed model, hereinafter referred to as "our model". Our model shows superior performance on multiple evaluation metrics. Specifically, it achieves the lowest mean absolute error (MAE) of 0.156, the highest mean intersection over union (mIoU) of 0.625, the highest pixel accuracy (PA) of 0.847, the largest Dice coefficient of 1.262, and the highest F1 score of 0.722. In contrast, the DeepLabV3 architecture records the second best results in terms of PA and Dice coefficient, but the F1 score is lower than our model. The PVT architecture shows significant competitiveness with an mIoU of 0.621 and an F1 score of 0.758; however, it performs worse than our model in terms of PA. The Res2Net and ResNet architectures produced comparable results, with slight differences in MAE and mIoU, while underperforming in the remaining metrics. In contrast, although the ResNeXt model achieved the second-lowest MAE of 0.255, it did not outperform the other models in any other metric.

[0155] Figure 2 This shows that our invented method performs better than other comparison methods in cloud detection tasks, can better capture cloud details and boundaries, and its overall performance is closest to the ground truth. Other methods, such as Res2Net, ResNet, and DeepLab, can segment the main cloud area, but they are insufficient in detail processing and edge preservation, especially in the detection of complex cloud layers and small cloud clusters.

[0156] Table 2 shows that the proposed model performs best in all indicators, especially in mIoU and PA indicators, indicating that it can capture more cloud details in cloud detection tasks with small errors and high accuracy. Compared with other models, the performance of the PVT model is also relatively good, but it is still slightly inferior to the proposed model in detail capture and cloud boundary processing.

[0157] Table 2 Quantitative comparison of experimental results

[0158]

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A high-resolution remote sensing image cloud detection method based on a transformer, characterized in that: The steps include: S1: construct an image cloud detection model, which includes a pyramid visual transformer PVT, a channel attention module CAB, a graph fusion module and a decoder module; Pyramid Visual Transformer PVT is used as the backbone network architecture. The image X is input into PVT, and PVT divides the image X into a multi-scale feature map FM. i ; S2: Using channel attention module CAB for FM i Adjust FD t =CAB(FM i ); In the first iteration, the initial fusion feature map FD is obtained 0 =CAB(FM1); S3: For the current number of cycles t, the current feature map FM i And the feature map FD output after the previous iteration t-1 Splice to get FM i ′; S4: The output of S3 is further processed by the channel attention module Then generate The features are input into the convolution module to obtain FD t , after each round, output the current feature FD t Continue to update the prediction as input for the next round of loop; S5: The final feature map FD is passed through the decoder module T Decoded into the final cloud detection result P i , then calculate the corresponding loss in They represent the weighted intersection loss of the i-th prediction result and the weighted binary cross entropy loss of the i-th prediction result respectively; S6: The final four-round prediction results are processed by the graph fusion processing module to generate the final cloud detection result P that integrates all the prediction information. final , then calculate the total loss in in, and Represent the fusion results P final The weighted intersection loss and weighted binary cross entropy loss of Calculate the gradient, and then use the AdamW optimizer to update the parameters of the image cloud detection model according to the gradient. When the number of loops reaches the total number of loops T, the loop ends and the optimal image cloud detection model is obtained. S7: For a new image, input it into the optimal image cloud detection model and output the detection result.

2. A method for detecting cloud in remote sensing images based on a high resolution converter as claimed in claim 1, characterized in that: In S1, PVT is used as a backbone network architecture for feature extraction, and each stage generates feature maps of different scales by extracting multi-scale feature maps from the input image X; each stage includes a core structure, and the core structure includes overlapping patch embedding and a Transformer encoder; Each stage of PVT calculates feature maps of different scales through the formula, the formula is as follows: FM i =PVT i (X),i=1,2,3,4.

3. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: In S2 and S4, the specific process of the channel attention module processing features is as follows: First, for FM i First use global average pooling to generate channel importance scores: g i =GAP(F) Where F represents FM in S2 i or FM in S4 i ′; Secondly, through a fully connected layer, FM i g i Converted to the channel weight W for each channel i : W i =σ(W i ′·g i ) Where σ represents the activation function Sigmoid; Finally, use W i Weight each channel of F to get 4. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: In S3, the goal of fusion is to combine the feature graph of the current cycle with the result of the previous cycle so that the model can be gradually optimized and the prediction can be refined. The specific formula is: FM i ′=cat(FM i ,FD t-1 ) Where cat(·) represents the concatenation operation in the channel dimension, t∈[1,T], where T represents the total number of loops, which is a preset hyperparameter.

5. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: The S4 convolution module operation is defined as: Where Conv(·) represents a standard convolution operation.

6. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: The workflow of the S5 decoder module is as follows: First, an upsampling operation is performed: the feature map is gradually upsampled to the same resolution as the input image; FM up =Upsample(FD T ), Where Upsample represents the upsampling operation; Secondly, through convolution decoding, the multi-channel feature map is decoded into a binary cloud detection map P: P=σ(Conv(FM up )); Where σ represents the activation function Sigmoid.

7. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: In S5, The calculation is as follows: Where T j and P j denote the true label and predicted value of pixel j, respectively, and w j represents the weight of pixel j; The calculation is as follows:

8. The high-resolution remote sensing image cloud detection method based on a transformer as claimed in claim 1, characterized in that: The workflow of the S6 graph fusion processing module is as follows: 1) Flatten the prediction output results: In the S5 stage, four different prediction results P1, P2, P3, and P4 are generated. Each result P i , i = 1, 2, 3, 4 is a two-dimensional feature map, which represents the predicted values ​​of different pixels. The prediction results of these different stages are flattened into nodes. For each pixel, the predicted values ​​of the four stages are combined to form a feature vector. Assuming there are N pixels in total, the feature vector of each pixel is expressed as: h n =[P1(n),P2(n),P3(n),P4(n)] T ,n=1,2,…N Where P i (n) represents the prediction result of pixel n in the i-th stage; Finally, we get a node feature matrix H with a shape of N×4, which means that H has N nodes and each node has 4 features: H=[h1,h2,…,h N ] T where h N It represents the predicted feature vector of the Nth pixel in multiple rounds of refinement iterations. 2) Calculation of adjacency matrix: Construct adjacency matrix A to represent the similarity between nodes; Among them, the adjacency matrix A is an N×N matrix, in which the elements A ij Represents the similarity between node i and node j; 3) Graph convolution operation: The node features are updated and fused through the graph convolution network GCN. The formula of graph convolution is as follows: H′=σ(D -1 (AHW) Among them, H′ is the updated node feature matrix with a shape of N×d, d represents the feature dimension after convolution, A represents the adjacency matrix, D is the degree matrix representing the degree of each node, which is used to normalize the adjacency matrix, W is the trainable weight matrix used to adjust the node features, and σ is the ReLU activation function; 4) Multi-layer graph convolution: Assume that after L layers of graph convolution operations, the final node feature matrix is: H L =σ(D -1 AH L- 1W L-1 ) Among them, H L-1 , W L-1 They represent the node feature matrix and weight matrix of the graph convolutional network at layer L-1 respectively; 5) Final fusion prediction results: After completing the graph convolution operation, the node feature matrix H L The prediction result needs to be restored to a two-dimensional form; each node feature corresponds to the final prediction result of a pixel point; the final output is: P final =reshape(H L ) where reshape is the flattened node feature matrix H L Restore the shape to the original image.

Images