Remote sensing image cloud detection method and system based on convolution and transformer cooperation

Abstract

This invention discloses a remote sensing image cloud detection method and system based on the collaborative cooperation of convolution and Transformer, belonging to the field of computer vision. The method includes the following steps: extracting low-level features of the preprocessed image through a Conv2D-BN-ReLU layer; and extracting dense pixel features F from the preprocessed image by sequentially passing the MS-FPM module and MRFM module through a CNN stream. c ; to make dense pixel features F c Input to the Transformer stream, and adjust the dense pixel features F through the IIM module. c The channel dimension will be adjusted to determine the dense pixel features F. c It is fused with the underlying features, and then dense pixel features F are obtained through multiple Swing Transformer modules. t First, fuse dense pixel features F i and dense pixel features F t Then, multiple local features are sequentially fused, and upsampling is performed after each fusion process. Finally, the semantic segmentation result is obtained by combining the results with supervisory information. This invention can solve the problem that existing technologies are limited to capturing only local or global information.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and involves technologies such as image processing and deep learning. Specifically, it relates to a remote sensing image cloud detection method and system based on the collaborative cooperation of convolution and Transformer. Background Technology

[0002] In recent years, the field of remote sensing imagery has developed rapidly, playing a vital role in military, agriculture and forestry, hydrology, and environmental protection. Two-thirds of the Earth's surface is covered by clouds, and clouds, as a crucial component of meteorology, have a key impact on predicting severe weather by observing their changes. However, the morphology and thickness of clouds are highly variable, making it difficult for satellite sensors to directly acquire accurate surface information under cloudy conditions. Therefore, accurate cloud detection algorithms are of great significance to research in meteorology, environmental science, and other related disciplines.

[0003] Traditional cloud detection methods typically rely on thresholds to detect both clouds and non-clouds. While simple and effective, using spectral and wavelength differences between clouds and other objects, threshold selection heavily depends on prior knowledge. Although this improves accuracy to some extent, it often results in false negatives and missed detections. In complex situations with various interference factors, it becomes difficult to distinguish clouds from other objects. In recent years, deep convolutional neural networks have made significant breakthroughs in computer vision, demonstrating powerful feature representation capabilities and achieving better performance and robustness. They can extract detailed information such as texture and edges from lower layers, as well as complex and abstract semantic information from deeper layers. They have become a major research method in semantic segmentation of remote sensing images. For example, Boundary Nets, utilizing scalable and differentiable boundary networks, effectively address the scale insensitivity and boundary ambiguity issues in cloud detection. LWCDnet, a lightweight encoder-decoder cloud detection network, simultaneously extracts spatial and contextual information in the encoder and backbone, improving cloud detection performance.

[0004] Previous semantic segmentation models mostly used convolutional networks to extract features, which prevented the models from capturing global features and affected cloud detection performance. To improve the aforementioned problems, models need to be able to extract not only local features but also efficiently obtain global features. Recently, Transformer architecture models have achieved great success in the field of natural language processing. Researchers have found that extending Transformer to computer vision tasks can achieve good results. However, directly applying these Transformer methods to cloud detection tasks does not yield very good segmentation performance. This is because clouds have diverse forms and high-resolution multispectral remote sensing images contain more channels, making accurate segmentation of cloud regions more difficult. Experiments have shown that the performance of these methods based on single theoretical models is far from satisfactory in achieving cloud detection results. Summary of the Invention

[0005] This invention provides a remote sensing image cloud detection method and system based on the collaborative cooperation of convolution and Transformer, in order to solve the problem that existing technologies are limited to capturing local or global information only.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A cloud detection method for remote sensing images based on the collaborative operation of convolution and Transformer includes the following steps:

[0008] Input the preprocessed image into the CNN stream and the Transformer stream;

[0009] In both the CNN stream and the Transformer stream, the low-level features of the preprocessed image are extracted through Conv2D-BN-ReLU layers, and then the low-level features in the CNN stream are obtained by max pooling to obtain the first local features.

[0010] Then, in the CNN stream, the preprocessed image is extracted sequentially through Stage 2, Stage 3, Stage 4, Stage 5, and MRFM modules to obtain dense pixel features F. c Stage 2 generates a second local feature, Stage 3 generates a third local feature, Stage 4 generates a fourth local feature, and Stage 5 outputs the feature map to the MRFM module. Stage 2, Stage 3, Stage 4, and Stage 5 each include an MS-FPM module.

[0011] The dense pixel feature F c Input is fed into a Transformer stream to construct an IIM module, and the dense pixel features F are adjusted through the IIM module.c The channel dimension is then determined, and the adjusted dense pixel features F are calculated. c The fused features obtained by fusing with the underlying features provide prior information for the Transformer stream;

[0012] In the Transformer stream, the obtained fused features are processed through the first, second, third, and fourth Swing Transformer modules to obtain dense pixel features F. t ;

[0013] A skip connection mechanism is used to first fuse dense pixel features F c and dense pixel features F t Then, the fourth local feature, the third local feature, the second local feature and the first local feature are fused sequentially, and upsampling is performed after each of the above fusion processes;

[0014] For each fusion process, supervision information is acquired and backpropagation is performed after overall supervised training. Finally, after upsampling the first local features, the semantic segmentation result is obtained by combining the corresponding supervision information.

[0015] In some embodiments, the preprocessed image includes remote sensing images and labeled images, and the preprocessed image is obtained through the following steps:

[0016] The remote sensing image and the label image are cropped, and all pixels in the label image except for cloud pixels are set to 0 to obtain a preprocessed image.

[0017] In some implementations, the MS-FPM module includes three convolutional kernel branches of different sizes, and the MS-FPM module obtains features corresponding to different receptive fields through the three convolutional kernels of different sizes.

[0018] In some implementations, the MRFM module includes five branches. Each branch first extracts detailed features and reduces channel dimension through a 1×1 Conv2D-BN-ReLU layer. Then, four of the branches use convolutional kernels of different sizes to obtain features corresponding to different receptive fields. Finally, the last branch performs image pooling to obtain channel features.

[0019] The MRFM module concatenates the features from different receptive fields and channel features to obtain multi-scale features. These multi-scale features are then passed through a 1×1 Conv2D-BN-ReLU layer to reduce the channel dimension, and finally added to the feature map output from Stage 5 to obtain the dense pixel feature F. c .

[0020] In some implementations, the dense pixel feature F c The channel dimension is adjusted to be equal to the channel dimension of the underlying feature.

[0021] In some implementations, the adjustment of the dense pixel features F via the IIM module c The channel dimension is determined by the following steps:

[0022] Dense pixel features F c After being processed through multiple 3×3 Conv2D-BN-ReLU layers, the data is upsampled and then added to and fused with the underlying features.

[0023] In some implementations, the following steps are also included:

[0024] Supervision information is obtained for each fusion process, and overall supervised training is performed based on the supervision information.

[0025] In some implementations, the overall loss in the overall supervised training is obtained by the following formula:

[0026]

[0027] Where y represents the truth value, f i d λ represents the output of each fusion process. i L represents the hyperparameter representing the weights between control losses. CTFuse L represents the total loss. BCE L represents the loss calculated using binary cross-entropy. IoU This represents the loss calculated using the crossover and union ratio.

[0028] A system for cloud detection in remote sensing images based on the collaborative operation of convolution and Transformer, comprising the following components:

[0029] The image acquisition module is used to input preprocessed images into the CNN stream and the Transformer stream;

[0030] The CNN processing module is used to extract the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, then obtain the first local features through max pooling, and then sequentially extract the dense pixel features F from the preprocessed image through Stage 2, Stage 3, Stage 4, Stage 5 and MRFM modules. cStage 2 generates a second local feature, Stage 3 generates a third local feature, Stage 4 generates a fourth local feature, and Stage 5 outputs the feature map to the MRFM module. Stage 2, Stage 3, Stage 4, and Stage 5 each include an MS-FPM module.

[0031] The Transformer processing module first extracts the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, and then extracts the dense pixel features F. c Input is fed into a Transformer stream to construct an IIM module, and the dense pixel features F are adjusted through the IIM module. c The channel dimension is then determined, and the adjusted dense pixel features F are calculated. c The fused features are added to and fused with the underlying features to provide prior information for the Transformer stream; finally, the fused features are passed sequentially through the first SwinTransformer module, the second SwinTransformer module, the third SwinTransformer module, and the fourth SwinTransformer module to obtain the dense pixel features F. t ;

[0032] The skip connection module first fuses dense pixel features F c and dense pixel features F t Then, the fourth local feature, the third local feature, the second local feature and the first local feature are fused sequentially, and upsampling is performed after each of the above fusion processes;

[0033] The supervised training module is used to acquire supervision information for each fusion process, perform overall supervised training and backpropagation, and finally, after upsampling the first local features, combine the corresponding supervision information to obtain the semantic segmentation result.

[0034] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention provides a cloud detection method for remote sensing images based on the collaborative cooperation of convolution and Transformer. It employs the Swin-Transformer method to extract global representation features. The Swin-Transformer model adopts a hierarchical design, and its representation is computed using shift windows. The shift windows compute self-attention through non-overlapping local windows and allow cross-window connections, thereby improving efficiency. Considering that remote sensing images include thick and thin clouds containing different background information, some previous networks simply extracted features. In this invention, the encoder network combines convolutional and Transformer frameworks to effectively capture local and global information, thus further enhancing the model's representational capabilities. Furthermore, a multi-scale feature pyramid module (MS-FPM) and a multi-receptive field module (MRFM) are proposed to construct the convolutional stream, which can effectively capture spatial context information, extract more accurate features, and generate high-level semantic representations. To further improve the model's feature representation capabilities, this invention proposes an Information Injection Module (IIM) to effectively combine the Transformer and convolutional streams. This feature extraction method effectively inherits the advantages of CNN and Transformer. Furthermore, this invention introduces a skip connection mechanism in the decoder to restore the image size through upsampling. Since convolutional networks contain more local information, they are fused with features from the encoder's convolutional branches during image size restoration, embedding low-level details into high-level semantic information. This compensates for lost low-level features and improves cloud detection accuracy. At the decoding end, feature maps from different levels are output to calculate auxiliary loss. Before calculating the loss, the output feature maps of each stage are upsampled to the input image size. By supervising the output feature maps at different levels, network convergence is accelerated, and the network's prediction accuracy is improved. In addition, this invention introduces overall joint supervised training during the detection process. Supervision information is acquired in a distributed manner at multiple stages of the decoder, and the losses are aggregated for overall supervised training. Attached Figure Description

[0037] Figure 1 A general block diagram of a remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer provided in an embodiment of the present invention;

[0038] Figure 2 (a) is a detailed flowchart of the MS-FPM module provided in an embodiment of the present invention;

[0039] Figure 2 (b) A detailed flowchart of the MRFM module provided in an embodiment of the present invention;

[0040] Figure 2(c) A detailed flowchart of the IIM module provided in an embodiment of the present invention;

[0041] Figure 3 The comparison of false detections for different models on the SPARCS dataset is shown, where (a-1) is the original image, (b-1) is the Ground truth model result, (c-1) is the U-Net model result, (d-1) is the PSPNet model result, (e-1) is the DeepLabv3+ model result, (f-1) is the CloudUNet model result, (g-1) is the CDnetv2 model result, (h-1) is the MFNet model result, (i-1) is the CANet model result, (j-1) is the BoundaryNets model result, and (k-1) is the Our method model result.

[0042] (a-2) is the original image 2, (b-2) is the Ground truth model result 2, (c-2) is the U-Net model result 2, (d-2) is the PSPNet model result 2, (e-2) is the DeepLabv3+ model result 2, (f-2) is the CloudUNet model result 2, (g-2) is the CDnetv2 model result 2, (h-2) is the MFNet model result 2, (i-2) is the CANet model result 2, (j-2) is the BoundaryNets model result 2, and (k-2) is the Our method model result 2.

[0043] Figure 3 In the image, missed pixels and falsely detected pixels are marked in blue and red, respectively.

[0044] Figure 4 This is a schematic diagram of the structure of the remote sensing image cloud detection system based on the collaborative cooperation of convolution and Transformer provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the electronic device structure used in this invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. The content described herein is for explanation rather than limitation of the present invention.

[0047] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification and claims of this invention are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, systems, products, or devices.

[0048] Convolutional Neural Networks (CNNs) have achieved significant breakthroughs and substantial progress in cloud detection. However, CNN-based feature extraction methods are limited to extracting local details, and their limited receptive field results in poor ability to extract global information. Therefore, it is highly significant to study the effectiveness of fusion methods that combine global and local features. This approach, in remote sensing image cloud detection tasks, not only efficiently extracts local features but also considers long-range interdependencies in the image, capturing features with global information. This effectively improves the network's feature extraction capabilities and enhances cloud detection performance.

[0049] Extending Transformers to computer vision tasks makes it easy to capture global information. In this work, the Swin-Transformer method is mainly used to extract global representation features. The Swin-Transformer model adopts a hierarchical design, and its representation is computed using shift windows. The shift windows compute self-attention through non-overlapping local windows and allow cross-window connections to improve efficiency. Furthermore, considering that remote sensing images include thick and thin clouds containing different background information is very challenging, and the model's feature extraction capability has a significant impact on the final segmentation result. Some previous networks simply extract features.

[0050] Therefore, in order to effectively utilize the local details and global semantic features of clouds in remote sensing images, this invention proposes CTFuse (CNN and Transformer fusion). Given an input image I∈R h×w×c Where h, w, and c represent the height, width, and number of channels of the input image, the input image is first processed through a 3×3 Conv2D-BN-ReLU layer to extract low-level features. Then, a convolutional network with MS-FPM and MRFM modules is used to extract deep features. High-level semantic representations are generated through convolutional networks. Following this, the IIM module will... c The fusion features and features obtained through a 3×3 Conv2D-BN-ReLU layer The features are fed into the Transformer stream to obtain features with global and local information. Finally, the output features of the Transformer stream and the convolutional stream are input into the decoder to obtain the segmentation result.

[0051] To achieve cloud detection in remote sensing images under complex scenarios, this invention adopts the following technical solution:

[0052] A cloud detection method for remote sensing images based on the collaborative operation of convolution and Transformer (overall structure diagram as shown in the figure) Figure 1 As shown in the diagram, this method uses a combination of Transformer and convolutional networks as the backbone to extract features at different levels. These two parts cooperate during feature extraction, enhancing the model's feature extraction capabilities and enabling it to more effectively capture high-level semantic features and low-level detailed information. The above method includes the following steps:

[0053] Step one involves using image preprocessing methods on a remote sensing image cloud detection dataset to obtain the input image for the model network. For a given dataset, it can be observed that the resolution of the original remote sensing images is usually very high. Due to GPU memory limitations, training with images of the original size is difficult. Therefore, each remote sensing image and label image is cropped to 384×384 pixels. Since this invention primarily focuses on cloud pixel detection, other categories in the dataset's labels are set as background pixels, i.e., pixels of other categories are set to 0, thus obtaining the network's input image and label image.

[0054] Step two: Input the image obtained after preprocessing in step one into the network. The encoder of this network contains two branch streams.

[0055] 2-1: CNN stream, the CNN stream consists of five stages, each stage is composed of the Multi-Scale Feature Pyramid Module (MS-FPM) (e.g.) Figure 2 As shown in the figure, the MS-FPM module aims to capture contextual information to enhance feature representation. It receives information from different receptive fields through three branches of convolutional kernels of different sizes, enabling the CNN stream to effectively capture local details of cloud targets in remotely sensed images. Specifically, it is shown in the following equation:

[0056] M = conv 5×5 (conv 3×3 (H))+conv 7×7 (conv 3×3 (H))+conv 1×1 (H)

[0057] Here, H represents the input feature map, and + indicates an element-wise addition operation. For example... Figure 1As shown, the CNN stream takes image I as input and generates dense pixel features. The dense pixel features of the s-th stage are represented as: (C S H S W S (These represent the number of channels, height, and width at stage s, respectively). To improve the segmentation capability and robustness for images at different scales, a multi-receptive field module (MRFM) is constructed after stage 5 of the CNN stream, such as... Figure 2 As shown, this module is inspired by ASPP (Atrous Spatial Pyramid Pooling). MRFM extracts and fuses features from different receptive fields from local to global perspectives, facilitating not only long-distance relationship modeling but also capturing local detail information. Specifically, MRFM comprises five branches. Each branch first uses a 1×1 Conv2D-BN-ReLU layer to extract detailed features from the input image and reduces dimensionality by decreasing the number of channels. The first four branches use larger convolutional kernels to effectively expand the receptive field, obtaining features with different receptive fields. The last branch is an image pooling operation that generates channel features. The features from the five branches are then concatenated to generate multi-scale features, which are further reduced using a 1×1 Conv2D-BN-ReLU layer to decrease the number of channels and reduce dimensionality. Finally, these features are added to the input to obtain the final output dense pixel feature F. c .

[0058] 2-2: Transformer stream, which takes image I and the output F of the last convolutional layer of the CNN stream as input. c As input, dense pixel features F are generated. t The Swin Transformer demonstrates significant advantages in image classification and dense prediction tasks, making it the choice for the Transformer Streaming baseline network in this study. Furthermore, this invention constructs an IIM module, which gradually reduces the number of channels and performs upsampling operations through a series of 3×3 Conv2D-BN-ReLU layers to improve F... c Adjust to low-level features Using the same channel dimension as input to the Transformer stream allows the Transformer stream to have prior information during feature extraction.

[0059] Step 3: Using the features with global and local information obtained in Step 2, a skip connection mechanism is introduced in the decoding stage to embed low-level detail information into high-level semantic information and gradually restore the cloud image size to obtain the final features used for cloud detection tasks.

[0060] CTFuse's decoder employs a design similar to U-Net, consisting of B Conv2D-BN-ReLU blocks connected by upsampling layers with a scaling factor of 2. Each block contains three Conv2D-BN-ReLU layers. Skip connections are used to fuse multi-scale features from the encoder and upsampled features. This invention connects shallow features with high-level features to reduce spatial information loss caused by downsampling.

[0061] Step four: Overall joint supervision and training.

[0062] Step 4-1: After step three, the final image used to calculate the loss and pixel classification is obtained. However, experiments have shown that a single loss function is insufficient to constrain the model. Therefore, supervision information is acquired in a distributed manner at multiple stages of the decoder, and the losses are aggregated for overall supervised training. To calculate these losses, using binary cross-entropy (BCE) and intersection-over-union ratio (IoU), the CTFuse loss can be expressed as:

[0063]

[0064] y represents the truth value, f i d λ represents the output of each stage of the decoder. i The hyperparameter represents the weights among the control losses.

[0065] Based on the above steps, this embodiment provides a system for remote sensing image cloud detection based on the collaborative cooperation of convolution and Transformer, including: an image acquisition module, a CNN processing module, a Transformer processing module, a skip connection module, and a supervised training module;

[0066] The image acquisition module inputs the preprocessed image to the CNN stream and the Transformer stream. The CNN processing module extracts the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, then obtains the first local features through max pooling, and then sequentially extracts the dense pixel features F from the preprocessed image through Stage 2, Stage 3, Stage 4, Stage 5 and MRFM modules. c ;

[0067] Specifically, Stage 2 generates a second local feature, Stage 3 generates a third local feature, Stage 4 generates a fourth local feature, and Stage 5 outputs the feature map to the MRFM module. Stage 2, Stage 3, Stage 4 and Stage 5 each include an MS-FPM module.

[0068] The Transformer processing module extracts the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, and then processes the dense pixel features F... c Input is fed into a Transformer stream to construct an IIM module, and the dense pixel features F are adjusted through the IIM module. c The channel dimension is then determined, and the adjusted dense pixel features F are calculated. c The features are added and fused with the underlying features to provide prior information for the Transformer stream. Finally, the fused features are sequentially passed through the first, second, third, and fourth SwinTransformer modules, and combined with the prior information to extract the preprocessed image, resulting in dense pixel features F. t ;

[0069] Among them, the skip connection module first fuses dense pixel features F c and dense pixel features F t Then, the fourth local feature, the third local feature, the second local feature, and the first local feature are fused sequentially, and upsampling is performed after each of the above fusion processes; the supervised training module acquires supervision information for each fusion process and performs overall supervised training and backpropagation. Finally, after upsampling after fusing the first local feature, the semantic segmentation result is obtained by combining the corresponding supervision information.

[0070] The dataset used in the experimental data of this invention is the SPARCS dataset.

[0071] SPARCS: The SPARCS dataset contains 80 Landsat 8 images, each with 10 channels and a resolution of 1000×1000. This invention cuts each image into 384×384 pixels with 50% overlap, resulting in 2000 images. These images are then divided into training and test sets at an 8:2 ratio, and a random horizontal flipping strategy is used during training to expand the training data.

[0072] 1. Experimental Details

[0073] The model of this invention was implemented using the PyTorch framework, and all models were trained on a single RTX A4000 NVIDIA GPU card. During network training, this invention used β1 = 0.9, β2 = 0.999, and ε = 10. -8 The Adam optimizer. Furthermore, this invention sets the batch size to 6 and the initial learning rate to 10. -5 For each experiment, the model was trained for a total of 200 epochs.

[0074] Throughout the experiment, precision (P), recall (R), overall accuracy (OA), and F1 score were used as the main evaluation metrics to assess the performance of the proposed method in cloud segmentation tasks. P represents the proportion of correctly predicted positive samples out of the actual predicted positive samples; R represents the proportion of correctly predicted positive samples out of the total positive samples; F1 score is the harmonic mean of precision and recall; and OA represents the accuracy of predicting the class for all pixels across all segmentation results. The formulas for each evaluation metric are as follows:

[0075]

[0076]

[0077]

[0078]

[0079] TP represents the number of correctly classified cloud pixels, TN represents the number of correctly classified non-cloud pixels, FP represents the number of pixels incorrectly classified as cloud pixels, and FN represents the number of pixels incorrectly classified as non-cloud pixels.

[0080] 2. Experimental results on the SPARCS dataset

[0081] Table 1: Experimental results on the SPARCS dataset

[0082]

[0083]

[0084] To verify the effectiveness of the method described in this invention, it was compared with state-of-the-art methods on the SPARCS dataset. Table 1 shows that the method of this invention achieves the best results in terms of accuracy (F), optima (OA), and retrieval rate (R), reaching 95.8%, 98.3%, and 96.1% respectively. Although its accuracy in SPARCS is not the highest, it is only 0.1% lower than the best-performing method, BoundaryNets. However, it lags significantly behind the method of this invention in other metrics. This is mainly because BoundaryNets focuses on mining discriminative information in cloud regions, leading to the loss of some cloud regions. The latest work, CANet, clearly lags far behind the method of this invention in all four metrics; therefore, the CTFuse proposed in this invention is the best overall method.

[0085] To demonstrate the advantages of the method proposed in this invention, a visual comparison was performed on the SPARCS test set, and the qualitative segmentation results are as follows: Figure 3As shown, the selected visualization images include both thick and thin clouds with varying background information, posing a significant challenge. It is evident that the method of this invention performs better in cloud detection in remote sensing images. In the thin cloud region of the first row, the network detected by this invention exhibits clear boundaries and excellent cloud detection performance. Other networks show varying degrees of missed detections and false detections, especially smaller networks such as U-Net and PSPNet, which detect more regions. In images with complex backgrounds, such as the second row, it is clear that the model of this invention has fine segmentation edges and good segmentation results. Even in scenes with many thick clouds, it demonstrates excellent segmentation performance. In summary, the model of this invention can accurately identify the location of clouds, precisely detect small and thin clouds, and effectively reduce background interference. Overall, the model of this invention outperforms other models.

[0086] As is known from common technical knowledge, the present invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones; all modifications within the scope of the present invention or equivalent to the scope of the present invention are included in the present invention.

Claims

1. A cloud detection method for remote sensing images based on the collaborative operation of convolution and Transformer, characterized in that, Includes the following steps: Input the preprocessed image into the CNN stream and the Transformer stream; In both the CNN stream and the Transformer stream, the low-level features of the preprocessed image are extracted through Conv2D-BN-ReLU layers, and then the low-level features in the CNN stream are obtained by max pooling to obtain the first local features. Then, in the CNN stream, the preprocessed image is sequentially passed through Stage 2, Stage 3, Stage 4, Stage 5, and MRFM modules to extract dense pixel features. Stage 2 generates a second local feature, Stage 3 generates a third local feature, Stage 4 generates a fourth local feature, and Stage 5 outputs the feature map to the MRFM module. Stage 2, Stage 3, Stage 4, and Stage 5 each include an MS-FPM module. The dense pixel features Input is fed into a Transformer stream to construct an IIM module, which is then used to adjust the dense pixel features. The channel dimension is then determined, and the adjusted dense pixel features are... The fused features obtained by fusing with the underlying features provide prior information for the Transformer stream; In the Transformer stream, the fused features are sequentially passed through the first Swing Transformer module, the second Swing Transformer module, the third Swing Transformer module, and the fourth Swing Transformer module to obtain dense pixel features. ; A skip connection mechanism is used to first fuse dense pixel features. and dense pixel features Then, the fourth local feature, the third local feature, the second local feature and the first local feature are fused sequentially, and upsampling is performed after each of the above fusion processes; For each fusion process, supervision information is acquired and overall supervised training is performed before backpropagation. Finally, after fusing the first local features and performing upsampling, the semantic segmentation result is obtained by combining the corresponding supervision information. The MS-FPM module includes three convolutional kernel branches of different sizes. The MS-FPM module obtains features corresponding to different receptive fields through the three convolutional kernels of different sizes. The MRFM module includes five branches. Each branch first extracts detailed features and reduces channel dimension through a 1×1 Conv2D-BN-ReLU layer. Then, four of the branches use convolution kernels of different sizes to obtain features corresponding to different receptive fields. The last branch performs image pooling to obtain channel features. The MRFM module concatenates the features from different receptive fields and channel features to obtain multi-scale features. These multi-scale features are then passed through a 1×1 Conv2D-BN-ReLU layer to reduce the channel dimension, and finally added to the feature map output from Stage 5 to obtain the dense pixel features. ; The dense pixel features are adjusted via the IIM module. The channel dimension is determined by the following steps: dense pixel features After being processed through multiple 3×3 Conv2D-BN-ReLU layers, the data is upsampled and then added to and fused with the underlying features as the input to the Transformer stream.

2. The remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer as described in claim 1, characterized in that, The preprocessed image includes remote sensing images and labeled images, and the preprocessed image is obtained through the following steps: The remote sensing image and the label image are cropped, and all pixels in the label image except for cloud pixels are set to 0 to obtain a preprocessed image.

3. The remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer as described in claim 1, characterized in that, The dense pixel features The channel dimension is adjusted to be equal to the channel dimension of the underlying feature.

4. The remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer as described in claim 1, characterized in that, It also includes the following steps: Supervision information is obtained for each fusion process, and overall supervised training is performed based on the supervision information.

5. The remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer as described in claim 1, characterized in that, In the overall supervised training, the overall loss is obtained by the following formula: Where y represents the truth value. This represents the output of each fusion process. Hyperparameters representing the weights between control losses, Indicates the total loss. This represents the loss calculated using binary cross-entropy. This represents the loss calculated using the crossover and union ratio.

6. A system for cloud detection in remote sensing images based on the collaborative operation of convolution and Transformer, characterized in that, The remote sensing image cloud detection method according to any one of claims 1 to 5 includes: The image acquisition module is used to input preprocessed images into the CNN stream and the Transformer stream; The CNN processing module is used to extract the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, then obtain the first local features through max pooling, and then sequentially extract the dense pixel features of the preprocessed image through Stage 2, Stage 3, Stage 4, Stage 5 and MRFM modules. Stage 2 generates a second local feature, Stage 3 generates a third local feature, Stage 4 generates a fourth local feature, and Stage 5 outputs the feature map to the MRFM module. Stage 2, Stage 3, Stage 4, and Stage 5 each include an MS-FPM module. The Transformer processing module first extracts the low-level features of the preprocessed image through a Conv2D-BN-ReLU layer, and then extracts the dense pixel features. Input is fed into a Transformer stream to construct an IIM module, which is then used to adjust the dense pixel features. The channel dimension is then determined, and the adjusted dense pixel features are... The fused features are added to and fused with the underlying features to provide prior information for the Transformer stream; finally, the fused features are sequentially passed through the first SwinTransformer module, the second SwinTransformer module, the third SwinTransformer module, and the fourth SwinTransformer module to obtain dense pixel features. ; The skip connection module first fuses dense pixel features. and dense pixel features Then, the fourth local feature, the third local feature, the second local feature and the first local feature are fused sequentially, and upsampling is performed after each of the above fusion processes; The supervised training module is used to acquire supervision information for each fusion process, perform overall supervised training and backpropagation, and finally, after fusing the first local features and performing upsampling, combine the corresponding supervision information to obtain the semantic segmentation result.

7. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable in the processor. When the processor executes the computer program, it implements the steps of the remote sensing image cloud detection method based on the collaborative cooperation of convolution and Transformer as described in any one of claims 1 to 5.

Images