An optical image segmentation method based on multi-branch convolutional attention model

Abstract

The present invention provides an optical image segmentation method based on a multi-branch convolutional attention model, which includes: (1) defining and instantiating a data set; (2) visualizing input data; (3) defining and instantiating a multi-branch convolutional attention network model; and (4) training the multi-branch convolutional attention network model. The multi-branch convolutional attention model proposed in the present invention uses a parallel module of convolution and Transformer, adds a direct jump connection in each convolution module, and introduces a fusion module to replace the common splicing operation. The fusion module can effectively fuse the global information and local information extracted by the two branches, and filter them to retain only the meaningful parts, thereby improving the model efficiency. The multi-branch convolutional attention model can better retain high-level semantic information, while combining low-level detail information to obtain more accurate segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of optical image recognition technology, and in particular to an optical image segmentation method based on a multi-branch convolutional attention model. Background Art

[0002] Cloud and snow segmentation is crucial in remote sensing, particularly for tasks such as surface information extraction and climate monitoring. Clouds obscure surface features, affecting the accuracy of surface classification. The extent and distribution of snow cover, on the other hand, are crucial indicators for assessing water reserves and environmental change. Accurate cloud and snow segmentation not only improves the resolution of remote sensing imagery but also provides reliable data for global change research. Furthermore, the morphological and spectral characteristics of clouds and snow exhibit certain similarities, complicating the segmentation task. Therefore, research on cloud and snow segmentation plays a crucial role in enhancing the application value of remote sensing imagery and promoting Earth science research.

[0003] The accuracy of feature extraction directly affects the final segmentation performance, which is particularly important in the cloud-snow segmentation task. In cloud-snow segmentation, image data often contains complex textures, illumination variations, and thin cloud cover. Previous studies have shown that Convolutional Neural Networks (CNNs) perform well in extracting local features, especially for edge detection and texture segmentation. However, CNNs are insufficient in capturing global contextual information, especially when dealing with large-scale spectral variations and cross-regional similarities, which can lead to information loss or inaccuracies.

[0004] In recent years, the Transformer architecture, with its powerful global modeling capabilities, has emerged as a potential solution to this problem. Using a self-attention mechanism, the Transformer is able to capture long-range contextual dependencies in image segmentation tasks. In this way, the Transformer effectively addresses the shortcomings of convolutional networks in extracting global information. This capability is particularly useful in cloud and snow segmentation, where it can better distinguish spectrally similar but spatially distinct regions.

[0005] Common CNN models (such as UNet and UNet++) demonstrate high efficiency in cloud-snow segmentation tasks. Through multi-scale convolutions and skip connections, these models are able to extract rich local features and effectively fuse shallow and deep features. However, when faced with spectrally complex snow or thin cloud areas, these models may struggle to accurately distinguish boundaries or similar pixels.

[0006] Transformer-based architectures such as the Vision Transformer (ViT) and the Swin Transformer have achieved initial success in cloud and snow segmentation. These architectures capture a wide range of contextual information through a global attention mechanism, resulting in more accurate classification of thick and thin cloud areas. However, Transformers have high computational overhead and relatively weak generalization capabilities for small sample sizes.

[0007] In recent years, some studies have proposed hybrid architectures that combine convolution and Transformer to improve the performance of cloud and snow segmentation. For example:

[0008] ConvNext-Transformer model: Combining the CNN and Transformer parallel processing paths improves the ability to extract local and global features, but its complex design leads to a high number of model parameters, large demands on hardware resources, and significantly increased training costs. In addition, the complex network structure increases the difficulty of debugging and optimization, and is prone to training instability. In this model, the CNN and Transformer paths run in parallel, and there may be information redundancy or conflict in the fusion phase, resulting in reduced feature extraction and classification performance. When processing spectrally complex regions, the fusion mechanism may not be able to fully integrate multi-scale features, affecting the model's ability to capture details, especially at the cloud-snow boundary or in thin cloud areas. This model tends to fuse macroscopic global information and local features, but its ability to refine boundary areas is relatively insufficient, which can easily lead to blurred boundaries in the segmentation results, especially in complex scenes where clouds and snow meet.

[0009] Swin-Unet model: Although the hierarchical Transformer architecture used in this model has improved global modeling capabilities, its windowing operation leads to the truncation of contextual information. In cloud and snow segmentation, the modeling ability of long-distance dependencies across windows is weak, and important global information may be missed. It attempts to achieve a balance between local and global feature extraction, but due to over-reliance on the hierarchical mechanism, the contribution of shallow features in multi-scale contexts may be weakened. Especially when dealing with complex boundary areas, the model may not be able to fully capture low-level detail features, resulting in poor boundary refinement. At the same time, the model is highly dependent on high-quality training data. In the cloud and snow segmentation task, the cost of annotating data is high. When the amount of data is insufficient, the model is prone to overfitting, and the generalization performance is limited.

[0010] In summary, it is necessary to make further innovations to the existing technologies. Summary of the Invention

[0011] In response to the technical problems existing in the above-mentioned background technology, the present invention proposes an optical image segmentation method based on a multi-branch convolutional attention model. The method has a reasonable conception. By introducing parallel branches to design a multi-branch structure, local and global feature information are processed simultaneously, avoiding the limitation of a single convolutional network that is only good at extracting local features; by efficiently integrating the features of convolution and Transformer, redundancy and conflict between feature information are avoided, thereby improving the accuracy of feature extraction, which can solve the technical problem of the existing technology in the cloud and snow segmentation task that improper processing of complex boundaries and details will significantly reduce the accuracy of the model.

[0012] To solve the above technical problems, the present invention provides an optical image segmentation method based on a multi-branch convolutional attention model, which includes the following steps:

[0013] (1) Define and instantiate the dataset;

[0014] (2) Visualize input data;

[0015] (3) Define and instantiate a multi-branch convolutional attention network model;

[0016] (4) Train the multi-branch convolutional attention network model.

[0017] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of step (1) is:

[0018] (1.01) Loading image data

[0019] Open the image data in each CSWV dataset and extract data of different bands from the image data;

[0020] (1.02) Stacked Image Data

[0021] Stack data from different bands together to form image data from different channels;

[0022] (1.03) Image data conversion arrangement

[0023] Convert the arrangement of image data from height, width, channel to channel, height, width;

[0024] (1.04) Image data normalization

[0025] Normalize the pixel values ​​of the image data, that is, divide each pixel value by 255.0, thereby scaling the pixel value range from the original 0 to 255 to the interval of 0 to 1, and obtain the input image data X;

[0026] (1.05) Read tag data

[0027] Load the tif files of cloud labels and snow labels corresponding to each image data, indicating whether each pixel of the image data belongs to cloud or snow respectively;

[0028] (1.06) Processing tag data

[0029] The data of the read cloud label and snow label tif files are converted into binary mask matrices. First, the mask of the background area is generated by checking whether the cloud label and the snow label are both 0. The generated binary mask matrix marks the background area as 1 and the rest of the area as 0. Then, the mask of the cloud area is generated by checking whether the cloud label is 255, indicating that the cloud area is 1 and the other areas are 0. The mask of the snow area is generated by checking whether the snow label is 255, marking the snow area as 1 and the other areas as 0. The mask matrix generated by the background area, the mask matrix generated by the cloud area, and the mask matrix generated by the snow area represent the categories of background, cloud, and snow respectively. They are put into a three-channel array, and each channel represents one of the mask matrices, thereby generating label data.

[0030] (1.07)Tag data type conversion

[0031] Convert the label data from integer type to floating point type by calling the astype method of NumPy array;

[0032] (1.08) Merge Mask

[0033] Stack the binary mask matrices of background, cloud, and snow into a three-channel matrix;

[0034] (1.09) Convert to Tensor

[0035] Convert both image data and label data into PyTorch tensors to obtain input label data Y;

[0036] (1.10) Return data

[0037] The obtained input image data X and input label data Y are returned as multiple pairs of input and target for multi-branch convolutional attention model training or inference.

[0038] The optical image segmentation method based on the multi-branch convolutional attention model, wherein: before loading the image data in the step (1), the CSWV dataset is uniformly cropped into 512×512×3 image data.

[0039] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of the step (2) is: visualizing the input image data X and the corresponding label data Y, separating the red, green, and blue bands and label data from the Pytorch tensor, stacking the red, green, and blue bands to generate an RGB image, converting the one-hot encoding of the label data Y into a category index and applying a custom color mapping to obtain a label image, drawing the RGB image and the label image into a subgraph, intuitively displaying the input and target output of the multi-branch convolutional attention network model, helping to verify whether the process of the steps (1.01)-(1.09) is correct, and providing visualization support for the optimization of the multi-branch convolutional attention network model.

[0040] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of step (3) is:

[0041] (3.1) The multi-branch module adopts a branch parallel structure as the backbone network of the multi-branch convolutional attention network model. The branch contains convolution branches, that is, two layers of 3×3 convolution with a residual structure are introduced. After each layer of convolution, a batch normalization BN and ReLU activation function s are performed. Finally, after the residual structure, the feature map x of the convolution branch output is obtained. c :

[0042] C=σ(BN(Conv 3×3 (f i ));

[0043] f i+1 =σ(BN(Conv 3×3 (C));

[0044] x C =Concat(f i ,f i+1 );

[0045] The other branch, the Transformer branch, runs in parallel with the convolutional branch. The Transformer branch uses a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers. This is achieved by introducing an optimized structure based on a combination of self-attention and convolution, namely an efficient "local-global-local" information interaction bottleneck. Thus, the multi-branch module uses the convolutional branch to extract local features from the image data and the Transformer branch to grasp the global features in the image data.

[0046] (3.2) A fusion module of a multi-branch convolutional attention network model is used to replace the simple splicing of multiple feature channels in the channel dimension to integrate the different features of the convolution branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation;

[0047] The fusion module integrates the different features of the convolution branch and the Transformer branch. The specific process is as follows: first, the high-level features are upsampled to the same scale as the low-level features, and then two convolution kernels with sizes of 1×3 and 3×1 are used to filter and extract features from the two respectively; then, batch normalization and activation function GELU are applied for further processing; then, the information of the two branches is spliced, and the spliced ​​features are again filtered and filtered by two convolution kernels with sizes of 1×3 and 3×1 to establish a complementary relationship and obtain the extracted multi-level fusion feature X. F At the decoding end of the multi-branch convolutional attention network, the category information of high-level features can guide the classification of low-level features, and the position information of low-level features complements the spatial information of high-level features.

[0048] The calculation process of the fusion module is as follows:

[0049] X1=G(BN(DOConv 3×1 (DOConv 1×3 (X in1 ))));

[0050] X2=G(BN(DOConv 3×1 (DOConv 1×3 (Up(X in2 )))));

[0051] W = Concat(X1,X2);

[0052] Y out =G(BN(DOConv 3×1 (DOConv 1×3 (W))));

[0053] Among them, X in1 and X in2 Represent the two inputs of the fusion module, Y out Represents the output of the fusion module; DOConv 3×1 Represents DO-Conv with a convolution kernel size of n×m, Up represents a bilinear interpolation 2x upsampling operation, Concat represents a splicing operation based on the channel dimension, BN and G represent batch normalization and nonlinear activation function GELU respectively; the calculation formula of GELU is as follows:

[0054]

[0055] Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, the extracted multi-level fusion feature X F Perform upsampling operation to make the feature map X F The spatial resolution of is gradually increased to a higher level, and the upsampled feature map X is obtained. FU ; Then, each layer’s X FU With the current layer's X F Merge, generate a new feature representation X through splicing operation FC ; Next, X FC With features X from higher levels F+1 The two are inputted into the fusion module for processing;

[0056] Finally, the fused features are passed to a classifier, the classfilier module of the multi-branch convolutional attention network model, for final prediction. The classifier compresses the number of feature map channels output by the top-level DE decoder to 3 categories, which is the classification probability of each pixel. The formula is:

[0057] Y=(DOConv(Upsample(DOConv(DOConv(Y F (1)))))).

[0058] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of realizing the "local-global-local" information interaction bottleneck in step (3.1) is:

[0059] (3.1.1) Local aggregation

[0060] Use efficient depthwise convolution to aggregate local information from neighboring features;

[0061] (3.1.2) Global Sparse Attention

[0062] Through self-attention, a sparse and evenly distributed representative feature set is formed for long-range information exchange, thereby obtaining updated information;

[0063] (3.1.3) Local transmission

[0064] Diffusion of updated information from representative features to non-representative features in the local neighborhood through transposed convolution;

[0065] (3.1.4) Finally, we get the feature map X output by the Transformer branch out :

[0066] X=LocalAgg(Norm(X in ))+X in ;

[0067] Y=FFN(Norm(X))+X;

[0068] Z=LocalProp(GlobalSparseAttn(Norm(Y)))+Y;

[0069] X out =FFN(Norm(Z))+Z;

[0070] Among them, FFN represents a two-layer perceptron and Norm represents a layer normalization operation.

[0071] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the step (4) is to divide the input CSWV data set into a training set and a test set, and use the multi-branch convolutional attention network model to perform image segmentation training on the image data in the training set and the corresponding cloud layer labels and snow layer labels; the specific implementation process is:

[0072] (4.1) First, the cross-entropy loss function is used to compare the mask output by the multi-branch convolutional attention network model with the true label to measure the accuracy of the multi-branch convolutional attention network in segmenting clouds, snow, and background. The cross-entropy measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross-entropy loss is smaller. The cross-entropy loss function is used to measure the difference between the category distribution predicted by the multi-branch convolutional attention network model and the true category distribution.

[0073] The output of the multi-branch convolutional attention network model is a probability distribution:

[0074] p=[p1,p2,…,p c ];

[0075] where p i It represents the predicted probability that the sample belongs to the i-th category, and the distribution of the true category is q=[q1,q2,…,q c ],q i Is the indicator variable of the category corresponding to the true label;

[0076] The formula for calculating cross entropy loss is:

[0077]

[0078] (4.2) Define the training callback mechanism

[0079] A callback function is used to save the weights of the multi-branch convolutional attention network model at any time; a learning rate scheduler is used to optimize the convergence path of the multi-branch convolutional attention network model; and an early stopping mechanism is used to prevent overfitting of the multi-branch convolutional attention network model. That is, when the verification loss does not improve within a specified number of rounds, the image segmentation training is automatically stopped.

[0080] (4.3) According to the feedback of the loss function, the weights of the multi-branch convolutional attention network model are adjusted, and finally accurate cloud and snow classification results are output; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, in which each pixel is classified as background, cloud or snow.

[0081] The optical image segmentation method based on the multi-branch convolutional attention model, wherein: the callback function in the step (4.2) is used to save the weight of the model when training the multi-branch convolutional attention network model, and dynamically save the weight of the multi-branch convolutional attention network model during the training process according to the loss of the training set.

[0082] The optical image segmentation method based on the multi-branch convolutional attention model, wherein: the learning rate scheduler in the step (4.2) automatically reduces the learning rate when the verification loss does not improve for a long time, so as to prevent the multi-branch convolutional attention network model from falling into a local minimum or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path to improve performance.

[0083] By adopting the above technical solution, the present invention has the following beneficial effects:

[0084] The optical image segmentation method based on the multi-branch convolutional attention model of the present invention is reasonably conceived. The proposed multi-branch convolutional attention model uses a module in which convolution layers and Transformer layers are connected in parallel, and introduces a fusion module to replace the common splicing operation. The fusion module can effectively fuse the global information and local information extracted by the two branches, and filter them to retain only the meaningful parts, thereby improving the model efficiency. The multi-branch convolutional attention model can better retain high-level semantic information, while combining low-level detail information to obtain more accurate segmentation results.

[0085] The optical image segmentation method based on the multi-branch convolutional attention model of the present invention is well-conceived. Compared with a single convolutional network, the multi-branch structure can process local and global feature information at the same time by introducing parallel branches, thus avoiding the limitation that a single convolutional network is only good at extracting local features. Although a single Transformer network is good at capturing global information, it lacks the ability to extract local edge and detail features; the multi-branch structure makes up for this shortcoming by combining convolution branches, so that the model can pay attention to local details while modeling globally. At the same time, unlike the method of simply combining convolutional layers and Transformer layers, the multi-branch structure efficiently integrates the features of convolutional layers and Transformer layers through a fusion module, avoiding redundancy and conflict between feature information, thereby improving the accuracy of feature extraction; the branch structure allows the convolutional layer and the Transformer layer to work in parallel in different branches, and can independently optimize the two types of feature extraction paths, so that the network has stronger generalization ability and information extraction efficiency.

[0086] The present invention further integrates shallow and deep features in the decoder through a multi-branch convolutional attention network, and improves the ability to segment cloud and snow edges and texture information in complex scenes through layer-by-layer refinement and reconstruction. It is difficult to achieve similar effects with a single convolution or single Transformer decoder.

[0087] The advantages of the present invention compared to the prior art are mainly reflected in the following aspects:

[0088] (1) For traditional convolution-based deep learning algorithms, the accuracy of feature extraction directly affects the final segmentation accuracy. Previous studies have shown that convolution is better at extracting local information, but lacks accuracy in grasping global information. The characteristics of Transformer can just make up for this shortcoming. The present invention uses convolution layers to extract local features in the image and Transformer layers to grasp global features, thereby effectively extracting cloud / snow features, accurately identifying cloud / snow areas, and optimizing edge details, making the segmentation results more refined.

[0089] (2) Traditional deep learning methods are similar to the simple combination of convolution and Transformer methods, which generally use Concat to splice features and cannot effectively filter features. The multi-branch structure in the present invention efficiently integrates the features of convolution and Transformer through the fusion module, avoiding redundancy and conflict between feature information, thereby improving the accuracy of feature extraction.

[0090] (3) The present invention introduces the Transformer block in EdgeViT, which is achieved by introducing a "local-global-local" information exchange bottleneck based on the optimal integration of self-attention mechanism and convolution. EdgeViTs has shown excellent performance in multiple tasks such as image classification, object detection and semantic segmentation, while significantly reducing computational overhead and latency.

[0091] (4) In the decoding stage, most networks now restore the original image size by directly upsampling the feature map, which easily causes information loss during the upsampling process; some networks only use a single convolution to decode the feature map. Although some important feature information is preserved, since the convolution only focuses on local features, it is impossible to establish long-distance connections in the feature map, and the recovery of large-scale cloud / snow areas is not ideal. This paper proposes a decoder module (DE) that combines convolution with Transformer to gradually restore the effective information in the feature map;

[0092] (5) In order to make the final segmentation result more accurate, a classfilier module is added to the network. It is mainly composed of upsampling and convolution modules. Output feature maps of different levels are drawn at the decoding end to calculate auxiliary losses. Its function is to accelerate the convergence of the network and improve the prediction accuracy. The addition of strip convolution at the output end makes the final output prediction map more refined. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0094] Figure 1 Schematic diagram of the multi-branch convolutional attention network structure involved in the optical image segmentation method based on the multi-branch convolutional attention model of the present invention;

[0095] Figure 2 A specific flowchart of the definition and instantiation of datasets involved in the optical image segmentation method based on the multi-branch convolutional attention model of the present invention. DETAILED DESCRIPTION

[0096] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0097] The present invention will be further explained below with reference to specific embodiments.

[0098] like Figure 1 As shown, this embodiment provides an optical image segmentation method based on a multi-branch convolutional attention model, comprising the following steps:

[0099] (1) Define and instantiate a dataset

[0100] The dataset was uniformly cropped to 512×512×3 image data. The red, green, and blue bands of the image data were loaded, and the corresponding cloud and snow label files were read to generate a binary mask. After stacking the red, green, and blue bands, their data types were converted to floating-point numbers. The pixel values ​​of the image data were normalized, that is, the pixel value range was scaled from the original 0 to 255 to the range of 0 to 1, to obtain the input image data X.

[0101] like Figure 2 As shown in Figure 1, when processing the label data for cloud and snow labels, the data from the read cloud and snow label TIF files needs to be converted into binary mask matrices, as shown in Table 1. First, a mask for the background region is generated by checking whether both the cloud and snow labels are 0. The resulting binary mask matrix marks the background region as 1 and the rest of the region as 0. Next, a mask for the cloud region is generated by checking whether the cloud label is 255, marking the cloud region as 1 and the rest of the region as 0. Similarly, a mask for the snow region is generated by checking whether the snow label is 255, marking the snow region as 1 and the rest of the region as 0. The mask matrices generated for the background region, the cloud region, and the snow region represent the background, cloud, and snow categories, respectively. These are placed into a three-channel array, with each channel representing one of the mask matrices, to generate the label data.

[0102] The label data is converted from integer type to floating-point type by calling the astype method of the NumPy array. This represents the background, cloud, and snow categories, with each channel representing one of them. The binary mask matrices for background, cloud, and snow are then stacked into a three-channel matrix to meet the label requirements of the multi-class segmentation task. Both the image data and the label data are converted to PyTorch tensors to obtain the input label data Y. The obtained input image data X and input label data Y are returned as multiple pairs of input and target for use in training or inference of the multi-branch convolutional attention model.

[0103] Table 1 Classification table of label data background, cloud and snow

[0104]

[0105] (2) Visualizing input data

[0106] The input image data X and the corresponding label data Y are visualized, the red, green, and blue bands and label data are separated from the tensor, the red, green, and blue bands are stacked to generate an RGB image, the label data Y is converted into a category index and a custom color mapping is applied to obtain a label image, and the RGB image and the label image are drawn into a subgraph to intuitively display the input and target output of the multi-branch convolutional attention network model, help verify whether the image preprocessing process in the above step (1) is correct, and provide visualization support for the optimization of the multi-branch convolutional attention network model.

[0107] (3) Define and instantiate a multi-branch convolutional attention network model

[0108] The multi-branch convolutional attention network model combines multi-branch modules with an attention mechanism to extract multi-scale features while dynamically focusing on salient areas. The decoder uses a fusion module to gradually fuse features from different levels, allowing for simultaneous grasp of both details and global information. Ultimately, the decoder's output features are passed to a classifier for final prediction. The number of feature map channels is compressed to 3 categories, representing the classification probability for each pixel. The following describes the detailed process:

[0109] The multi-branch module adopts a branch parallel structure as the backbone network of the multi-branch convolutional attention network model. The branch contains convolution branches, that is, two layers of 3×3 convolution with a residual structure are introduced. After each layer of convolution, a batch normalization BN and ReLU activation function s are performed to finally obtain the feature map x output by the convolution branch. c ;

[0110] C=σ(BN(Conv 3×3 (f i ));

[0111] fi+1 =σ(BN(Conv 3×3 (C));

[0112] x C =Concat(f i ,f i+1 );

[0113] At the same time, the other branch, the Transformer branch, runs in parallel with the convolutional branch. The Transformer branch uses a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers. This is achieved by introducing an optimized structure based on the combination of self-attention and convolution, namely an efficient "local-global-local" information interaction bottleneck, as follows:

[0114] Local aggregation: Use effective depth-wise convolution to aggregate local information from adjacent features;

[0115] Global sparse attention: self-attention is used to form a sparse and evenly distributed set of representative features for long-range information exchange, thereby obtaining updated information;

[0116] Local propagation: diffuses updated information from representative features to non-representative features in the local neighborhood via transposed convolution.

[0117] The specific process of realizing the “local-global-local” information exchange bottleneck is as follows: Finally, the feature map X output by the Transformer branch is obtained. out :

[0118] X=LocalAgg(Norm(X in ))+X in ;

[0119] Y=FFN(Norm(X))+X;

[0120] Z=LocalProp(GlobalSparseAttn(Norm(Y)))+Y;

[0121] X out =FFN(Norm(Z))+Z;

[0122] Among them, FFN represents a two-layer perceptron and Norm represents a layer normalization operation.

[0123] Therefore, the multi-branch module uses the convolution branch to extract local features in the image data and the Transformer branch to grasp the global features in the image data.

[0124] Subsequently, a fusion module of a multi-branch convolutional attention network model is used to replace the simple splicing of multiple feature channels in the channel dimension to integrate the different features of the convolution branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation.

[0125] The fusion module integrates the different features of the convolution branch and the Transformer branch. The specific process is as follows:

[0126] First, the high-level features are upsampled to the same scale as the low-level features, and then two convolution kernels with sizes of 1×3 and 3×1 are used to filter and extract features from the two respectively; then, batch normalization and activation function GELU are applied for further processing; then, the information of the two branches is spliced, and the spliced ​​features are again filtered and filtered by two convolution kernels with sizes of 1×3 and 3×1 to establish a complementary relationship and obtain the extracted multi-level fusion feature X F At the decoding end of the multi-branch convolutional attention network, the category information of high-level features can guide the classification of low-level features, and the position information of low-level features supplements the spatial information of high-level features. The calculation process of the fusion module is as follows:

[0127] X1=G(BN(DOConv 3×1 (DOConv 1×3 (X in1 ))));

[0128] X2=G(BN(DOConv 3×1 (DOConv 1×3 (Up(X in2 )))));

[0129] W = Concat(X1,X2);

[0130] Y out =G(BN(DOConv 3×1 (DOConv 1×3 (W))));

[0131] Among them, X in1 and X in2 Represent the two inputs of the fusion module, Y out Represents the output of the fusion module, DOConv 3×1 Represents DO-Conv with a convolution kernel size of n×m, Up represents a bilinear interpolation 2x upsampling operation, Concat represents a concatenation operation based on the channel dimension, BN and G represent batch normalization and nonlinear activation function GELU respectively. The calculation formula of GELU is as follows:

[0132]

[0133] Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, the extracted multi-level fusion feature X F Perform upsampling operation to make the feature map X F The spatial resolution of is gradually increased to a higher level, and the upsampled feature map X is obtained. FU . Then, each layer’s X FU With the current layer's X F Merge, generate a new feature representation X through splicing operation FC Then, X FC With features X from higher levels F+1 The two branches are input into the fusion module for processing. In this process, not only the feature information from the convolution branch and the Transformer branch is integrated, but also the upsampled feature map X is included. FU The decoder effectively combines the advantages of shallow and deep features, further enhancing the network's ability to capture multi-scale features and detailed information.

[0134] Finally, the fused features are passed to a classifier, namely the classfilier module, for final prediction. The classifier compresses the number of feature map channels output by the top-level DE decoder to 3 categories, which is the classification probability of each pixel. The formula is:

[0135] Y=(DOConv(Upsample(DOConv(DOConv(Y F (1))))));

[0136] (4) Training multi-branch convolutional attention network model

[0137] The input CSWV dataset is divided into a training set and a test set. A multi-branch convolutional attention network model is used to perform image segmentation training on the image data in the training set and the corresponding cloud and snow layer labels. The following describes the specific implementation process:

[0138] First, the cross entropy loss function is used to compare the mask output by the multi-branch convolutional attention network model with the true label to measure the accuracy of the multi-branch convolutional attention network in segmenting clouds, snow, and background. The cross entropy measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross entropy loss is smaller. The cross entropy loss function is used to measure the difference between the category distribution predicted by the multi-branch convolutional attention network model and the true category distribution, which can well guide the optimization of the multi-branch convolutional attention network model. For the multi-classification problem that needs to be solved by the present invention, the output of the multi-branch convolutional attention network model is a probability distribution p = [p1, p2, ..., p c ], where p i It represents the predicted probability that the sample belongs to the i-th category, and the distribution of the true category is q=[q1,q2,…,q c ],q i Is the indicator variable of the category corresponding to the true label. The formula for calculating cross entropy loss is:

[0139]

[0140] Subsequently, during the image segmentation training process, three training callback mechanisms are added:

[0141] A callback function is used to save the weights of the multi-branch convolutional attention network model at any time; the callback function is used to save the weights of the model when training the multi-branch convolutional attention network model, and dynamically save the weights of the model during the training process according to the loss of the training set.

[0142] A learning rate scheduler (such as ReduceLROnPlateau) is used to optimize the convergence path of the multi-branch convolutional attention network model, making the training process more stable and improving the final performance. When the verification loss does not improve for a long time, the learning rate scheduler automatically reduces the learning rate to prevent the model from falling into a local minimum or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path, thereby improving performance.

[0143] An early stopping mechanism is used to prevent overfitting of the multi-branch convolutional attention network model. That is, when the validation loss does not improve within a specified number of rounds, the image segmentation training is automatically stopped. The training parameters are set as follows:

[0144] Table 2 Model training parameters

[0145] Parameter name Parameter settings illustrate Learning rate optimizer Adam The learning rate optimizer selects the Adam optimizer Loss Function CrossEntropyLoss The loss function is cross entropy loss. Total training cycle 1000 The total number of training cycles is set to 1000 Batch size 3 The batch size is set to 3 Learning rate 0.001 The initial learning rate is set to 0.001

[0146] According to the feedback of the loss function, the weights of the multi-branch convolutional attention network model are adjusted, and finally an accurate cloud and snow classification result is output; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, in which each pixel is classified as background, cloud or snow.

[0147] The present invention has a reasonable concept and can solve the technical problem in the prior art of cloud and snow segmentation tasks that improper processing of complex boundaries and details will significantly reduce the accuracy of the model.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optical image segmentation method based on a multi-branch convolutional attention model, characterized in that: The following steps are involved: (1) Define and instantiate the CSWV dataset; (2) Visualize input data; (3) Define and instantiate a multi-branch convolutional attention network model; the specific process is: (3.1) The multi-branch module adopts a branch parallel structure as the backbone network of the multi-branch convolutional attention network model. The branch contains convolution branches, that is, two layers of 3×3 convolution with a residual structure are introduced. After each convolution layer, a batch normalization BN and ReLU activation function are performed. s , and finally obtain the feature map output by the convolution branch through the residual structure x c : ； ； ； The other branch, the Transformer branch, runs in parallel with the convolutional branch. The Transformer branch uses a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers. This is achieved by introducing an optimized structure based on a combination of self-attention and convolution, namely an efficient "local-global-local" information exchange bottleneck. The multi-branch module uses the convolutional branch to extract local features from the image data and the Transformer branch to grasp the global features in the image data. The specific process of implementing the "local-global-local" information exchange bottleneck is as follows: (3.1.1) Local aggregation Use efficient depthwise convolution to aggregate local information from neighboring features; (3.1.2) Global Sparse Attention Through self-attention, a sparse and evenly distributed representative feature set is formed for long-range information exchange, thereby obtaining updated information; (3.1.3) Local transmission Diffusion of updated information from representative features to non-representative features in the local neighborhood through transposed convolution; (3.1.4) Finally, we get the feature map output by the Transformer branch X out : ； ； ； ； in, FFN represents a two-layer perceptron, Norm Representation layer normalization operation; (3.2) Using a fusion module of a multi-branch convolutional attention network model to replace the simple splicing of multiple feature channels in the channel dimension, it is used to integrate the different features of the convolution branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation; The fusion module integrates the different features of the convolution branch and the Transformer branch. The specific process is as follows: first, the high-level features are upsampled to the same scale as the low-level features, and then two convolution kernels with sizes of 1×3 and 3×1 are used to filter and extract features from the two respectively; then, batch normalization and activation function GELU are applied for further processing; then, the information of the two branches is spliced ​​together, and the spliced ​​features are again screened and filtered by two convolution kernels with sizes of 1×3 and 3×1 to establish a complementary relationship and obtain the extracted multi-level fusion features. X F At the decoding end of the multi-branch convolutional attention network, the category information of high-level features can guide the classification of low-level features, and the position information of low-level features complements the spatial information of high-level features. The calculation process of the fusion module is as follows: ； ； ； ； in, X in1 and X in2 Represent the two inputs of the fusion module, Y out represents the output of the fusion module; DOConv 3×1 Represents DO-Conv with a convolution kernel size of n×m, Up Indicates a bilinear interpolation 2x upsampling operation, Concat Represents the splicing operation based on the channel dimension, BN and G Represent batch normalization and nonlinear activation function GELU respectively; the calculation formula of GELU is as follows: ； Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, the extracted multi-level fusion feature X F Perform upsampling operation to make the feature map X F The spatial resolution of the upsampled feature map is gradually increased to a higher level. X FU ; Then, each layer X FU With the current layer X F Merge, generate new feature representations through splicing operations X FC ; Then, X FC With features from higher levels X F+1 The two are inputted into the fusion module for processing; Finally, the fused features are passed to a classifier, the classfilier module of the multi-branch convolutional attention network model, for final prediction. The classifier compresses the number of channels of the feature map output by the top-level DE decoder to the number of categories 3, which is the classification probability of each pixel point, and the formula is: ； (4) Train the multi-branch convolutional attention network model.

2. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, wherein: The specific process of step (1) is as follows: (1.01) Loading image data Open the image data in each CSWV dataset and extract data of different bands from the image data; (1.02) Stacking image data Stack data from different bands together to form image data from different channels; (1.03) Image data conversion arrangement Convert the arrangement of image data from height, width, channel to channel, height, width; (1.04) Image data normalization Normalize the pixel values ​​of the image data, that is, divide each pixel value by 255.0, thereby scaling the pixel value range from the original 0 to 255 to the interval of 0 to 1, and obtain the input image data X; (1.05) Read tag data Load the tif files of cloud labels and snow labels corresponding to each image data, indicating whether each pixel of the image data belongs to cloud or snow respectively; (1.06) Processing label data The data of the read cloud label and snow label tif files are converted into binary mask matrices. First, the mask of the background area is generated by checking whether the cloud label and the snow label are both 0. The generated binary mask matrix marks the background area as 1 and the rest of the area as 0. Then, the mask of the cloud area is generated by checking whether the cloud label is 255, indicating that the cloud area is 1 and the rest of the area is 0. The mask of the snow area is generated by checking whether the snow label is 255, marking the snow area as 1 and the rest of the area as 0. The mask matrices generated by the background area, the cloud area, and the snow area represent the categories of background, cloud, and snow, respectively. They are put into a three-channel array, where each channel represents one of the mask matrices, thereby generating label data. (1.07) Tag data type conversion Convert the label data from integer type to floating point type by calling the astype method of NumPy array; (1.08) Merge Mask Stack the binary mask matrices of background, cloud, and snow into a three-channel matrix; (1.09) Convert to Tensor Convert both image data and label data into PyTorch tensors to obtain input label data Y; (1.10) Return data The obtained input image data X and input label data Y are returned as multiple pairs of input and target for multi-branch convolutional attention model training or inference.

3. The optical image segmentation method based on a multi-branch convolutional attention model according to claim 2, wherein: Before loading the image data in step (1), the CSWV dataset is uniformly cropped into image data of 512×512×3.

4. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 2, wherein: The specific process of step (2) is as follows: visualizing the input image data X and the corresponding label data Y, separating the red, green, and blue bands and label data from the Pytorch tensor, stacking the red, green, and blue bands to generate an RGB image, converting the one-hot encoding of the label data Y into a category index and applying a custom color mapping to obtain a label image, drawing the RGB image and the label image into a subgraph, and intuitively displaying the input and target output of the multi-branch convolutional attention network model, helping to verify whether the process of steps (1.01)-(1.09) is correct, and providing visualization support for the optimization of the multi-branch convolutional attention network model.

5. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, wherein: The step (4) is to divide the input CSWV dataset into a training set and a test set, and use the multi-branch convolutional attention network model to perform image segmentation training on the image data in the training set and the corresponding cloud layer labels and snow layer labels; the specific implementation process is: (4.1) First, the cross-entropy loss function is used to compare the mask output by the multi-branch convolutional attention network model with the true label to measure the accuracy of the multi-branch convolutional attention network in segmenting clouds, snow, and background. The cross-entropy loss function measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross-entropy loss is smaller. The cross-entropy loss function is used to measure the difference between the category distribution predicted by the multi-branch convolutional attention network model and the true category distribution. The output of the multi-branch convolutional attention network model is a probability distribution: p=[p 1 ,p 2 ,…, p c ] ； in p i Indicates that the sample belongs to i The predicted probability of the class, the distribution of the true class is q=[q 1 ,q 2 ,…,q c ] , q i Is the indicator variable of the category corresponding to the true label; The calculation formula for cross entropy loss is: ； (4.2) Define the training callback mechanism A callback function is used to save the weights of the multi-branch convolutional attention network model at any time; a learning rate scheduler is used to optimize the convergence path of the multi-branch convolutional attention network model; and an early stopping mechanism is used to prevent overfitting of the multi-branch convolutional attention network model. That is, when the verification loss does not improve within a specified number of rounds, the image segmentation training is automatically stopped. (4.3) Based on the feedback of the loss function, the weights of the multi-branch convolutional attention network model are adjusted to finally output accurate cloud and snow classification results; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, in which each pixel is classified as background, cloud or snow.

6. The optical image segmentation method based on a multi-branch convolutional attention model according to claim 1, wherein: The callback function in the step (4.2) is used to save the weights of the model when training the multi-branch convolutional attention network model, and dynamically save the weights of the multi-branch convolutional attention network model during the training process according to the loss of the training set.

7. The optical image segmentation method based on a multi-branch convolutional attention model according to claim 1, wherein: The learning rate scheduler in step (4.2) automatically reduces the learning rate when the verification loss does not improve for a long time, so as to prevent the multi-branch convolutional attention network model from falling into a local minimum or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path to improve performance.

Images