Optical image segmentation method based on multi-branch convolution attention model

Abstract

The invention provides an optical image segmentation method based on a multi-branch convolution attention model. The method comprises the following steps: (1) defining and instantiating a data set; (2) visually inputting data; (3) defining and instantiating a multi-branch convolutional attention network model; and (4) training the multi-branch convolutional attention network model. According to the multi-branch convolution attention model provided by the invention, through convolution and Transfomer parallel modules, a direct jump connection is added in each convolution module, and meanwhile, a fusion module is introduced to replace common splicing operation, so that global information and local information extracted by two branches can be effectively fused, and the fusion efficiency is improved. The model is filtered, and only meaningful parts are reserved, so that the model efficiency is improved; the multi-branch convolution attention model can better reserve high-level semantic information, and meanwhile, a more accurate segmentation result is obtained in combination with low-level detail information.

Description

Technical Field

[0001] The present invention relates to the technical field of optical image recognition, and particularly relates to an optical image segmentation method based on a multi-branch convolutional attention model. Background Art

[0002] The segmentation of clouds and snow is of great significance in the field of remote sensing, especially in tasks such as surface information extraction and climate monitoring. The presence of clouds will obscure surface features and affect the accuracy of surface classification, while the extent and distribution of snow cover are important indicators for evaluating water resource reserves and environmental changes. Accurately segmenting clouds and snow can not only improve the resolution of remote sensing images, but also provide reliable data support for global change research. In addition, the shapes and spectral characteristics of clouds and snow have certain similarities, which increases the difficulty of the segmentation task. Therefore, the research on cloud and snow segmentation technology plays an important role in improving the application value of remote sensing images and promoting earth science research.

[0003] The accuracy of feature extraction directly affects the final segmentation performance, which is particularly important in the cloud and snow segmentation task. In cloud and snow segmentation, image data usually contains problems such as complex textures, illumination changes, and thin cloud cover. Previous studies have shown that Convolutional Neural Networks (CNNs) perform well in extracting local features, especially having significant advantages in edge detection and texture segmentation. However, CNNs have deficiencies in capturing global context information. Especially when dealing with large-scale spectral changes and cross-regional similarities, information loss or inaccuracies are likely to occur.

[0004] In recent years, the Transformer architecture has become a potential method to solve this problem due to its powerful global modeling ability. The Transformer uses the self-attention mechanism and can capture long-range context dependencies in image segmentation tasks. In this way, the Transformer can effectively make up for the shortcoming of the convolutional network in global information extraction. Especially in cloud and snow segmentation, this ability can better distinguish regions that are spectrally similar but different in spatial positions.

[0005] Common CNN models (such as UNet, UNet++) show high efficiency in the cloud and snow segmentation task. These models can extract rich local features through multi-scale convolution and skip connections, and effectively fuse shallow and deep features. However, in the face of snow-covered areas or thin cloud areas with high spectral complexity, the model may have difficulty accurately distinguishing boundaries or similar pixels.

[0006] Architectures such as Vision Transformer (ViT) and Swin Transformer based on Transformer have achieved preliminary results in cloud and snow segmentation tasks. They capture large-scale context information through the global attention mechanism, enabling more accurate classification of thick and thin cloud regions. However, Transformer has a high computational cost and relatively weak generalization ability for small-sample data.

[0007] In recent years, some studies have proposed hybrid architectures that fuse convolution and Transformer to improve the performance of cloud and snow segmentation. For example:

[0008] ConvNext-Transformer model: Combines the parallel processing paths of CNN and Transformer. Although it improves the ability to extract local and global features, its complex design results in a large number of model parameters, high demand for hardware resources, and a significant increase in training costs. In addition, the complex network structure increases the difficulty of debugging and optimization, and is prone to problems of unstable training. In this model, the CNN and Transformer paths run in parallel, and there may be problems of information redundancy or conflict in the fusion stage, leading to a decline in feature extraction and classification performance. When dealing with spectrally complex regions, the fusion mechanism may not be able to fully integrate multi-scale features, affecting the model's ability to capture details, especially at the cloud and snow boundaries or in thin cloud regions. This model tends to fuse macroscopic global information and local features, but its ability to refine boundary regions is relatively insufficient, easily resulting in blurred boundaries in the segmentation results, especially performing poorly in complex scenarios at the cloud and snow junctions.

[0009] Swin-Unet model: Although the hierarchical Transformer architecture adopted by this model has improved global modeling ability, its windowing operation leads to truncation of context information. In cloud and snow segmentation, its ability to model long-range dependencies across windows is weak, and it may miss important global information. It attempts to achieve a balance in local and global feature extraction, but due to excessive dependence on the hierarchical mechanism, the contribution of shallow features in the multi-scale context may be weakened. Especially when dealing with complex boundary regions, the model may not be able to fully capture low-level detail features, resulting in poor boundary refinement effects. At the same time, this model has a high dependence on high-quality training data. In the cloud and snow segmentation task, the cost of labeled data is high, and the model is prone to overfitting when the data volume is insufficient, and its generalization performance is limited.

[0010] In summary, it is necessary to further innovate the existing technologies. Summary of the Invention

[0011] In view of the technical problems existing in the above-mentioned background art, the present invention proposes an optical image segmentation method based on a multi-branch convolutional attention model. Its concept is reasonable. By introducing a parallel branch to design a multi-branch structure, local and global feature information are processed simultaneously, avoiding the limitation that a single convolutional network is only good at extracting local features. By efficiently integrating the features of convolution and Transformer, the redundancy and conflict between feature information are avoided, thereby improving the accuracy of feature extraction. It can solve the technical problem that in the segmentation task of clouds and snow in the prior art, improper processing of complex boundaries and details will significantly reduce the accuracy of the model.

[0012] To solve the above technical problems, an optical image segmentation method based on a multi-branch convolutional attention model provided by the present invention includes the following steps:

[0013] (1) Define and instantiate the dataset;

[0014] (2) Visualize the input data;

[0015] (3) Define and instantiate the multi-branch convolutional attention network model;

[0016] (4) Train the multi-branch convolutional attention network model.

[0017] For the optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of the step (1) is as follows:

[0018] (1.01) Load the image data

[0019] Open the image data in each CSWV dataset, and extract the data of different bands from the image data;

[0020] (1.02) Stack the image data

[0021] Stack the data of different bands together to form image data of different channels;

[0022] (1.03) Convert the arrangement mode of the image data

[0023] Convert the arrangement mode of the image data from height, width, channel to channel, height, width;

[0024] (1.04) Normalize the image data

[0025] Normalize the pixel values of the image data, that is, divide each pixel value by 255.0, so as to scale the value range of the pixel values from the original 0 to 255 to the interval of 0 to 1, and obtain the input image data X;

[0026] (1.05) Read the label data

[0027] Load the tif files of cloud labels and snow labels corresponding to each image data, which respectively indicate whether each pixel point of the image data belongs to cloud or snow;

[0028] (1.06) Process label data

[0029] Convert the data of the read tif files of cloud labels and snow labels into a binary mask matrix. First, the mask of the background area is generated by checking whether the cloud label and the snow label are both 0. The generated binary mask matrix marks the background area as 1 and the rest of the areas as 0; then, the mask of the cloud area is generated by checking whether the cloud label is 255, indicating that the cloud area is 1 and other areas are 0; the mask of the snow area is generated by checking whether the snow label is 255, marking the snow area as 1 and other areas as 0; the mask matrix generated by the background area, the mask matrix generated by the cloud area, and the mask matrix of the snow area respectively represent the categories of background, cloud, and snow, and are put into a three-channel array, where each channel represents one of the mask matrices, thus generating label data;

[0030] (1.07) Label data type conversion

[0031] Convert the label data from integer type to floating-point type by calling the astype method of the NumPy array;

[0032] (1.08) Merge masks

[0033] Stack the binary mask matrices of background, cloud, and snow into a three-channel matrix;

[0034] (1.09) Convert to tensor

[0035] Convert both the image data and the label data into PyTorch tensors to obtain the input label data Y;

[0036] (1.10) Return data

[0037] Return the obtained input image data X and input label data Y as multiple pairs of inputs and targets for training or inference of the multi-branch convolutional attention model.

[0038] For the optical image segmentation method based on the multi-branch convolutional attention model, wherein: before loading the image data in the step (1), the CSWV dataset is uniformly cropped into image data of 512×512×3.

[0039] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of step (2) is as follows: Visualize the input image data X and the corresponding label data Y, separate the red, green, and blue bands and the label data from the Pytorch tensor, stack the red, green, and blue bands to generate an RGB image, convert the one-hot encoding of the label data Y into class indices and apply a custom color mapping to obtain a label image, and draw the RGB image and the label image into subgraphs to visually display the input and target output of the multi-branch convolutional attention network model, helping to verify whether the process of steps (1.01)-(1.09) is correct and providing visual support for the optimization of the multi-branch convolutional attention network model.

[0040] The optical image segmentation method based on the multi-branch convolutional attention model, wherein the specific process of step (3) is as follows:

[0041] (3.1) The multi-branch module adopts a branch parallel structure and serves as the backbone network of the multi-branch convolutional attention network model. The branch contains a convolutional branch, that is, two 3×3 convolutions with a residual structure are introduced. After each convolution, there is also a batch normalization BN and a ReLU activation function s. Finally, through the residual structure, the feature map x output by the convolutional branch is obtained. c :

[0042] C = σ(BN(Conv 3×3 (f i )));

[0043] f i+1 = σ(BN(Conv 3×3 (C));

[0044] x C = Concat(f i , f i+1 );

[0045] Another branch: the Transformer branch, which is parallel to the convolutional branch. The Transformer branch adopts a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers, which is realized by introducing an optimized structure combining self-attention and convolution, that is, an efficient "local-global-local" information interaction bottleneck. Thus, the multi-branch module uses the convolutional branch to extract local features in the image data and the Transformer branch to capture global features in the image data.

[0046] (3.2) The fusion module using the multi-branch convolutional attention network model replaces the simple concatenation of multiple feature channels in the channel dimension, and is used to integrate the different features of the convolutional branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation;

[0047] The specific process of the fusion module integrating the different features of the convolutional branch and the Transformer branch is as follows: First, the high-level features are upsampled to the same scale as the low-level features, and then the two are filtered and feature-extracted respectively using two convolutions with kernel sizes of 1×3 and 3×1; Next, batch normalization and the activation function GELU are applied for further processing; Subsequently, the information of the two branches is concatenated, and the concatenated features are screened and filtered again through two convolutions with kernel sizes of 1×3 and 3×1 to establish a complementary relationship, obtaining the extracted multi-level fusion feature X F ; at the decoding end of the multi-branch convolutional attention network, the category information of the high-level features can guide the classification of the low-level features, and the position information of the low-level features complements the spatial information of the high-level features;

[0048] The calculation process of the fusion module is as follows:

[0049] X1 = G(BN(DOConv 3×1 (DOConv 1×3 (X in1 ))));

[0050] X2 = G(BN(DOConv 3×1 (DOConv 1×3 (Up(X in2 ))));

[0051] W = Concat(X1, X2);

[0052] Y out = G(BN(DOConv 3×1 (DOConv 1×3 (W))));

[0053] Among them, X in1 and X in2 respectively represent the two inputs of the fusion module, and Y out represents the output of the fusion module; DOConv 3×1 represents DO-Conv with a kernel size of n×m, Up represents the bilinear interpolation 2-fold upsampling operation, Concat represents the concatenation operation based on the channel dimension, and BN and G respectively represent batch normalization and the non-linear activation function GELU; The calculation formula of GELU is as follows:

[0054]

[0055] Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, first, the multi-level fusion features X extracted F are subjected to an upsampling operation to gradually increase the spatial resolution of the feature map X F to a higher level, obtaining the upsampled feature map X FU ; Subsequently, each layer of X FU is merged with the X of the current layer F to generate a new feature representation X through a concatenation operation FC ; Then, X FC and the feature X from a higher level F+1 are jointly input into the fusion module for processing;

[0056] Finally, the fused features are passed to a classifier, namely the classfilier module of the multi-branch convolutional attention network model, for final prediction; the classifier compresses the number of channels of the feature map output by the highest-level DE decoder to the number of categories 3, which is the classification probability of each pixel point, and the formula is:

[0057] Y = (DOConv(Upsample(DOConv(DOConv(Y F (1)))))).

[0058] In the optical image segmentation method based on the multi-branch convolutional attention model, the specific process of implementing the "local-global-local" information interaction bottleneck in the step (3.1) is as follows:

[0059] (3.1.1) Local aggregation

[0060] Use effective depth convolution to aggregate local information from adjacent features;

[0061] (3.1.2) Global sparse attention

[0062] Form a sparse and uniform distribution of representative feature sets through self-attention for remote information exchange, thereby obtaining updated information;

[0063] (3.1.3) Local propagation

[0064] Diffuse the updated information from the representative features to the non-representative features in the local neighborhood through transposed convolution;

[0065] (3.1.4) Finally, obtain the feature map X output by the Transformer branch out :

[0066] X = LocalAgg(Norm(X in )) + X in ;

[0067] Y = FFN(Norm(X)) + X;

[0068] Z = LocalProp(GlobalSparseAttn(Norm(Y))) + Y;

[0069] X out = FFN(Norm(Z)) + Z;

[0070] Among them, FFN represents a two - layer perceptron, and Norm represents a layer normalization operation.

[0071] In the optical image segmentation method based on the multi - branch convolutional attention model, in step (4), the input CSWV dataset is divided into a training set and a test set, and the multi - branch convolutional attention network model is used to perform image segmentation training on the image data and the corresponding cloud layer labels and snow layer labels in the training set; the specific implementation process is as follows:

[0072] (4.1) First, the cross - entropy loss function is adopted to compare the mask output by the multi - branch convolutional attention network model with the true label, and to measure the accuracy of the multi - branch convolutional attention network in segmenting clouds, snow, and background; cross - entropy measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross - entropy loss is smaller; the cross - entropy loss function is used to measure the difference between the class distribution predicted by the multi - branch convolutional attention network model and the true class distribution.

[0073] The output of the multi - branch convolutional attention network model is a probability distribution:

[0074] p = [p1, p2, …, p c ;

[0075] Among them, p i represents the predicted probability that the sample belongs to the i - th class, and the distribution of the true class is q = [q1, q2, …, q c , q i is an indicator variable corresponding to the true label class.

[0076] The calculation formula for cross - entropy loss is:

[0077]

[0078] (4.2) Define the training callback mechanism

[0079] Use a callback function to save the weights of the multi-branch convolutional attention network model at any time; use a learning rate scheduler to optimize the convergence path of the multi-branch convolutional attention network model; use an early stopping mechanism to prevent the multi-branch convolutional attention network model from overfitting, that is, when the validation loss does not improve within a specified number of epochs, automatically stop the image segmentation training.

[0080] (4.3) According to the feedback of the loss function, adjust the weights of the multi-branch convolutional attention network model, and finally output an accurate cloud and snow classification result; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, where each pixel is classified as background, cloud or snow.

[0081] The optical image segmentation method based on the multi-branch convolutional attention model, wherein: the callback function in the step (4.2) is used to save the weights of the model when training the multi-branch convolutional attention network model, and dynamically save the weights of the multi-branch convolutional attention network model according to the loss of the training set during the training process.

[0082] The optical image segmentation method based on the multi-branch convolutional attention model, wherein: the learning rate scheduler in the step (4.2) automatically reduces the learning rate when the validation loss has not improved for a long time, to avoid the multi-branch convolutional attention network model falling into local minima or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path to improve performance.

[0083] Adopting the above technical solutions, the present invention has the following beneficial effects:

[0084] The optical image segmentation method based on the multi-branch convolutional attention model of the present invention is reasonably conceived. The proposed multi-branch convolutional attention model has a module with parallel convolutional layers and Transformer layers, and at the same time introduces a fusion module (Fusion Module) to replace the common splicing operation. It can effectively fuse the global information and local information extracted by the two branches, and filter them, only retaining the meaningful parts, to improve the model efficiency; the multi-branch convolutional attention model can better retain high-level semantic information, and at the same time combine low-level detail information to obtain a more accurate segmentation result.

[0085] The optical image segmentation method based on the multi-branch convolutional attention model of the present invention is reasonably conceived. Compared with a single convolutional network, the multi-branch structure can process local and global feature information simultaneously by introducing parallel branches, avoiding the limitation that a single convolutional network is only good at extracting local features. Although a single Transformer network is good at capturing global information, its ability to extract local edge and detail features is insufficient; the multi-branch structure makes up for this defect by combining convolutional branches, enabling the model to also pay attention to local details while performing global modeling. At the same time, different from the method of simply combining convolutional layers and Transformer layers, the multi-branch structure efficiently integrates the features of convolutional layers and Transformer layers through a fusion module, avoiding redundancy and conflict between feature information, thereby improving the accuracy of feature extraction; the branch structure allows convolutional layers and Transformer layers to work in parallel in different branches, and can independently optimize the two types of feature extraction paths, making the network have stronger generalization ability and information extraction efficiency.

[0086] In the present invention, the multi-branch convolutional attention network further fuses shallow and deep features in the decoder, and improves the segmentation ability of cloud / snow edges and texture information in complex scenes by layer-by-layer refinement and reconstruction, while a single convolutional or single Transformer decoder is difficult to achieve a similar effect.

[0087] The advantages of the present invention compared with the prior art are mainly reflected in the following aspects:

[0088] (1) For traditional deep learning algorithms based on convolution, since the accuracy of feature extraction directly affects the final segmentation accuracy, previous studies have shown that convolution is excellent at extracting local information, but lacks accuracy in grasping global information, and the characteristics of Transformer can just make up for this shortcoming; the present invention uses convolutional layers to extract local features in images and Transformer layers to grasp global features, thereby being able to effectively extract cloud / snow features, accurately identify cloud / snow regions at the same time, and optimize edge details, making the segmentation result more refined;

[0089] (2) Traditional deep learning methods similar to simply combining convolution and Transformer generally use Concat to splice features and cannot effectively screen features; the multi-branch structure in the present invention efficiently integrates the features of convolution and Transformer through a fusion module, avoiding redundancy and conflict between feature information, thereby improving the accuracy of feature extraction;

[0090] (3) The present invention introduces the Transformer block in EdgViT, which is achieved by introducing an "local-global-local" information exchange bottleneck based on the optimal integration of self-attention mechanism and convolution. In multiple tasks such as image classification, object detection, and semantic segmentation, EdgeViTs have shown excellent performance, and at the same time, can significantly reduce the computational overhead and latency;

[0091] (4) In the decoding stage, most current networks directly upsample the feature map to restore the original image size, which is very likely to cause information loss during the upsampling process; some networks only use a single convolution to decode the feature map. Although some important feature information is retained, since convolution only focuses on local features and cannot establish long-distance connections in the feature map, the restoration of large-scale cloud / snow regions is not ideal. The present invention proposes a decoder module (DE) that combines convolution and Transformer to gradually restore the effective information in the feature map;

[0092] (5) In order to make the final segmentation result more accurate, a classfilier module is added to the network, which is mainly composed of an upsampling and a convolution module. Different levels of output feature maps are introduced at the decoding end to calculate the auxiliary loss, which functions to accelerate the convergence of the network and improve the prediction accuracy. The addition of strip-shaped convolution at the output end makes the final output prediction map more refined. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0094] Figure 1 Schematic diagram of the multi-branch convolutional attention network structure involved in the optical image segmentation method based on the multi-branch convolutional attention model of the present invention;

[0095] Figure 2 Specific flowchart of defining and instantiating the dataset involved in the optical image segmentation method based on the multi-branch convolutional attention model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0096] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0097] The present invention will be further explained and illustrated below in conjunction with specific implementation manners.

[0098] As Figure 1 shown, an optical image segmentation method based on a multi-branch convolutional attention model provided in this embodiment includes the following steps:

[0099] (1) Define and instantiate the dataset

[0100] The dataset is uniformly cropped into image data of 512×512×3. The red, green, and blue bands of the image data are loaded, and the corresponding cloud and snow label files are read to generate binary masks. After stacking the red, green, and blue bands, their data types are converted to floating-point numbers, and at the same time, the pixel values of the image data are normalized, that is, the value range of the pixel values is scaled from the original 0 to 255 to the interval of 0 to 1, and the input image data X is obtained.

[0101] As Figure 2 shown, when processing the label data of cloud labels and snow labels, it is necessary to convert the data of the tif files of the read cloud labels and snow labels into binary mask matrices. As shown in Table 1, first, the mask of the background area is generated by checking whether the cloud label and the snow label are both 0. The generated binary mask matrix marks the background area as 1 and the remaining areas as 0. Then, the mask of the cloud area is generated by checking whether the cloud label is 255, indicating that the cloud area is 1 and other areas are 0. Similarly, the mask of the snow area is generated by checking whether the snow label is 255, marking the snow area as 1 and other areas as 0; the mask matrices generated from the background area, the cloud area, and the snow area respectively represent the categories of background, cloud, and snow, and are placed in a three-channel array, with each channel representing one of the mask matrices, thereby generating label data.

[0102] Convert the label data from integer type to floating - point type by calling the astype method of the NumPy array; thereby representing the categories of background, cloud, and snow respectively, with each channel representing one of them; subsequently, stack the binary mask matrices of background, cloud, and snow into a three - channel matrix to meet the label requirements in the multi - class segmentation task. Convert both the image data and the label data into PyTorch tensors to obtain the input label data Y. Return the obtained input image data X and input label data Y as multiple pairs of input and target for use in training or inference of the multi - branch convolutional attention model.

[0103] Table 1 Definition Table of the Categories of Background, Cloud, and Snow in Label Data

[0104]

[0105] (2) Visualize the input data

[0106] Perform visualization processing on the input image data X and the corresponding label data Y. Separate the red, green, and blue bands and the label data from the tensor. Stack the red, green, and blue bands to generate an RGB image. Convert the label data Y to class indices and apply a custom color mapping to obtain a label image. Draw the RGB image and the label image into sub - plots to visually display the input and target output of the multi - branch convolutional attention network model, helping to verify whether the image pre - processing process in step (1) above is correct and providing visual support for optimizing the multi - branch convolutional attention network model.

[0107] (3) Define and instantiate the multi - branch convolutional attention network model

[0108] The multi - branch convolutional attention network model combines a multi - branch module and an attention mechanism, which can not only extract multi - scale features but also dynamically focus on significant regions; in the decoder part, different - level features are gradually fused through a fusion module to achieve a simultaneous grasp of details and global information; finally, the output features of the decoder are passed to a classifier for final prediction. The number of channels of the feature map is compressed to the number of classes 3, which is the classification probability of each pixel point. The following introduces the specific process:

[0109] The multi - branch module adopts a parallel - branch structure as the backbone network of the multi - branch convolutional attention network model. Each branch contains a convolutional branch, that is, two - layer 3×3 convolutions with a residual structure are introduced. After each convolution, there is also a batch normalization BN and a ReLU activation function s, and finally, the feature map x output by the convolutional branch is obtained c ;

[0110] C = σ(BN(Conv 3×3 (f i ));

[0111] fi+1 = σ(BN(Conv 3×3 (C));

[0112] x C = Concat(f i , f i+1 );

[0113] Meanwhile, another branch: the Transformer branch, runs in parallel with the convolutional branch. The Transformer branch adopts a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers, which is implemented by introducing an optimized structure combining self-attention and convolution, namely, an efficient "local-global-local" information interaction bottleneck, as follows:

[0114] Local aggregation: Use effective depth convolution to aggregate local information from adjacent features;

[0115] Global sparse attention: Form a sparse and uniformly distributed representative feature set through self-attention for remote information exchange, thereby obtaining updated information;

[0116] Local propagation: Diffuse the updated information from the representative features to the non-representative features in the local neighborhood through transposed convolution.

[0117] The specific process of implementing the "local-global-local" information exchange bottleneck is as follows, and finally obtain the feature map X output by the Transformer branch out :

[0118] X = LocalAgg(Norm(X in )) + X in ;

[0119] Y = FFN(Norm(X)) + X;

[0120] Z = LocalProp(GlobalSparseAttn(Norm(Y))) + Y;

[0121] X out = FFN(Norm(Z)) + Z;

[0122] Among them, FFN represents a two-layer perceptron, and Norm represents a layer normalization operation.

[0123] Thus, the multi-branch module uses the convolutional branch to extract local features in the image data and the Transformer branch to capture global features in the image data.

[0124] Subsequently, the fusion module of the multi-branch convolutional attention network model is used to replace the simple concatenation of multiple feature channels in the channel dimension for integrating different features of the convolutional branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation.

[0125] The specific process of the fusion module for integrating different features of the convolutional branch and the Transformer branch is as follows:

[0126] First, the high-level features are upsampled to the same scale as the low-level features, and then the two are filtered and feature-extracted using two convolutions with kernel sizes of 1×3 and 3×1 respectively; then, batch normalization and the activation function GELU are applied for further processing; subsequently, the information of the two branches is concatenated, and the concatenated features are screened and filtered again through two convolutions with kernel sizes of 1×3 and 3×1 to establish a complementary relationship and obtain the extracted multi-level fusion feature X F ; at the decoding end of the multi-branch convolutional attention network, the class information of the high-level features can guide the classification of the low-level features, and the position information of the low-level features supplements the spatial information of the high-level features; the calculation process of the fusion module is as follows:

[0127] X1 = G(BN(DOConv 3×1 (DOConv 1×3 (X in1 ))));

[0128] X2 = G(BN(DOConv 3×1 (DOConv 1×3 (Up(X in2 ))));

[0129] W = Concat(X1, X2);

[0130] Y out = G(BN(DOConv 3×1 (DOConv 1×3 (W))));

[0131] Among them, X in1 and X in2 respectively represent the two inputs of the fusion module, Y out represents the output of the fusion module, DOConv 3×1 represents DO-Conv with a kernel size of n×m, Up represents the bilinear interpolation 2-fold upsampling operation, Concat represents the concatenation operation based on the channel dimension, and BN and G respectively represent batch normalization and the non-linear activation function GELU. The calculation formula of GELU is as follows:

[0132]

[0133] Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, first, for the extracted multi-level fusion feature X F perform an upsampling operation to gradually increase the spatial resolution of the feature map X F to a higher level, obtaining the upsampled feature map X FU . Subsequently, for each layer, merge X FU with the X F of the current layer, and generate a new feature representation X FC through a concatenation operation. Then, input X FC and the feature X F+1 from a higher level into the fusion module for processing. In this process, not only the feature information from the convolutional branch and the Transformer branch is integrated, but also the upsampled feature map X FU is included. The decoder effectively combines the advantages of shallow and deep features, further enhancing the network's performance in capturing multi-scale features and detailed information.

[0134] Finally, the fused features are passed to a classifier, namely the classfilier module, for final prediction; the classifier compresses the number of channels of the feature map output by the highest-level DE decoder to the number of classes 3, which is the classification probability of each pixel point, and the formula is:

[0135] Y = (DOConv(Upsample(DOConv(DOConv(Y F (1))))));

[0136] (4) Training the multi-branch convolutional attention network model

[0137] Divide the input CSWV dataset into a training set and a test set, and use the multi-branch convolutional attention network model to perform image segmentation training on the image data and the corresponding cloud layer labels and snow layer labels in the training set; the following introduces the specific implementation process:

[0138] First, the cross-entropy loss function is adopted to compare the mask output by the multi-branch convolutional attention network model with the true label, and measure the accuracy of the multi-branch convolutional attention network for cloud, snow, and background segmentation. Cross-entropy measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross-entropy loss is smaller. The cross-entropy loss function is used to measure the difference between the class distribution predicted by the multi-branch convolutional attention network model and the true class distribution, and can well guide the optimization of the multi-branch convolutional attention network model. For the multi-classification problem to be solved by the present invention, the output of the multi-branch convolutional attention network model is a probability distribution p = [p1, p2, …, p c , where p i represents the predicted probability that the sample belongs to the i-th class, and the distribution of the true class is q = [q1, q2, …, q c , q i is the indicator variable corresponding to the true label class. The formula for calculating the cross-entropy loss is:

[0139]

[0140] Subsequently, during the image segmentation training process, three training callback mechanisms are added:

[0141] Use a callback function that saves the weights of the multi-branch convolutional attention network model at any time; the callback function is used to save the weights of the model during the training of the multi-branch convolutional attention network model, and dynamically save the weights of the model during the training according to the loss of the training set.

[0142] Use a learning rate scheduler (such as ReduceLROnPlateau) to optimize the convergence path of the multi-branch convolutional attention network model, make the training process more stable and improve the final performance; the learning rate scheduler automatically reduces the learning rate when the validation loss has not improved for a long time, avoiding the model falling into local minima or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path, thereby improving the performance.

[0143] Use an early stopping mechanism to prevent the multi-branch convolutional attention network model from overfitting, that is, when the validation loss has not improved within the specified number of epochs, automatically stop the image segmentation training. The training parameter settings are as follows:

[0144] Table 2 Model Training Parameter Table

[0145] Parameter Name Parameter Setting Description Learning Rate Optimizer Adam The learning rate optimizer selects the Adam optimizer Loss Function CrossEntropyLoss The loss function selects the cross-entropy loss Total Number of Training Epochs 1000 The total number of training epochs is set to 1000 Batch Size 3 The batch size is set to 3 Learning Rate 0.001 The initial learning rate is set to 0.001

[0146] Adjust the weights of the multi-branch convolutional attention network model according to the feedback of the loss function, and finally output an accurate cloud and snow classification result; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, where each pixel is classified as background, cloud or snow.

[0147] The concept of the present invention is reasonable, and it can solve the technical problem that in the cloud and snow segmentation task of the prior art, improper handling of complex boundaries and details will significantly reduce the accuracy of the model.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optical image segmentation method based on a multi-branch convolutional attention model, characterized in that It includes the following steps: (1) Define and instantiate the dataset; (2) Visualize the input data; (3) Define and instantiate the multi-branch convolutional attention network model; (4) Train the multi-branch convolutional attention network model.

2. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, wherein The specific process of step (1) is as follows: (1.01) Load the image data Open the image data in each CSWV dataset, and extract data of different bands from the image data; (1.02) Stack the image data Stack the data of different bands together to form image data of different channels; (1.03) Convert the arrangement mode of the image data Convert the arrangement mode of the image data from height, width, channel to channel, height, width; (1.04) Normalize the image data Normalize the pixel values of the image data, that is, divide each pixel value by 255.0, so as to scale the value range of the pixel values from the original 0 to 255 to the interval of 0 to 1, and obtain the input image data X; (1.05) Read the label data Load the tif files of cloud labels and snow labels corresponding to each image data, which respectively represent whether each pixel point of the image data belongs to cloud or snow; (1.06) Process the label data Convert the data of the read tif files of cloud labels and snow labels into binary mask matrices. First, the mask of the background area is generated by checking whether the cloud label and the snow label are both 0. The generated binary mask matrix marks the background area as 1 and the rest of the areas as 0; then, the mask of the cloud area is generated by checking whether the cloud label is 255, indicating that the cloud area is 1 and other areas are 0; the mask of the snow area is generated by checking whether the snow label is 255, marking the snow area as 1 and other areas as 0; the mask matrices generated by the background area, the cloud area and the snow area respectively represent the categories of background, cloud and snow, and are put into a three-channel array, and each channel represents one of the mask matrices, thus generating the label data; (1.07) Convert the label data type Convert the label data from integer type to floating point type by calling the astype method of the NumPy array; (1.08) Merge the masks Stack the binary mask matrices of background, cloud and snow into a three-channel matrix; (1.09) Convert to tensors Convert both the image data and the label data into PyTorch tensors to obtain the input label data Y; (1.10) Return the data Return the obtained input image data X and input label data Y as multiple pairs of input and target for training or inference of the multi-branch convolutional attention model.

3. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 2, wherein: Before loading the image data in step (1), the CSWV dataset is uniformly cropped into image data of 512×512×3.

4. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 2, wherein, The specific process of step (2) is as follows: Visualize the input image data X and the corresponding label data Y, separate the red, green, and blue bands and the label data from the Pytorch tensor, stack the red, green, and blue bands to generate an RGB image, convert the one-hot encoding of the label data Y into class indices and apply a custom color map to obtain a label image, draw the RGB image and the label image into subplots to visually display the input and target output of the multi-branch convolutional attention network model, help verify whether the process of steps (1.01)-(1.09) is correct, and provide visual support for the optimization of the multi-branch convolutional attention network model.

5. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, wherein , The specific process of step (3) is as follows: (3.1) The multi-branch module adopts a parallel structure of branches and serves as the backbone network of the multi-branch convolutional attention network model. Among them, the branches include convolutional branches, that is, two 3×3 convolutions with residual structures are introduced. After each convolution, there is also a batch normalization BN and a ReLU activation function s. Finally, through the residual structure, the feature map x output by the convolutional branch is obtained. c : C = σ(BN(Conv 3×3 (f i )); f i+1 = σ(BN(Conv 3×3 (C)); x C = Concat(f i , f i+1 ); Another branch: the Transformer branch, which is parallel to the convolutional branch. The Transformer branch adopts a lightweight Vision Transformer architecture: the Transformer block in Edge Vision Transformers, which is realized by introducing an optimized structure combining self-attention and convolution, namely an efficient "local-global-local" information interaction bottleneck; thus, the multi-branch module uses the convolutional branch to extract local features in the image data and the Transformer branch to grasp the global features in the image data. (3.2) Use the fusion module of the multi-branch convolutional attention network model to replace the simple splicing of multiple feature channels in the channel dimension for integrating different features of the convolutional branch and the Transformer branch; in the fusion module, DO-Conv is used to replace the traditional Conv convolution operation. The specific process of the fusion module integrating different features of the convolutional branch and the Transformer branch is as follows: First, the high-level features are upsampled to the same scale as the low-level features, and then the two are filtered and feature-extracted respectively using two convolutional kernels with sizes of 1×3 and 3×1; then, batch normalization and the activation function GELU are applied for further processing. Subsequently, the information of the two branches is concatenated, and the concatenated features are screened and filtered again through two convolutions with kernel sizes of 1×3 and 3×1 to establish a complementary relationship, and the extracted multi-level fusion feature X is obtained. F ; At the decoding end of the multi-branch convolutional attention network, the category information of the high-level features can guide the classification of the low-level features, while the position information of the low-level features complements the spatial information of the high-level features. The calculation process of the fusion module is as follows: X1 = G(BN(DOConv 3×1 (DOConv 1×3 (X in1 )))); X2 = G(BN(DOConv 3×1 (DOConv 1×3 (Up(X in2 ))))); W = Concat(X1, X2); Y out = G(BN(DOConv 3×1 (DOConv 1×3 (W)))); Among them, X in1 and X in2 respectively represent the two inputs of the fusion module, Y out represents the output of the fusion module; DOConv 3×1 represents DO-Conv with a convolution kernel size of n×m, Up represents the bilinear interpolation 2x upsampling operation, Concat represents the concatenation operation based on the channel dimension, BN and G respectively represent batch normalization and the non-linear activation function GELU; the calculation formula of GELU is as follows: Next, in the implementation of the DE decoder of the multi-branch convolutional attention network model, first, the extracted multi-level fusion feature X F is subjected to an upsampling operation to gradually increase the spatial resolution of the feature map X F to a higher level, obtaining the upsampled feature map X FU ; subsequently, each layer of X FU is merged with the X F of the current layer, and a new feature representation X FC is generated through a concatenation operation; then, X FC and the feature X F+1 from a higher level are jointly input into the fusion module for processing; Finally, the fused features are passed to a classifier, that is, the classfilier module of the multi-branch convolutional attention network model, for final prediction; the classifier compresses the number of channels of the feature map output by the highest-level DE decoder to the number of classes 3, which is the classification probability of each pixel point, and the formula is: Y = (DOConv(Upsample(DOConv(DOConv(Y F (1))))))。 6. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 5, characterized in that, The specific process of the implementation of the "local-global-local" information interaction bottleneck in step (3.1) is as follows: (3.1.1) Local aggregation Use effective depth convolution to aggregate local information from adjacent features. (3.1.2) Global sparse attention Form a sparse and uniformly distributed representative feature set through self-attention for remote information exchange, thereby obtaining updated information. (3.1.3) Local propagation Diffuse the updated information from the representative features to the non-representative features in the local neighborhood through transposed convolution. (3.1.4)Finally obtain the feature map X output by the Transformer branch out : X = LocalAgg(Norm(X in )) + X in ; Y = FFN(Norm(X)) + X; Z = LocalProp(GlobalSparseAttn(Norm(Y))) + Y; X out = FFN(Norm(Z)) + Z; Among them, FFN represents a two-layer perceptron, and Norm represents a layer normalization operation.

7. The optical image segmentation method based on a multi-branch convolutional attention model according to claim 1, wherein, Step (4) is to divide the input CSWV dataset into a training set and a test set, and use the multi-branch convolutional attention network model to perform image segmentation training on the image data and the corresponding cloud layer labels and snow layer labels in the training set; the specific implementation process is as follows: (4.1) First, use the cross-entropy loss function to compare the mask output by the multi-branch convolutional attention network model with the true label to measure the accuracy of the multi-branch convolutional attention network in cloud, snow, and background segmentation; the cross-entropy measures the distance between the predicted probability distribution and the true distribution. When the prediction is closer to the true distribution, the cross-entropy loss is smaller; the cross-entropy loss function is used to measure the difference between the class distribution predicted by the multi-branch convolutional attention network model and the true class distribution. The output of the multi-branch convolutional attention network model is a probability distribution: p = [p1, p2, …, p c ; where p i represents the predicted probability that the sample belongs to the i-th class, and the distribution of the true classes is q = [q1, q2, …, q c , q i is an indicator variable for the class corresponding to the true label; The calculation formula for the cross-entropy loss is: (4.2) Define a training callback mechanism Use a callback function to save the weights of the multi-branch convolutional attention network model at any time; use a learning rate scheduler to optimize the convergence path of the multi-branch convolutional attention network model; use an early stopping mechanism to prevent the multi-branch convolutional attention network model from overfitting, that is, when the validation loss does not improve within a specified number of epochs, automatically stop the image segmentation training. (4.3) According to the feedback of the loss function, adjust the weights of the multi-branch convolutional attention network model, and finally output accurate cloud and snow classification results; the output of the multi-branch convolutional attention network model is a pixel-level segmentation image, where each pixel is classified as background, cloud, or snow.

8. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, characterized in that: The callback function in step (4.2) is used to save the weights of the model during the training of the multi-branch convolutional attention network model, and dynamically save the weights of the multi-branch convolutional attention network model during the training process according to the loss of the training set.

9. The optical image segmentation method based on the multi-branch convolutional attention model according to claim 1, characterized in that: The learning rate scheduler in step (4.2) automatically reduces the learning rate when the validation loss has not improved for a long time, to avoid the multi-branch convolutional attention network model falling into a local minimum or training stagnation. By gradually reducing the learning rate, the model finds a better convergence path to improve performance.

Images