Remote sensing image semantic segmentation method and system based on prototype learning

Abstract

The invention relates to a remote sensing image semantic segmentation method and system based on prototype learning, which dynamically optimizes feature representation through prototype learning, captures multi-scale context information in combination with a mask Transform, and significantly improves segmentation precision while keeping model efficiency, especially for a segmentation effect of small targets and complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and remote sensing image processing technology, specifically to a method and system for semantic segmentation of remote sensing images based on prototype learning. Background Technology

[0002] Semantic segmentation of remote sensing images is one of the core tasks in computer vision and remote sensing, aiming to classify each pixel in a remote sensing image and identify land cover categories such as buildings, roads, vegetation, and water bodies. This technology has significant application value in land use monitoring, urban planning, environmental assessment, and disaster management. With the development of deep learning, methods based on convolutional neural networks (CNNs) and Transformers have made significant progress in this field, but many technical challenges remain.

[0003] Traditional methods primarily rely on manually designed features (such as texture and spectral features) combined with machine learning classifiers (such as support vector machines and random forests). However, these methods have limited feature representation capabilities and struggle to handle the complex scenes in remote sensing images. The introduction of fully convolutional networks (FCNs) first achieved end-to-end pixel-level prediction. Subsequently, U-Net improved feature transfer efficiency through its encoding / decoding structure and skip connections, while the DeepLab series enhanced multi-scale feature extraction capabilities through dilated convolutions and spatial pyramid pooling. However, these CNN-based methods are limited by their local receptive field, making it difficult to model long-range dependencies and resulting in poor performance in complex scenes.

[0004] The most relevant existing technology is the remote sensing image segmentation method using the Transformer architecture. For example... Figure 1 As shown, these methods typically include an encoder-decoder structure. The encoder first segments the input remote sensing image into image patches and converts them into embedding vectors. Then, it extracts features through multiple layers of Transformer blocks, each containing a self-attention mechanism to model global contextual relationships. The decoder then gradually restores the spatial resolution through upsampling and skip connections, ultimately outputting pixel-level segmentation results. Typical implementations, such as SETR, use a pure Transformer encoder combined with a CNN decoder, while Swin-UNet uses a hierarchical window attention mechanism to reduce computational complexity. Although these methods can capture long-range dependencies, they still suffer from problems such as insufficiently compact feature representation and low computational efficiency when dealing with multi-scale targets and complex backgrounds. In particular, when dealing with high-resolution remote sensing images, their memory consumption increases quadratically, severely limiting their practical application effectiveness.

[0005] Multi-scale feature fusion is a commonly used strategy that combines features from different scales to handle diverse target scales in remote sensing images. For example, a deep learning method based on dilated convolution can extract multi-scale features while maintaining resolution, improving the segmentation results for both large-scale and small-scale targets. Contextual information modeling is also a key method to improve segmentation accuracy, enhancing classification precision by considering the contextual information surrounding pixels. Existing techniques have proposed a self-attention mechanism-based method that learns long-range dependencies between pixels, mitigating the impact of background noise on the segmentation results.

[0006] However, despite these methods' progress in global feature learning, they still fall short in improving local details, inter-class differentiation, and intra-class compactness. Traditional pixel-based metric learning methods, while performing well in inter-class separation, face challenges in intra-class compactness, target detail modeling, and small target segmentation. Therefore, how to improve intra-class pixel clustering (compactness) while ensuring clear inter-class separation through effective prototype learning strategies remains an important research direction in semantic segmentation of remote sensing images. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a remote sensing image semantic segmentation method and system based on prototype learning, which solves the problems of insufficient segmentation accuracy, poor adaptability to multi-scale targets, and loose intra-class features in existing remote sensing image segmentation technologies due to complex background interference.

[0008] The technical solution adopted in this invention is:

[0009] A semantic segmentation method for remote sensing images based on prototype learning includes the following steps:

[0010] A dual-branch network is constructed, including a multi-scale feature extraction branch, which uses a Swing Transformer encoder to extract multi-level features from the input remote sensing image, and uses a Unetformer decoder to fuse high and low-level features and output a sequence of feature maps with increasing resolution; and a contextual prototype optimization branch, which uses a multi-scale low-rank adaptor encoder to generate basic features and combines a prototype learning module to dynamically optimize prototype vectors for each category.

[0011] The prototype vector is optimized through a masked cross-attention module: the feature map output from the decoder of the multi-scale feature extraction branch is interacted with the prototype vector of the context prototype optimization branch to generate a coarse segmentation mask; the coarse segmentation mask is used to constrain the cross-attention weights, focusing the foreground region and suppressing background interference; the prototype vector is updated through self-attention and feedforward networks.

[0012] Calculate the joint loss function, which includes the segmentation loss, calculating the cross-entropy loss and Dice loss for the final prediction mask and the intermediate coarse segmentation mask; and the prototype optimization loss, which optimizes the inter-class separability and intra-class compactness of the prototype vectors through the prototype separability optimization loss and the prototype compactness contrast loss.

[0013] Output the semantic segmentation results of the remote sensing image.

[0014] Furthermore, the encoder of the multi-scale feature extraction branch uses the Swing Transformer to extract features hierarchically, outputting multi-level feature maps. The input image is X, and the output of each encoder layer is represented as:

[0015]

[0016] Where SwTrans(·) represents the mapping function of the Swing Transformer. This represents the output of the first layer encoder. This represents the output of the i-th layer encoder;

[0017] The decoder output is represented as:

[0018]

[0019] Where UFormer(·) represents the mapping function of a single-layer decoder in Unetformer, and WSum(·) represents the weighted fusion function. F represents the output of the innermost encoder. N This represents the output of the innermost decoder, corresponding to the innermost encoder. F represents the output of the i-th layer encoder. i This represents the output of the decoder corresponding to the i-th layer encoder.

[0020] Furthermore, to better narrow the semantic gap between features, refine semantic features, and improve the segmentation accuracy of the model, a feature refinement head is used to refine the F1 score of the first-layer decoder output. To refine the features, that is:

[0021]

[0022] Where FRH(·) represents the mapping function of FRH, and WSum(·) represents the weighted fusion function.

[0023] Furthermore, in the context prototype optimization branch, assume there are C categories, each category has k prototype vectors, and each prototype vector has c channels, for a total of C×k prototype vectors. The context prototype optimization branch randomly initializes the prototype vector for each category, i.e.:

[0024]

[0025] in, Let k represent the prototype vectors of the i-th category in P0, where i∈[1,…,C];

[0026] Prototype vectors and the output F of the multi-scale low-rank adaptor encoder s Information exchange through cross-attention:

[0027] Using the weight matrix W q P0 is mapped to the query space using the weight matrix W. k and W v The features F extracted from the basic model s The features are transformed into key and value features, thus obtaining feature representation E1;

[0028] Layer normalization is performed on E1 to alleviate the gradient vanishing problem during training;

[0029] Self-attention is used to further enhance the expressive power of the prototype vector E1, and the attention result is added to the prototype vector E1 through skip connections, so that the model can update the prototype vector while retaining the original information.

[0030] The updated prototype vector is then passed through a feedforward network to enhance its nonlinear feature representation, resulting in the final prototype vector P1, which is rich in semantic information.

[0031]

[0032] P1=FFN(Att(LN(E1))+E1)+E1 (6)

[0033] Where × denotes matrix multiplication, LN(·) represents the softmax activation function, LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network.

[0034] Furthermore, in the masked cross-attention module, let the input of the t-th MCA module be the prototype vector P. t The feature map F output by the decoder of the corresponding MFEB layer. N-t ,t∈[1,2,3] andF0;

[0035] Generate coarse segmentation mask M t ,Right now:

[0036] M t =σ(P t ×F0) (7)

[0037] Where σ(·) represents the sigmoid activation function;

[0038] P t With F N-t Perform cross-attention calculation, mask M t The calculation of attention weights is constrained and introduced into the cross-attention mechanism, namely:

[0039]

[0040] Where × represents matrix multiplication, W q W k and W v The weight matrix representing attention. H(M) represents the softmax activation function. t ) is based on the mask M t The function that implements matrix masking is defined as follows:

[0041]

[0042] The masked cross-attention module will update the cross-attention result E. t+1 After further optimization using a feedforward neural network and self-attention, the final updated prototype vector P is obtained. t+1 ,Right now:

[0043] P t+1 =FFN(Att(LN(E t+1 ))+E t+1 )+E t+1 (10)

[0044] Where LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network.

[0045] Furthermore, the cross-entropy loss and Dice loss L are used. dice They are represented as follows:

[0046]

[0047] Where N and K represent the number of samples and the number of categories, respectively, and M... (n) One-hot encoding representing the true semantic tag. This represents the final prediction mask of the network output and the coarse segmentation mask of the intermediate process, n∈[1,…,N], It is the confidence level that sample n belongs to class k.

[0048] The total loss is expressed as follows:

[0049]

[0050] Furthermore, in the prototype optimization loss,

[0051] Prototype separability optimization loss L PDO The calculation is as follows:

[0052]

[0053] in, It is pixel i n The assigned positive prototype, the goal of the loss function is to make the pixel embedding i n With the prototype Minimize the distance, p - It is the negative prototype set, defined as That is, all other prototypes that do not belong to the normal prototype, where τ is the temperature coefficient;

[0054] Prototype compactness contrast loss L PCC The calculation is as follows:

[0055]

[0056] Among them, ||i n ||2 and It is the L2 norm of pixel embedding and prototype;

[0057] Final loss function It can be represented as:

[0058]

[0059] A remote sensing image semantic segmentation system based on prototype learning, characterized in that it includes:

[0060] The multi-scale feature extraction module is configured to extract and fuse multi-scale features using a Swing Transformer encoder and a Unetformer decoder.

[0061] The prototype optimization module, including a multi-scale low-rank adaptor encoder and a cascaded mask cross-attention module, is used to dynamically update the class prototype vector.

[0062] The loss calculation module is used to jointly optimize the segmentation loss and the prototype optimization loss; and

[0063] The segmentation output module is used to generate pixel-level semantic segmentation masks.

[0064] The positive effects of this invention are:

[0065] This invention addresses the issue of insufficient segmentation accuracy for small targets in traditional methods through an innovative multi-scale prototype dynamic optimization mechanism. A unique mask-based cross-attention architecture effectively suppresses background interference. Attention masks are dynamically generated from the coarse segmentation results, enabling the model to focus on key regions. Furthermore, a dual-branch design achieves an optimal balance between computational efficiency and segmentation accuracy. Attached Figure Description

[0066] Figure 1 This refers to the encoding and decoding framework in existing technologies;

[0067] Figure 2 This is the network structure of the present invention;

[0068] Figure 3 This is a network structure diagram of the masked cross-attention method of the present invention. Detailed Implementation

[0069] As attached Figure 2 , 3 As shown, this invention discloses a remote sensing image semantic segmentation method based on prototype learning, comprising the following steps:

[0070] A dual-branch network is constructed, including a multi-scale feature extraction branch, which uses a Swing Transformer encoder to extract multi-level features from the input remote sensing image, and uses a Unetformer decoder to fuse high and low-level features and output a sequence of feature maps with increasing resolution; and a contextual prototype optimization branch, which uses a multi-scale low-rank adaptor encoder to generate basic features and combines a prototype learning module to dynamically optimize prototype vectors for each category.

[0071] The prototype vector is optimized through a masked cross-attention module: the feature map output from the decoder of the multi-scale feature extraction branch is interacted with the prototype vector of the context prototype optimization branch to generate a coarse segmentation mask; the coarse segmentation mask is used to constrain the cross-attention weights, focusing the foreground region and suppressing background interference; the prototype vector is updated through self-attention and feedforward networks.

[0072] Calculate the joint loss function, which includes the segmentation loss, calculating the cross-entropy loss and Dice loss for the final prediction mask and the intermediate coarse segmentation mask; and the prototype optimization loss, which optimizes the inter-class separability and intra-class compactness of the prototype vectors through the prototype separability optimization loss and the prototype compactness contrast loss.

[0073] Output the semantic segmentation results of the remote sensing image.

[0074] The steps of the present invention are described in detail below, and its network structure is as follows: Figure 2As shown, it comprises two branches: the Multi-scale Feature Extraction Branch (MFEB) and the Contextual Prototype Refinement Branch (CPRB). The MFEB is built based on the Swin Transformer and U-Net architecture (Swin-UNet) and is primarily responsible for image feature extraction and semantic segmentation. The CPRB consists of a Multi-scale Low-Rank Adapted Encoder (MLRAE) and prototype learning. It combines prototype vector generation and update mechanisms to further optimize feature representations to improve segmentation accuracy.

[0075] The multi-scale feature extraction branch is an improved U-Net architecture consisting of a Swin Transformer as the encoder and a Unetformer as the decoder. The Swin Transformer utilizes its hierarchical and windowed self-attention mechanism to effectively extract multi-scale features from remote sensing images while handling long-range dependencies. The decoder part uses the Unetformer decoder, which adds a Transformer module to the traditional U-Net structure to enhance the decoder's global information processing capabilities. It also achieves the fusion of low-level features and high-level semantic information through skip connections, ensuring the fineness and accuracy of the segmentation results. Let the input image be X, then the output of each encoder layer can be expressed as:

[0076]

[0077] Where SwTrans(·) represents the mapping function of the Swing Transformer. This represents the output of the first layer encoder. This represents the output of the i-th layer encoder. Subsequently, the output of the decoder can be represented as:

[0078]

[0079] Where UFormer(·) represents the mapping function of a single-layer decoder in Unetformer, and WSum(·) represents the weighted fusion function. F represents the output of the innermost encoder. N This represents the output of the innermost decoder, corresponding to the innermost encoder. F represents the output of the i-th layer encoder. i This represents the output of the decoder corresponding to the encoder at layer i.

[0080] Finally, to better narrow the semantic gap between features, refine semantic features, and improve the segmentation accuracy of the model, a Feature Refinement Head (FRH) was used to refine the F1 and F2 scores of the first-layer decoder output. To refine the features, that is:

[0081]

[0082] Where FRH(·) represents the mapping function of FRH, and WSum(·) represents the weighted fusion function.

[0083] In the contextual prototype optimization branch, prototype learning plays a crucial role. The goal of prototype learning is to represent each category as one or more prototype vectors, enabling the prototypes to represent the core features of the category, and continuously optimizing these prototype vectors through model training. Through cross-attention and self-attention mechanisms, the model learns typical representations of each category, making prototype vectors within the same category more similar and increasing the distance between prototype vectors from different categories, thereby enhancing the discriminative power between categories. Under the cross-attention mechanism, the prototype vectors generated by CPRB interact with the output of the decoder in MFEB (hereinafter referred to as the decoder), further updating the representational power of the prototype vectors. The self-attention mechanism helps improve the discriminative power between categories, ensuring that the features of each category are accurately represented in the prototype vectors. Furthermore, this invention employs an online clustering algorithm based on fast Sinkhorn to obtain the pixel-prototype assignment matrix and introduces PDO loss and PCC loss to shorten the distance between prototype vectors of the same category and widen the distance between prototype vectors of different categories, thereby optimizing the category representation.

[0084] Assume there are C categories, each category has k prototype vectors, and each prototype vector has c channels, for a total of C×k prototype vectors. CPRB first randomly initializes the prototype vector for each category, i.e.:

[0085]

[0086] in, Let represent the k prototype vectors of the i-th category in P0, where i∈[1,…,C]. The prototype vectors are related to the MLRAE output F. s Information interaction is achieved through cross-attention to supplement the semantic information in the initialized prototype vector. Specifically, this invention uses a weight matrix W. q P0 is mapped to the query space using the weight matrix W. k and W v The features F extracted from the basic models The prototype vector E1 is transformed into key-value features. Next, layer normalization is applied to E1 to alleviate the vanishing gradient problem during training. Then, self-attention is used to further enhance the expressive power of the prototype vector E1, and the attention result is added to the prototype vector E1 via skip connections, allowing the model to update the prototype vector while preserving the original information. Finally, the updated prototype vector is passed through a feedforward network to enhance its non-linear feature expression, resulting in the final semantically rich prototype vector P1. That is:

[0087]

[0088] P1=FFN(Att(LN(E1))+E1)+E1 (6)

[0089] Where × denotes matrix multiplication, LN(·) represents the softmax activation function, LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network.

[0090] To better integrate the multi-scale information extracted by MFEB, this invention constructs a Masked Cross Attention (MCA) module. Multiple cascaded MCA modules achieve a coarse-to-fine integration of multi-scale information into the prototype vector, thereby enabling continuous updating of the prototype vector and further optimizing the model's feature representation to improve segmentation accuracy. The MCA module is described below. The Masked Cross Attention module includes masked attention, layer normalization, self-attention, and MLP, and its network structure is as follows: Figure 3 As shown, the MCA module interacts with the prototype vectors and feature maps F at different levels of the MFEB decoder, while introducing masking to progressively optimize the prototypes for each category from coarse to fine, making them better suited to complex semantic segmentation tasks. Let the input of the t-th MCA module be the prototype vector P. t The feature map F output by the decoder of the corresponding MFEB layer. N-t ,t∈[1,2,3] and F0(F N F4 is the output of the innermost decoder, and F4 is the output of the outermost decoder. Therefore, the MCA module mainly uses a masked cross-attention mechanism to process P. t and F N-t Interact to obtain the updated prototype vector P t+1 The MCA module introduces a coarse segmentation mask M. t It guides the update of prototype vectors, directing them towards more accurate category features, thereby optimizing segmentation accuracy.

[0091] Specifically, the MCA module first calculates P t Interacting with F0 generates a coarse segmentation mask M. t ,Right now:

[0092] M t =σ(P t ×F0) (7)

[0093] Where σ(·) represents the sigmoid activation function. Through P t Interacting with F0, the MCA module obtains M t Masking can effectively guide the network to focus on the foreground area while reducing interference from the background area.

[0094] Subsequently, the MCA module will P t With F N-t Cross-attention calculation is performed. Simultaneously, to further enhance the model's focus on the foreground region, a mask M is used. t The calculation of attention weights, introduced into the cross-attention mechanism, is constrained to focus only on the foreground region and ignore the background, i.e.:

[0095]

[0096] Where × represents matrix multiplication, W q W k and W v The weight matrix representing attention. This represents the softmax activation function. h(M) t ) is based on the mask M t The function that implements matrix masking is defined as follows:

[0097]

[0098] Finally, the MCA module updates the cross-attention result E. t+1 After further optimization using a feedforward neural network and self-attention, the final updated prototype vector P is obtained. t+1 ,Right now:

[0099] P t+1 =FFN(Att(LN(E t+1 ))+E t+1 )+E t+1 (10)

[0100] Where LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network. Through this process, the prototype vector P t+1It has received more refined updates, enabling it to more accurately represent the semantic features of each category.

[0101] This invention designs multiple loss functions to guide the training process, mainly including supervision of the final prediction result, coarse segmentation mask, and prototype vector. For the final prediction result and coarse segmentation mask, cross-entropy loss (CE Loss) and Dice loss (Dice Loss) are used. For the prototype vector, prototype disentanglement optimization loss (Prototype Disentanglement Optimization Loss L) is used. PDO ) and Prototype Compactness Contrastive Loss PCC Specifically:

[0102] Besides the final prediction result, the loss calculated from the coarse segmentation mask in the intermediate process can provide intermediate supervision information, better guiding network training. The cross-entropy loss L is used. CE And Dice lost L dice These can be expressed as Equation 11 and Equation 12, respectively, and the total loss is shown in Equation 13.

[0103]

[0104] Where N and K represent the number of samples and the number of categories, respectively. M (n) One-hot encoding representing the true semantic tag. This represents the final prediction mask of the network output and the coarse segmentation mask of the intermediate process, n∈[1,…,N], It is the confidence level that sample n belongs to class k.

[0105] This invention proposes a dynamic prototype allocation mechanism that achieves optimized pixel-prototype matching through online Sinkhorn clustering and joint index encoding. By comparing and learning the similarity between pixels and prototypes, this invention ensures that the features of each pixel are similar to its assigned prototype and can be distinguished from other prototypes, thereby enhancing the accuracy of semantic segmentation. This process can be achieved by optimizing the prototype and the prototype separability optimization loss (L... PDO To achieve this.

[0106] For each category c j There is a set of training pixels Each pixel It is the feature embedding of that pixel. c K prototypes assigned to category c Pixel allocation is achieved through a probability matrix L c ∈{0,1} K×NLet i represent the value of each i. i,n =[i i,n,1 i i,n,2 ,…,i i,n,K ] is pixel i n Assigned to prototype p c,k The probability of L. To ensure consistency in allocation and avoid over-concentration of prototypes, this invention first employs an online clustering algorithm based on fast Sinkhorn to cluster L. c Optimize:

[0107] indexes = Sinkhorn(L c (17)

[0108] Where, indexes is the hard-assigned result, which represents each pixel i n Final assigned prototype According to indexes, if pixel i n Assigned to category c j The m-th prototype, i.e., indexes j (i n = m, where each category corresponds to one index. j , then i n A composite index can be represented as:

[0109] proto_target[i] = j × K + m (18)

[0110] This means that each pixel i n Based on the maximum value of proto_target[i] among the K prototypes corresponding to the pixel, assign it to the corresponding category c. i The m-th prototype is given below. To optimize the assignment between pixels and prototypes, the training objective of this invention is to maximize the posterior probability of pixel assignment, ensuring that each pixel embedding is as similar as possible to its assigned prototype and maintains sufficient distance from other unrelated prototypes. This can be achieved by introducing a contrastive loss. Specifically, this invention uses the following prototype separability optimization loss (L... PDO To optimize the model:

[0111]

[0112] in It is pixel i n The assigned positive prototype, the goal of the loss function is to make the pixel embedding i n With the prototype Minimize the distance. - It is the negative prototype set, defined as That is, all other prototypes that do not belong to the positive prototype are used to increase the distance between pixel embeddings and irrelevant prototypes, thereby enhancing intra-class compactness and inter-class separability. τ is a temperature coefficient that controls the concentration of similarity distribution. In the optimization process, it is necessary not only to increase the separability between prototypes of different categories, but also to minimize the differences of pixels of the same type within each category of prototypes, so that pixels belonging to the same prototype are as close as possible. To this end, this invention introduces the prototype compactness contrast loss (L... PCC ) is defined as:

[0113]

[0114] ||i n ||2 and This is the L2 norm of the pixel embedding and prototype, used to prevent overfitting and over-reliance on a few samples. Similar to the segmentation loss, the L2 norm of the pixel embedding and prototype is calculated for each stage. PDO and L PCC Loss, final loss function It can be represented as:

[0115]

[0116] This invention employs a dual-branch collaborative architecture, integrating the Swin-Unet multi-scale feature extraction branch and the contextual prototype optimization branch, achieving feature complementarity through cross-branch attention. The mask cross-attention architecture utilizes coarse segmentation results to generate dynamic attention masks, employing a progressive feature refinement strategy from coarse to fine to effectively achieve foreground focusing and background suppression. The prototype-guided decoder design injects the optimized prototype vector as category prior knowledge into each stage of the decoder, combining online clustering algorithms and the PDO / PCC loss function to achieve adaptive updating of the prototype vector, realizing end-to-end optimization of pixel-prototype similarity.

[0117] Compared to the existing Unetformer technology, this invention addresses the problem of insufficient segmentation accuracy for small targets in traditional methods through an innovative multi-scale prototype dynamic optimization mechanism. Experiments show that on the Vaihingen dataset, the prototype-based remote sensing image semantic segmentation method proposed in this invention significantly improves both mF1 and mIoU metrics compared to the comparison methods. Specifically, for FTUNetformer, the method improves mF1 by 1.27% and mIoU by 2.06%. Compared to the best comparison algorithm FTNetformer+SAM, mF1 and mIoU are improved by 0.43% and 0.57%, respectively. The unique mask cross-attention architecture effectively suppresses background interference. This architecture dynamically generates attention masks from the coarse segmentation results, enabling the model to focus on key regions, while the dual-branch design achieves an optimal balance between computational efficiency and segmentation accuracy.

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on prototype learning, characterized in that... It includes the following steps: A dual-branch network is constructed, including a multi-scale feature extraction branch, which uses a Swing Transformer encoder to extract multi-level features from the input remote sensing image, and uses a Unetformer decoder to fuse high and low-level features and output a sequence of feature maps with increasing resolution; and a contextual prototype optimization branch, which uses a multi-scale low-rank adaptor encoder to generate basic features and combines a prototype learning module to dynamically optimize prototype vectors for each category. The prototype vector is optimized through a masked cross-attention module: the feature map output by the decoder of the multi-scale feature extraction branch is interacted with the prototype vector of the contextual prototype optimization branch to generate a coarse segmentation mask; the coarse segmentation mask is used to constrain the cross-attention weights, focusing the foreground region and suppressing background interference. The prototype vector is updated using self-attention and a feedforward network. Calculate the joint loss function, which includes the segmentation loss, calculating the cross-entropy loss and Dice loss for the final prediction mask and the intermediate coarse segmentation mask; and the prototype optimization loss, which optimizes the inter-class separability and intra-class compactness of the prototype vectors through the prototype separability optimization loss and the prototype compactness contrast loss. Output the semantic segmentation results of the remote sensing image.

2. The remote sensing image semantic segmentation method based on prototype learning according to claim 1, characterized in that... The encoder of the multi-scale feature extraction branch uses the Swing Transformer to extract features hierarchically, outputting multi-level feature maps. The input image is X, and the output of each encoder layer is represented as follows: Where SwTrans(·) represents the mapping function of the Swing Transformer. This represents the output of the first layer encoder. This represents the output of the i-th layer encoder; The decoder output is represented as: Where UFormer(·) represents the mapping function of a single-layer decoder in Unetformer, and WSum(·) represents the weighted fusion function. F represents the output of the innermost encoder. N This represents the output of the innermost decoder, corresponding to the innermost encoder. F represents the output of the i-th layer encoder. i This represents the output of the decoder corresponding to the i-th layer encoder.

3. The remote sensing image semantic segmentation method based on prototype learning according to claim 2, characterized in that... To better narrow the semantic gap between features, refine semantic features, and improve the segmentation accuracy of the model, a feature refinement head is used to refine the F1 score of the first-layer decoder output. To refine the features, that is: Where FRH(·) represents the mapping function of FRH, and WSum(·) represents the weighted fusion function.

4. The remote sensing image semantic segmentation method based on prototype learning according to claim 1, characterized in that... In the contextual prototype optimization branch, assume there are C categories, each category has k prototype vectors, and each prototype vector has c channels, for a total of C×k prototype vectors. The context prototype optimization branch randomly initializes the prototype vector for each category, i.e.: in, Let k represent the prototype vectors of the i-th category in P0, where i∈[1,…,C]; Prototype vectors and the output F of the multi-scale low-rank adaptor encoder s Information exchange through cross-attention: Using the weight matrix W q P0 is mapped to the query space using the weight matrix W. k and W v The features F extracted from the basic model s The features are transformed into key and value features, thus obtaining feature representation E1; Layer normalization is performed on E1 to alleviate the gradient vanishing problem during training; Self-attention is used to further enhance the expressive power of the prototype vector E1, and the attention result is added to the prototype vector E1 through skip connections, so that the model can update the prototype vector while retaining the original information. The updated prototype vector is then passed through a feedforward network to enhance its nonlinear feature representation, resulting in the final prototype vector P1, which is rich in semantic information. P1=FFN(Att(LN(E1))+E1)+E1 (6) Where × denotes matrix multiplication, LN(·) represents the softmax activation function, LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network.

5. The remote sensing image semantic segmentation method based on prototype learning according to claim 1, characterized in that the mask... In the cross-attention module, let the input of the t-th MCA module be the prototype vector P. t The feature map F output by the decoder of the corresponding MFEB layer. N-t ,t∈[1,2,3] andF0; Generate coarse segmentation mask M t ,Right now: M t =σ(P t ×F0) (7) Where σ(·) represents the sigmoid activation function; P t With F N-t Perform cross-attention calculation, mask M t The calculation of attention weights is constrained and introduced into the cross-attention mechanism, namely: Where × represents matrix multiplication, W q W k and W v The weight matrix representing attention. H(M) represents the softmax activation function. t ) is based on the mask M t The function that implements matrix masking is defined as follows: The masked cross-attention module will update the cross-attention result E. t+1 After further optimization using a feedforward neural network and self-attention, the final updated prototype vector P is obtained. t+1 ,Right now: P t+1 =FFN(At(LN(E t+1 ))+E t+1 )+E t+1 (10) Where LN(·) represents the layer normalization operation, Att(·) represents the self-attention mapping function, and FFN(·) represents the mapping function corresponding to the feedforward neural network.

6. The remote sensing image semantic segmentation method based on prototype learning according to claim 1, characterized in that... The cross-entropy loss L used CE And Dice lost L dice They are represented as follows: Where N and K represent the number of samples and the number of categories, respectively, and M... (n) One-hot encoding representing the true semantic tag. This represents the final prediction mask of the network output and the coarse segmentation mask of the intermediate process, n∈[1,…,N], It is the confidence level that sample n belongs to class k. The total loss is expressed as follows:

7. The remote sensing image semantic segmentation method based on prototype learning according to claim 6, characterized in that... In the prototype optimization loss, Prototype separability optimization loss L PDO The calculation is as follows: in, It is pixel i n The assigned positive prototype, the goal of the loss function is to make the pixel embedding i n With the prototype Minimize the distance, p - It is the negative prototype set, defined as That is, all other prototypes that do not belong to the normal prototype, where τ is the temperature coefficient; Prototype compactness contrast loss L PCC The calculation is as follows: Where, ||i n ||2 and It is the L2 norm of pixel embedding and prototype; Final loss function It can be represented as:

8. A remote sensing image semantic segmentation system based on prototype learning, characterized in that... It includes The multi-scale feature extraction module is configured to extract and fuse multi-scale features using a Swing Transformer encoder and a Unetformer decoder. The prototype optimization module, including a multi-scale low-rank adaptor encoder and a cascaded mask cross-attention module, is used to dynamically update the class prototype vector. The loss calculation module is used to jointly optimize the segmentation loss and the prototype optimization loss; and The segmentation output module is used to generate pixel-level semantic segmentation masks.

Images