Remote sensing semantic segmentation method for label guiding prototype alignment
By constructing a local and global dual-path alignment mechanism and utilizing the high-confidence pixel features of labeled samples to build category prototypes, the problem of feature alignment of unlabeled samples in semi-supervised semantic segmentation of remote sensing images is solved, achieving higher segmentation accuracy and training stability.
Patent Information
- Application Number
- CN202511761321.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-01-09
AI Technical Summary
Existing semi-supervised semantic segmentation methods for remote sensing images cannot adequately align the features of unlabeled samples to the semantic space of labeled samples, resulting in unstable model training and blurred category boundaries.
By constructing a local and global dual-path alignment mechanism, a category prototype is built using the high-confidence pixel features of labeled samples, establishing semantic connections between labeled and unlabeled samples, optimizing the feature representation of unlabeled samples, and achieving explicit alignment at the feature level using a shared encoder-decoder network, a prototype building module, a global prototype memory, and a feature fusion module.
It significantly improves the segmentation accuracy and training stability of remote sensing images in complex scenes, and enhances the consistency of feature clustering and the ability to distinguish categories.
Smart Images
Figure CN121305084A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of pattern recognition and machine learning technology, and more specifically to a remote sensing semantic segmentation method based on label-guided prototype alignment. Background Technology
[0002] Remote sensing image semantic segmentation is the process of classifying each pixel in a remote sensing image into a specific land cover category (such as buildings, roads, vegetation, etc.). In recent years, the rapid development of deep learning technology has significantly promoted the research progress of semantic segmentation tasks, and it has been widely used in fields such as remote sensing image analysis, urban planning, and environmental monitoring.
[0003] While deep learning has significantly improved the accuracy of semantic segmentation of remote sensing images, existing segmentation networks generally rely on a large number of pixel-level labeled samples for training. In remote sensing scenarios, images are high-resolution, complex in category, and require annotation by professionals, making it difficult to obtain high-quality and sufficient labeled data. To address this, semi-supervised semantic segmentation (SSRS) methods have been proposed to assist learning using a large number of unlabeled images under limited annotation conditions.
[0004] Most mainstream semi-supervised segmentation methods currently employ consistency constraints or pseudo-label learning mechanisms. These methods maintain predictive consistency by enhancing images with varying degrees of strength, or guide model learning using high-confidence pseudo-labels. However, these methods often treat labeled and unlabeled data independently, relying solely on pseudo-labels or consistency constraints for indirect supervision. The lack of direct interaction at the feature level prevents unlabeled sample features from being fully aligned to the semantic space of labeled samples, leading to unstable model training and blurred class boundaries.
[0005] Therefore, how to improve the performance of semi-supervised semantic segmentation of remote sensing images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] In view of the above problems, this invention is proposed to provide a label-guided prototype alignment remote sensing semantic segmentation method that overcomes or at least partially solves the above problems. It establishes semantic connections between labeled and unlabeled samples at the feature level, constructs class prototypes using high-confidence pixel features of labeled samples, and guides the alignment and optimization of unlabeled sample features through a local and global dual-path alignment mechanism, thereby improving the segmentation accuracy and feature discriminativeness of the model in complex remote sensing scenarios.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a remote sensing semantic segmentation method with label-guided prototype alignment, comprising the following steps: Step 1: Collect labeled and unlabeled samples to construct a training dataset; Step 2: Construct a label-guided dual-prototype alignment network and train and optimize it using the training dataset to obtain a remote sensing image semantic segmentation model. The remote sensing image semantic segmentation model includes a shared encoder-decoder network, a prototype construction module, a global prototype memory, and a feature fusion module. The shared encoder-decoder network extracts features and generates prediction results. The prototype construction module constructs local prototypes based on the extracted features and prediction results. The global prototype memory updates the global prototype based on the local prototypes. The feature fusion module fuses the local prototypes and global prototypes to align unlabeled samples to obtain fused features, which are then fed back to the shared encoder-decoder network to obtain the segmentation results. Step 3: Input the acquired remote sensing image to be identified into the remote sensing image semantic segmentation model to obtain the segmentation result.
[0008] Preferably, labeled samples are input into a shared encoder-decoder network to extract labeled features and generate labeled prediction results; unlabeled samples are input into a shared encoder-decoder network to extract unlabeled features and combine them with fused features to generate segmentation results.
[0009] Preferably, the prototype building module is based on tagged features. Labeled prediction results The specific process of building a local prototype is as follows: Step 211: Select the top k% of pixels in the labeled samples whose original label and the predicted label are the same for each category c, and whose confidence scores are ranked from largest to smallest. Form a semantic center set S. c During the prediction process, each pixel generates a set of probability values corresponding to the original label of each category. The highest probability value is used as the confidence level, and the corresponding label is used as the label prediction result. The selected pixels are considered to have high semantic confidence and can be used to represent the semantic center of the category. Step 212: Calculate the local prototype for each category using a weighted average based on the semantic center set and the corresponding confidence scores. The expression is as follows: ; in, S represents the local prototype of category c; c This represents the set of semantic centers corresponding to category c; w represents the feature vector of the i-th pixel in the pixel set of category c; iThis represents the confidence level corresponding to the i-th pixel; local prototypes are dynamically generated in each batch during the training process, which can capture the semantic distribution changes of the current data and reflect the differences in short-term features.
[0010] Preferably, the global prototype memory is generated and updated for each category based on the local prototypes using a momentum update mechanism; the global prototype is represented as: ; in, Represents the global prototype of category c; This represents the momentum factor, used to balance historical semantic information with current sample features.
[0011] The technical effect of the above-mentioned technical solution is that, through this two-stage construction method, the local prototype reflects the distribution of category features within a short-term batch, while the global prototype accumulates stable semantic centers across batches. The combination of the two can provide accurate feature references for unlabeled samples and achieve explicit semantic alignment in the feature space.
[0012] Preferably, the feature fusion module includes a Softmax layer, a concatenate layer, and three Linear layers, used to achieve explicit alignment between unlabeled sample features and the semantic prototypes of labeled samples at the feature level, thereby improving the discriminability and semantic consistency of unlabeled features. Specifically, it includes the following steps: Step 221: Process the unlabeled features through the first Linear layer and the second Linear layer respectively. The prototype feature f, which consists of prototypes of all categories p Projecting onto the prototype space, we calculate the cosine similarity between the unlabeled features and each category prototype, expressed as: ; in, ∈[-1,1] represents unlabeled feature f u With the prototype p of category c c The cosine similarity between them represents the degree of proximity between them in the feature space; the prototype of category c. For local prototypes and / or global prototypes; during feature alignment, you can choose to use only local prototypes or global prototypes as needed, or you can use both at the same time to obtain stronger semantic consistency; when both types of prototypes are enabled at the same time, feature fusion is performed using the global prototype to obtain a more stable category distribution, and local prototype fusion is performed based on the updated features to achieve an iterative strengthening alignment strategy from global to local. Step 222: In the concatenate layer, the weights obtained from the cosine similarity through the Softmax layer are weighted and fused with the prototype to obtain the alignment features projected onto the prototype space. The calculation formula is as follows: ; in, Indicates the alignment features after alignment; represents the prototype of category j; C represents the total number of categories; sim represents the similarity calculation; exp represents the exponential calculation of the Softmax layer; the weights calculated through a Softmax layer ensure that the model pays more attention to the prototypes of categories with higher similarity, while enhancing the discriminativeness and stability of feature clustering. Step 223: After feature projection through the third Linear layer, the aligned features are linearly fused with the original unlabeled features via residual connections to obtain fused features. To prevent information loss, this is represented as: ; in, Indicates fusion characteristics; This represents the adjustment coefficient, used to control the update magnitude guided by the prototype. When both local and global prototypes are used simultaneously, the fusion features corresponding to the local prototype and the fusion features corresponding to the global prototype are added together and fused to obtain the final fusion feature.
[0013] Preferably, the shared encoder-decoder network includes two sets of encoders and decoders that share parameters; one set of encoders and decoders extracts labeled features from labeled samples and generates labeled prediction results; the other set of encoders extracts unlabeled features from unlabeled samples, and the decoder generates segmentation results based on the unlabeled features and the fused features.
[0014] Preferably, a label-guided dual-prototype alignment network is trained by constructing two types of prototype-level constraint losses L. The two types of prototype-level constraint losses include prototype feature consistency loss and class separation loss, expressed as: ; ; ; in, This represents the prototype feature consistency loss; This represents the category separation loss; C represents the total number of categories; Represents a local prototype of category c; Represents the global prototype of category c; m represents the preset similarity threshold; Represents the global prototype of category c1; Represents the global prototype of category c2; This indicates taking the 2-norm.
[0015] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a remote sensing semantic segmentation method with label-guided prototype alignment. It makes full use of labeled data to construct local feature prototype and global feature prototype libraries, establishes the display relationship between labeled and unlabeled samples at the feature level, and optimizes the feature representation of unlabeled samples by introducing a dual-path alignment mechanism in the feature fusion module through category prototypes. This realizes feature layer interaction and semantic alignment between labeled and unlabeled samples, effectively improving feature clustering consistency and category discrimination ability. This design can significantly improve the segmentation accuracy and training stability of remote sensing images in complex scenes, and achieve significant performance improvement in semi-supervised semantic segmentation tasks of remote sensing images. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of the overall framework of the remote sensing image semantic segmentation model provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the feature fusion module structure provided in an embodiment of the present invention; Figure 3 This is a visualization diagram of the prediction results of different methods provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the visualization results of the t-SNE feature distribution provided in the embodiments of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] This invention discloses a remote sensing semantic segmentation method based on label-guided prototype alignment, comprising the following steps: S1: Collect labeled and unlabeled samples to construct a training dataset; S2: Construct a label-guided dual prototype alignment network (LDPA-Net) and train and optimize it using the training dataset to obtain a remote sensing image semantic segmentation model. The remote sensing image semantic segmentation model includes a shared encoder-decoder network, a prototype construction module, a global prototype memory, and a feature fusion module. The shared encoder-decoder network extracts features and generates prediction results. The prototype construction module constructs local prototypes based on the extracted features and prediction results. The global prototype memory updates the global prototype based on the local prototypes. The feature fusion module fuses the local prototypes and global prototypes to align unlabeled samples to obtain fused features, which are then fed back to the shared encoder-decoder network to obtain the segmentation result. The data processing flow is as follows: Figure 1 As shown; S3: Input the acquired remote sensing image to be identified into the remote sensing image semantic segmentation model to obtain the segmentation result.
[0020] Furthermore, labeled samples are input into the shared encoder-decoder network to extract labeled features and generate labeled prediction results; unlabeled samples are input into the shared encoder-decoder network to extract unlabeled features and combine them with fused features to generate segmentation results.
[0021] Furthermore, the prototype building module is based on tagged features Labeled prediction results The specific process of building a local prototype is as follows: S211: Select the pixels in the labeled samples whose original label and the predicted label are the same for each category c, and whose confidence scores are ranked from largest to smallest, forming the semantic center set S. c During the prediction process, each pixel generates a set of probability values corresponding to the original label of each category. The highest probability value is used as the confidence level, and the corresponding label is used as the label prediction result. The selected pixels are considered to have high semantic confidence and can be used to represent the semantic center of the category. S212: Calculate the local prototype for each category using a weighted average based on the semantic center set and the corresponding confidence scores. The expression is as follows: ; in, S represents the local prototype of category c; c This represents the set of semantic centers corresponding to category c; w represents the feature vector of the i-th pixel in the pixel set of category c; i This represents the confidence level corresponding to the i-th pixel; local prototypes are dynamically generated in each batch during the training process, which can capture the semantic distribution changes of the current data and reflect the differences in short-term features.
[0022] Furthermore, the global prototype memory generates and updates the global prototype for each category in the global prototype memory based on the local prototypes using a momentum update mechanism; the global prototype is represented as: ; in, Represents the global prototype of category c; The momentum factor is used to balance historical semantic information with current sample features. When a new local prototype is generated, it is integrated into the global prototype through the momentum update mechanism to maintain the long-term stability of category features. The global prototype memory stores at most T historical global prototypes for each category to prevent semantic drift.
[0023] Furthermore, the feature fusion module structure is as follows: Figure 2 As shown, it includes a Softmax layer, a concatenate layer, and three Linear layers, used to achieve explicit alignment between unlabeled sample features and the semantic prototypes of labeled samples at the feature level, thereby improving the discriminability and semantic consistency of unlabeled features. Specifically, it includes the following steps: S221: The unlabeled features are processed through the first Linear layer and the second Linear layer respectively. The prototype feature f, which consists of prototypes of all categories p Projecting onto the prototype space, we calculate the cosine similarity between the unlabeled features and each category prototype, expressed as: ; in, ∈[-1,1] represents unlabeled feature f u With the prototype p of category c c The cosine similarity between them represents the degree of proximity between them in the feature space; the prototype of category c. For local prototypes and / or global prototypes; during feature alignment, you can choose to use only local prototypes or global prototypes as needed, or you can use both at the same time to obtain stronger semantic consistency; when both types of prototypes are enabled at the same time, feature fusion is performed using the global prototype to obtain a more stable category distribution, and local prototype fusion is performed based on the updated features to achieve an iterative strengthening alignment strategy from global to local. S222: In the concatenate layer, the weights obtained from the cosine similarity through the Softmax layer are weighted and fused with the prototype to obtain the alignment features projected onto the prototype space, represented as: ; in, Indicates alignment features; represents the prototype of category j; C represents the total number of categories; sim represents the similarity calculation; exp represents the exponential calculation of the Softmax layer; the weights calculated through a Softmax layer ensure that the model pays more attention to the prototypes of categories with higher similarity, while enhancing the discriminativeness and stability of feature clustering. S223: After the aligned features are projected through the third Linear layer, they are linearly fused with the original unlabeled features via residual connections to obtain fused features. To prevent information loss, this is represented as: ; in, Indicates fusion characteristics; This represents the adjustment coefficient, used to control the update magnitude guided by the prototype. When both local and global prototypes are used simultaneously, the fusion features corresponding to the local prototype and the fusion features corresponding to the global prototype are added together and fused to obtain the final fusion feature.
[0024] Furthermore, the shared encoder-decoder network includes two sets of encoders and decoders that share parameters; one set of encoders and decoders extracts labeled features from labeled samples and generates labeled prediction results; the other set of encoders extracts unlabeled features from unlabeled samples, and the decoder generates segmentation results based on unlabeled features and fused features.
[0025] Furthermore, a label-guided dual-prototype alignment network is trained by constructing two types of prototype-level constraint losses, L. These two types of prototype-level constraint losses include prototype feature consistency loss and class separation loss, denoted as: ; ; ; in, This represents the prototype feature consistency loss; This represents the category separation loss; C represents the total number of categories; Represents a local prototype of category c; Represents the global prototype of category c; m represents the preset similarity threshold; Represents the global prototype of category c1; Represents the global prototype of category c2; The model adopts the 2-norm. Prototype feature consistency loss constrains local and global prototypes to maintain consistency within the same category, making the category centers learned in different batches more stable. By minimizing the Euclidean distance between them, it achieves consistent alignment between local features and long-term semantic memory. Optimizing the model through prototype feature consistency loss reduces the offset between prototypes of the same category, ensuring the continuity and stability of semantic representation during training. Class separation loss enhances the discriminativeness between prototypes of different categories and avoids semantic overlap between classes. By applying a margin constraint to the cosine similarity between global prototype pairs, it ensures sufficient semantic distance between different categories in the feature space. When the similarity between two prototypes of different categories is greater than m, the loss function generates a penalty term, thereby forcing the model to expand the inter-class distance and strengthen the semantic boundaries of different land cover categories. These two types of prototype-level constraint losses significantly improve the model's semantic discrimination ability and segmentation accuracy through dual constraints, providing a stable feature foundation for subsequent feature consistency and pseudo-label optimization.
[0026] In one specific embodiment, a systematic experimental analysis was conducted to verify the effectiveness and superiority of the present invention. The internationally publicly available remote sensing semantic segmentation datasets ISPRS Vaihingen and Potsdam were used in the experiments. Both datasets contain various typical land cover categories (impervious surfaces, buildings, trees, low vegetation, vehicles, etc.). The datasets were randomly selected as labeled samples at ratios of 1 / 8 and 1 / 16, respectively, with the remainder used as unlabeled samples to simulate the limited labeling conditions in real-world remote sensing scenarios. To fully verify the model performance, several advanced methods, including Fixmatch, Unimatch, LSST, WSCL, and PCSSS, were selected for comparative experiments.
[0027] (1) Qualitative evaluation: To conduct a qualitative evaluation, the semantic segmentation results generated by each method on the test set are visualized, such as... Figure 3 As shown, where Figure 3 In the image, (a) is the input image, (b) is the ground truth annotation, (c) is the Unimatch model trained on 1 / 8 of the labeled data, (d) is the Unimatch model trained on 1 / 4 of the labeled data, (e) is the PCSSS model trained on 1 / 8 of the labeled data, (f) is the PCSSS model trained on 1 / 4 of the labeled data, (g) is the method of this invention trained on 1 / 8 of the labeled data, and (h) is the method of this invention trained on 1 / 4 of the labeled data. The results show that this invention can generate more clearly defined and semantically coherent segmentation maps in complex terrain boundary areas (such as roads and buildings, trees and grass); while traditional pseudo-labeling methods often suffer from category confusion or boundary breaks in such areas. Figure 4 The features shown are visualized, where Figure 4In the image, (a) shows the original encoder feature space before prototype alignment, (b) shows the feature distribution visualization result of local prototype optimization, (c) shows the feature distribution visualization result of global prototype optimization, and (d) shows the feature distribution visualization result of the LDPA-Net model using dual prototype alignment with local and global prototypes. The results show that the feature distribution of traditional semi-supervised methods is loose and has significant inter-class overlap; while the dual prototype alignment mechanism of this invention makes the feature space exhibit a structure of intra-class aggregation and inter-class separation, with labeled and unlabeled samples forming stable cluster centers in the semantic space.
[0028] (2) Quantitative Evaluation: To quantitatively evaluate the model performance, the mean Intersection over Union (mIoU), overall accuracy (OA), and mean F1 score were used as indicators. Experimental results show that the present invention significantly outperforms existing methods on both remote sensing datasets. Specific indicators are shown in Tables 1 and 2.
[0029] Table 1 Test results of the Vaihingen dataset
[0030] Table 2. Test results on the Potsdam dataset.
[0031] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0032] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A remote sensing semantic segmentation method based on label-guided prototype alignment, characterized in that, Includes the following steps: Step 1: Collect labeled and unlabeled samples to construct a training dataset; Step 2: Construct a label-guided dual-prototype alignment network and train and optimize it using the training dataset to obtain a remote sensing image semantic segmentation model; the remote sensing image semantic segmentation model includes a shared encoder-decoder network, a prototype construction module, a global prototype memory, and a feature fusion module; The shared encoder-decoder network extracts features and generates prediction results; the prototype building module constructs local prototypes based on the extracted features and prediction results; the global prototype memory updates the global prototype based on the local prototypes; the feature fusion module fuses the local prototypes and global prototypes to align unlabeled samples to obtain fused features, and feeds them back to the shared encoder-decoder network to obtain segmentation results. Step 3: Input the acquired remote sensing image to be identified into the remote sensing image semantic segmentation model to obtain the segmentation result.
2. The remote sensing semantic segmentation method with tag-guided prototype alignment as described in claim 1, characterized in that, Labeled samples are input into a shared encoder-decoder network to extract labeled features and generate labeled prediction results; unlabeled samples are input into a shared encoder-decoder network to extract unlabeled features and combine them with fused features to generate segmentation results.
3. The remote sensing semantic segmentation method with tag-guided prototype alignment as described in claim 2, characterized in that, Prototype building blocks are based on tagged features Labeled prediction results The specific process of building a local prototype is as follows: Step 211: Select the top k% of pixels in the labeled samples whose original label and the predicted label are the same for each category c, and whose confidence scores are ranked from largest to smallest. Form a semantic center set S. c ; Step 212: Calculate the local prototype for each category using a weighted average based on the semantic center set and the corresponding confidence scores. The expression is as follows: ； in, S represents the local prototype of category c; c This represents the set of semantic centers corresponding to category c; w represents the feature vector of the i-th pixel in the pixel set of category c; i This represents the confidence level corresponding to the i-th pixel.
4. The remote sensing semantic segmentation method with tag-guided prototype alignment as described in claim 1, characterized in that, A global prototype memory is used to generate and update the global prototype for each category in the global prototype memory based on the local prototypes using a momentum update mechanism; the global prototype is represented as: ； in, Represents the global prototype of category c; Represents a local prototype of category c; This represents the momentum factor.
5. The remote sensing semantic segmentation method with tag-guided prototype alignment as described in claim 1, characterized in that, The feature fusion module includes a Softmax layer, a concatenate layer, and three Linear layers, and specifically includes the following steps: Step 221: Process the unlabeled features through the first Linear layer and the second Linear layer respectively. The prototype feature f, which consists of prototypes of all categories p Projecting onto the prototype space, we calculate the cosine similarity between the unlabeled features and each category prototype, expressed as: ； in, ∈[-1,1] represents unlabeled feature f u With the prototype p of category c c Cosine similarity between them; prototype of category c For local prototypes and / or global prototypes; Step 222: In the concatenate layer, the weights obtained from the cosine similarity through the Softmax layer are weighted and fused with the prototype to obtain the alignment features, represented as: ； in, Indicates the alignment features after alignment; Represents the prototype of category j; C represents the total number of categories; sim represents similarity calculation; exp represents the exponential calculation of the Softmax layer; Step 223: After the aligned features are projected through the third Linear layer, they are linearly fused with the original unlabeled features through residual connections to obtain the fused features, represented as: ； in, Indicates fusion characteristics; This represents the adjustment coefficient. When both local and global prototypes are used simultaneously, the fusion features corresponding to the local prototype and the fusion features corresponding to the global prototype are added together and fused to obtain the final fusion feature.
6. The remote sensing semantic segmentation method with tag-guided prototype alignment as described in claim 1, characterized in that, The shared encoder-decoder network consists of two sets of encoders and decoders that share parameters. One set of encoders and decoders extracts labeled features from labeled samples and generates labeled prediction results. The other set of encoders extracts unlabeled features from unlabeled samples, and the decoder generates segmentation results based on the unlabeled features and the fused features.
7. The remote sensing semantic segmentation method with label-guided prototype alignment as described in claim 1, characterized in that, A label-guided dual-prototype alignment network is trained by constructing two types of prototype-level constraint losses, L. These two types of prototype-level constraint losses include prototype feature consistency loss and class separation loss, denoted as: ； ； ； in, This represents the prototype feature consistency loss; This represents the category separation loss; C represents the total number of categories; Represents a local prototype of category c; Represents the global prototype of category c; m represents the preset similarity threshold; Represents the global prototype of category c1; Represents the global prototype of category c2; This indicates taking the 2-norm.