Semi-supervised domain adaptation semantic segmentation method and system based on feature prototype

Abstract

The application discloses a feature prototype-based semi-supervised domain adaptation semantic segmentation method, comprising the following steps: acquiring labeled source domain images, labeled target domain images and unlabeled target domain images; constructing a model, wherein the model comprises an encoder, a decoder, a classifier and a style conversion network; performing supervised training on the segmentation model by using the labeled source domain images and the labeled target domain images, and calculating feature prototypes by using features in an intermediate layer of the classifier; performing unsupervised training on the style conversion network by using the target domain images and the source domain images; performing image-level alignment on converted images of the decoder and the style conversion network respectively; improving feature-level alignment by using the feature prototypes; and processing the feature prototypes according to the sources of acquisition respectively. Through feature prototype contrast learning, the feature extraction process is constrained, the ability of the network to extract semantic features is enhanced, the generalization performance of the model is improved, and better segmentation effects can be achieved on the target domain data distribution.

Description

Technical Field

[0001] The present invention relates to a method in the field of computer vision and image processing, and in particular to a semi-supervised domain adaptation semantic segmentation method and system based on feature prototypes. Background Art

[0002] Image domain adaptation technology aims to improve the generalization ability of deep models. The model can learn on one data distribution and then improve its generalization on another data distribution through domain adaptation technology, thereby improving the model's task performance on another data distribution.

[0003] Image segmentation is a fundamental problem in the field of computer vision. Semantic segmentation is a pixel-level classification problem that aims to classify each pixel in a given image according to a predefined target category.

[0004] Semi-supervised domain adaptation semantic segmentation leverages the two aforementioned techniques to achieve more accurate semantic segmentation through domain adaptation. The model learns from a large amount of labeled source domain data, a small amount of labeled target domain data, and a large amount of unlabeled target domain data, achieving better segmentation on the target domain data. While neural networks currently perform well on various computer vision tasks, most models are data-driven and require supervised training. In reality, data collection and labeling are very costly. This difficulty is particularly prominent in the field of medical imaging, where the human effort required to ensure accurate labeling requires the involvement of professional doctors. Given the current prevalence of large amounts of labeled data in one domain and small amounts in another, domain adaptation technology is highly valuable, reducing reliance on data labeling while improving model generalization. Image semantic segmentation is also crucial as it underpins many applications. Domain adaptation semantic segmentation, which combines these two techniques, has become a research hotspot in both academia and industry. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a semi-supervised domain adaptation semantic segmentation method and system based on feature prototypes.

[0006] According to one aspect of the present invention, a semi-supervised domain adaptation semantic segmentation method based on feature prototypes is provided, comprising:

[0007] Obtain a training dataset, where the training dataset includes labeled source domain images, labeled target domain images, and unlabeled target domain images;

[0008] Building a model, the model including an encoder, a decoder, a classifier, and a style transfer network;

[0009] Performing supervised training of a segmentation model using the annotated source domain image and the annotated target domain image, wherein the segmentation model includes an encoder and a classifier;

[0010] Calculating feature prototypes using features of the middle layer of the classifier;

[0011] Performing unsupervised training on the style conversion network using the target domain image and the source domain image;

[0012] Performing image-level alignment on the decoder and the style transfer network's converted images respectively;

[0013] improving feature-level alignment using the feature prototypes;

[0014] The feature prototypes are processed separately according to their acquisition sources.

[0015] Preferably, the image-level alignment includes:

[0016] The target domain image is converted into a source domain style image by the decoder, and image-level aligned with the source domain image;

[0017] The source domain image is converted into a target domain style image through the style conversion network, and image-level alignment is performed with the target domain image;

[0018] The feature-level alignment includes:

[0019] Using contrastive learning, feature prototypes from labeled source and target domains are aligned;

[0020] Using contrastive learning, feature prototypes from labeled and unlabeled target domains are aligned;

[0021] The features of the unlabeled target domain image and the source domain features are aligned.

[0022] Preferably, the supervised training segmentation model is performed based on the predicted segmentation result of the classifier, and the loss function of the encoder and the classifier is a superposition of weighted cross entropy loss and dice loss:

[0023]

[0024] Among them, the plus sign is preceded by weighted cross entropy loss, and the plus sign is followed by dice loss. They are used together to supervise the segmentation results. In the formula, y represents the true label. Represents the output of the network, y c represents the region where the foreground class is c in the ground-truth annotation.

[0025] Preferably, the calculation of the feature prototype is performed based on the intermediate layer features of the classifier;

[0026] The calculation of the feature prototype includes two categories:

[0027] For annotated source domain images and annotated target domain images, the correct regions predicted in the segmentation are output by the real annotation calculation model; the features of the regions are selected and averaged in the length and width dimensions to obtain annotated feature prototypes;

[0028] For an unlabeled target domain image, a probability threshold is set; based on the probability of each point in the segmentation prediction of the model, an area greater than the probability threshold is selected as an area for calculating a feature prototype; and the semantic features of the area are calculated to obtain a feature prototype.

[0029] Preferably, the style transfer network adopts a CycleGAN-style encoding and decoding structure, and the training method is to perform adversarial learning together with the discriminant network. The process is unsupervised, and it is necessary to make the discriminant network unable to distinguish whether the input image is real or generated by the network.

[0030] Preferably, the feature-level alignment is performed via feature prototypes;

[0031] For the feature prototype p extracted from the labeled input image s→t or Let p m Represents the feature prototype of the mth semantic category;

[0032] will p m Calculate the cosine similarity with all prototypes stored in each semantic interval in the feature dictionary;

[0033] Sort the similarities from small to large, and select the top k maximum values ​​to calculate the average similarity;

[0034] Using p m Comparative learning with the similarity of each semantic category;

[0035] For the feature prototype extracted from the unlabeled data in the target domain It needs to be used for comparative learning with the target domain semantic prototype updated by exponential sliding average;

[0036] Preferably, the feature prototype is extracted from the classifier intermediate layer features through correlation processing;

[0037] If the prototype is extracted from a labeled source domain or target domain, it needs to be stored in the feature dictionary. At the same time, the feature prototype extracted from the latter needs to be used for sliding average update of the target domain semantic feature prototype;

[0038] If the prototype is extracted from an unlabeled target domain, it does not need to be stored in the feature dictionary and is only used for comparative learning.

[0039] Preferably, feature prototypes from labeled data are stored in a feature dictionary according to whether they are extracted from the target domain or the source domain, and what semantic categories they represent. The feature dictionary partitions and stores prototypes from labeled data according to semantic categories. The dictionary is set to a length, and the data in it will be updated as the neural network is updated. When the dictionary is full, the old feature prototypes will be deleted first.

[0040] In addition, the prototypes from the labeled target domain will be used to update and maintain the semantic prototypes of the target domain separately, and the update method adopts exponential sliding average; i It represents the prototype of a semantic category in the target domain updated in the i-th iteration, and is expressed as represents the feature prototype of this category obtained from the labeled target domain data in the i-th iteration, and the sliding update process is as follows:

[0041]

[0042] According to a second aspect of the present invention, a semi-supervised domain adaptation semantic segmentation system based on feature prototypes is provided, comprising:

[0043] Encoder module: This module extracts features from the target domain style image;

[0044] Decoder module: This module uses the features extracted by the encoder to generate or reconstruct the image of the source domain style;

[0045] Classifier module: This module uses the features extracted by the encoder to predict the segmentation results, and the features of the intermediate layers are also used to calculate the feature prototypes;

[0046] Style conversion module: This module accepts the source domain style image as input and outputs the image converted into the target domain style;

[0047] Feature prototype module: This module is used to calculate the feature prototype from the intermediate layer features of the classifier, update and maintain the feature dictionary, and update the target domain semantic prototype through the sliding average method.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The embodiment of the present invention constrains the process of feature extraction of convolutional neural networks through feature prototype comparative learning, enhances the network's ability to extract semantic features, improves the generalization performance of the model, and enables the model to achieve better segmentation effects on new data distributions.

[0050] The present invention proposes a semi-supervised domain-adaptive semantic segmentation method and system based on feature prototypes. This method uses a small number of labeled target domain data samples and a large number of labeled source domain data samples for model training. It then uses a convolutional neural network to extract feature prototypes containing rich semantic information. These prototypes are then used to create and maintain a feature dictionary that is partitioned into different semantic categories and stores feature prototypes from different training images for subsequent feature prototype comparative learning. This approach makes it possible to utilize information from multiple samples during training.

[0051] In the embodiment of the present invention, feature prototypes extracted from a small number of labeled target domain data samples are used to independently update the target domain semantic prototype via an exponential sliding average method. This prototype is then used for comparative learning alongside feature prototypes extracted from unlabeled target domain data, thereby narrowing the feature distributions of labeled and unlabeled target domain data. Because the previous comparative learning between the prototypes extracted from labeled data and the prototypes stored in the feature dictionary narrowed the semantic feature distributions between labeled data from different domains, and now the feature distributions between labeled and unlabeled data from the same domain are narrowed, better alignment is achieved at the feature level, the generalization performance of the model is enhanced, and the segmentation effect on the target domain distribution is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0053] Figure 1 Flowchart of a semi-supervised domain adaptation segmentation method based on feature prototypes according to an embodiment of the present invention;

[0054] Figure 2 4 is a block diagram of a semi-supervised domain adaptation segmentation method based on feature prototypes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0056] like Figure 1 As shown, the semi-supervised domain adaptation semantic segmentation method based on an embodiment of the present invention includes the following processes:

[0057] S11, the training data is a large number of labeled source domain images, a small number of labeled target domain images, and a large number of unlabeled target domain images; the convolutional network mainly consists of four networks: encoder, decoder, classifier, and style transfer, which extract features from these images in different domains;

[0058] S12, the labeled source and target domain images are used for supervised training of the segmentation model, and the features of the intermediate layers of the classifier are used to calculate the feature prototype;

[0059] S13, unlabeled target domain images are used to train the style transfer network in an unsupervised adversarial manner, and the style transfer images generated by the network are used for image-level alignment. The style transfer results of the source domain images are used to extract feature prototypes, and the feature-level alignment is improved through related operations.

[0060] S14, the feature prototype is extracted from the middle layer feature of the classifier after relevant processing. If the prototype is extracted from the labeled source domain or target domain, it needs to be stored in the feature dictionary. At the same time, the feature prototype extracted from the latter also needs to be used for sliding average update of the target domain semantic feature prototype; if the prototype is extracted from the unlabeled target domain, it does not need to be stored in the feature dictionary, but is only used for comparative learning.

[0061] In this embodiment, the purpose of image-level alignment is to make the image styles of the two domains as similar as possible through style conversion. This is called alignment, which can also be understood as reducing the difference between the two domains (data distribution). Feature-level alignment is processed at the feature level to make the features extracted by the model from the two domain images as similar as possible. The purpose of obtaining feature prototypes is to expand and update the content of the feature dictionary, which will be used in subsequent related calculations; but it is more to use feature prototypes to improve the effect of semantic feature distribution alignment between different domains and reduce the difference in semantic features extracted from data in different domains.

[0062] In a preferred embodiment of S11 provided by the present invention, all source-domain images include pixel-level annotations, while the target-domain images are largely unannotated. The small number of annotated target-domain images accounts for less than a quarter of the total number of target-domain images, and can even be as little as 5%. A style transfer network is used to convert source-domain style images into target-domain style images. The encoder is used only to extract features from the target-domain style image. The decoder then generates the source-domain style image, and the classifier uses the features extracted by the encoder to predict segmentation results. The classifier's intermediate layer features are also used to extract feature prototypes.

[0063] In a preferred embodiment of S12 provided by the present invention, supervised training is performed only on the predicted segmentation results of the classifier, and the loss function for training the encoder and the classifier is a superposition of the weighted cross entropy loss and the dice loss, as shown below:

[0064]

[0065] The weighted cross entropy loss is before the plus sign in the above formula, and the dice loss is after the plus sign. They are used together to supervise the segmentation results. In the formula, y represents the true label. Represents the output of the network, y c represents the region where the foreground class is c in the ground-truth annotation.

[0066] In the preferred embodiment of S12 provided by the present invention, the calculation of the feature prototype is performed on the middle layer features of the classifier. The reason is that the feature distance is closer to the segmentation output at this time, so it contains more semantically relevant information, which is very suitable for extracting the feature prototype. Since the model input can be an annotated image or an unannotated image, this embodiment designs two methods for extracting feature prototypes for these two situations. For the annotated input image, regardless of whether it is from the source domain or the target domain, the calculation method of the feature prototype is the same.

[0067] First, we use the real annotations to calculate the correct predicted areas in the model output segmentation. Then, we select the features of these areas and average them in the length and width dimensions to obtain the prototype with annotated features. The process can be expressed as follows:

[0068]

[0069]

[0070] In the above formula, argmax represents the function of selecting the channel with the largest value in two dimensions, y represents the true label, Represents the model segmentation prediction, M represents the mask of the predicted correct area, y c＝i represents the area of ​​category i in the real annotation, N i Represents the total number of pixels in M, f latent Represents the features of the middle layer of the classifier, and the final result is the feature prototype (column vector) p for the semantic category i i .

[0071] If the input is an unlabeled target domain image, you need to set a probability threshold first. Then, based on the probability of each point in the model segmentation prediction, select the area greater than the threshold as the area for calculating the feature prototype. The larger the probability, the lower the uncertainty, which means that the model predicts the category of the pixel point more accurately. Therefore, the semantic feature prototype calculated using these areas will be more representative. The process can be expressed as follows:

[0072]

[0073]

[0074] In the above formula, p represents the set threshold, and softmax normalizes the value range of the segmentation prediction logits to the interval [0,1] so that the probability can be expressed. It represents the feature prototype of the unlabeled target domain image with semantic category i, and the meanings of the other symbols are the same as before.

[0075] In a preferred embodiment of S13 provided by the present invention, the unlabeled target domain image is used to train a style transfer network to convert the source domain style image into the target domain style image. The neural network adopts a CycleGAN-style codec structure, and the training method is to use a discriminant network for adversarial learning. The process is unsupervised, and it is necessary to make the discriminant network unable to distinguish whether the input image is real or network-generated. The neural network converts the source domain image into the target domain style, thereby producing the target domain style image, so that labels can be used for supervision during the classifier training process, so that the styles of the source domain image and the target domain image become similar, which is image-level data distribution alignment.

[0076] In a preferred embodiment of S13 provided by the present invention, feature-level alignment is performed by feature prototypes. Feature prototypes extracted from the labeled source domain or target domain are compared with the prototypes stored in the feature dictionary for learning, thereby increasing the similarity of semantic features of the same category and reducing the similarity of semantic features of different categories. At the same time, because the dictionary stores the features from the target domain image x t and the source domain image x that is converted into the target domain style s→t Therefore, contrastive learning brings the semantic feature distributions of the annotated data of these two domains closer.

[0077] Specifically: For the feature prototype p extracted from the labeled input image s→t or Here we use p to represent it, let p m Represents the feature prototype of the mth semantic category. First, p m Calculate the cosine similarity of all prototypes stored in each semantic interval in the feature dictionary, sort the similarities from small to large, select the top k maximum values ​​to calculate the average similarity, and finally use p m Comparative learning is performed with the similarity of each semantic category. The first process can be expressed as follows:

[0078]

[0079] In the above formula represents the i-th feature prototype of the k-th semantic category interval stored in the feature dictionary, and L represents the length of the dictionary. The next step is to average the similarity ranking. The second process can be expressed as follows:

[0080]

[0081] Get p m After the similarity with the prototypes of each semantic category is obtained, feature prototype comparison learning can be performed. The third process can be expressed as follows:

[0082]

[0083] In the above formula, C represents the number of semantic categories and τ represents the temperature coefficient.

[0084] In order to make full use of a small amount of labeled target domain data, the present invention provides a preferred embodiment, in which the prototypes extracted therefrom are used to separately update and maintain the target domain semantic prototypes for comparative learning to improve the feature alignment of labeled and unlabeled target domain data.

[0085] Specifically, if the feature prototype is extracted from the target domain unlabeled data, use To represent it, it needs to be used to compare and learn with the target domain semantic prototype updated by exponential sliding average. Given a feature prototype from the unlabeled target domain and belonging to the mth semantic category The contrast loss is then calculated as follows:

[0086]

[0087] In the above formula, p i,k It represents the target domain semantic prototype updated by the sliding average method alone, and the subscript represents the target domain semantic prototype of category k in the i-th iteration.

[0088] Overall, feature prototypes are used for alignment in two different scenarios: the first is to align features between a large amount of labeled source domain data and a small amount of target domain data, and the second is to align features between a small amount of labeled target domain data and a large amount of unlabeled target domain data. This approach fully utilizes the small amount of labeled examples in the target domain. Furthermore, since feature alignment between data in the same domain is less difficult than between data in different domains, this method can also be viewed as a progressive alignment approach.

[0089] In order to make full use of the information in multiple samples and make feature prototype comparative learning possible, the present invention provides an embodiment that uses a feature dictionary to store feature prototypes from labeled data. In a preferred embodiment of executing S14, the feature prototype is calculated through the features of the middle layer of the classifier. The classifier is used to output the segmentation prediction result. The features of the middle layer contain more semantic features, and the number of channels is relatively small, so it is very suitable for extracting feature prototypes. The feature prototype process for labeled data and unlabeled data has been explained in the previous article. The feature prototypes from the labeled data will be stored in the feature dictionary according to whether they are extracted from the target domain or the source domain, and what semantic categories they represent. The feature dictionary partitions and stores the prototypes from the labeled data according to the semantic category. At the same time, the dictionary is set with a length, and the data in it will be continuously updated as the neural network is updated. When the dictionary is full, the old feature prototypes will be deleted first.

[0090] In addition, the prototypes from the labeled target domain will also be used to update and maintain the semantic prototypes of the target domain separately. The update method uses exponential sliding average to make full use of these small amounts of labeled data. i It represents the prototype of a semantic category in the target domain updated in the i-th iteration, and is expressed as represents the feature prototype of this category obtained from the labeled target domain data in the i-th iteration, and the sliding update process is as follows:

[0091]

[0092] like Figure 2 FIG2 is a block diagram of a semi-supervised domain adaptation semantic segmentation system based on feature prototypes according to an embodiment of the present invention.

[0093] Please refer to Figure 2 The feature prototype-based semi-supervised domain adaptation semantic segmentation system of this embodiment is used to implement the feature prototype-based semi-supervised domain adaptation semantic segmentation method of the above embodiment, which includes: an encoder module, a decoder module, a classifier module, a style conversion module and a feature prototype module.

[0094] Encoder module: This module extracts features from the target domain style image;

[0095] Decoder module: This module uses the features extracted by the encoder to generate or reconstruct the image of the source domain style;

[0096] Classifier module: This module uses the features extracted by the encoder to predict the segmentation results, and the features of the intermediate layers are also used to calculate the feature prototypes;

[0097] Style conversion module: This module accepts the source domain style image as input and outputs the image converted into the target domain style;

[0098] Feature prototype module: This module is used to calculate the feature prototype from the intermediate layer features of the classifier, update and maintain the feature dictionary, and update the target domain semantic prototype through the sliding average method.

[0099] The technologies used in the above modules can be described in the semi-supervised domain adaptation semantic segmentation method based on feature prototypes, which will not be repeated here.

[0100] In summary, the semi-supervised domain-adaptive semantic segmentation method and system based on feature prototypes provided in the above embodiments use deep learning technology, and utilize feature prototypes and feature dictionaries to improve the ability of neural networks to extract features, and regularize semantic features to further improve the feature alignment effect, so that the model can learn on target domain data with only a small amount of annotations and output better target domain segmentation prediction results.

[0101] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, that is, the embodiments in the system can be understood as preferred examples for implementing the method, which will not be elaborated here.

[0102] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered a hardware component, and the devices included therein for implementing the various functions can also be considered as structures within the hardware component; the devices for implementing the various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0103] Disclosed herein are only preferred embodiments of the present invention. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention and are not intended to limit the present invention. Any modifications and variations made by those skilled in the art within the scope of this specification are intended to fall within the scope of protection of the present invention.

Claims

1. A semi-supervised domain adaptation semantic segmentation method based on feature prototypes, characterized by: Obtain a training dataset, where the training dataset includes labeled source domain images, labeled target domain images, and unlabeled target domain images; Building a model, the model including an encoder, a decoder, a classifier, and a style transfer network; Performing supervised training of a segmentation model using the annotated source domain image and the annotated target domain image, wherein the segmentation model includes an encoder and a classifier; Calculating feature prototypes using features of the middle layer of the classifier; Performing unsupervised training on the style conversion network using the target domain image and the source domain image; Performing image-level alignment on the decoder and the style transfer network's converted images respectively; improving feature-level alignment using the feature prototypes; Processing the feature prototypes respectively according to their acquisition sources; The calculation of the feature prototype includes two categories: For labeled source and target images, the model outputs the predicted correct regions in the segmentation using the ground truth annotations. Select the features of the correct area and average them in the length and width dimensions to obtain a prototype with labeled features; For unlabeled target domain images, set a probability threshold; According to the probability of each point in the segmentation prediction of the model, the area greater than the probability threshold is selected as the area for calculating the feature prototype; The semantic features of the region of the feature prototype are calculated to obtain the feature prototype.

2. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 1 is characterized in that The image-level alignment includes: The target domain image is converted into a source domain style image by the decoder, and image-level aligned with the source domain image; The source domain image is converted into a target domain style image through the style conversion network, and image-level alignment is performed with the target domain image; The feature-level alignment includes: Using contrastive learning, feature prototypes from labeled source and target domains are aligned; Using contrastive learning, feature prototypes from labeled and unlabeled target domains are aligned; The features of the unlabeled target domain image and the source domain features are aligned.

3. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 1, characterized in that The supervised training segmentation model is performed based on the predicted segmentation results of the classifier, and the loss function of the encoder and classifier is the superposition of weighted cross entropy loss and dice loss: Among them, the plus sign is the weighted cross entropy loss, and the plus sign is the dice loss. In the formula, y represents the true label. Represents the output of the classifier, y c represents the region where the foreground class is c in the ground-truth annotation.

4. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 1, characterized in that The style transfer network uses a CycleGAN-style codec structure and is trained by adversarial learning with the discriminant network. The process is unsupervised and the discriminant network must be unable to distinguish whether the input image is real or generated by the network.

5. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 2, characterized in that The feature-level alignment is performed based on the feature prototype; For the feature prototype p extracted from the labeled input image s→t or Let p m Represents the feature prototype of the mth semantic category; will p m Calculate the cosine similarity with all prototypes stored in each semantic interval in the feature dictionary; Sort the cosine similarities from small to large, and select the first k maximum values ​​to calculate the average similarity; Using p m Comparative learning is performed with the average similarity of each semantic category; For the feature prototype extracted from the unlabeled data in the target domain It needs to be used for comparative learning with the target domain semantic prototype updated by exponential sliding average.

6. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 1, characterized in that The feature prototypes are processed separately according to the acquisition sources, including: If the prototype is extracted from a labeled source domain or target domain, it is stored in the feature dictionary; The feature prototype extracted from the target domain is used for sliding average to update the target domain semantic feature prototype; If the prototype is extracted from the unlabeled target domain, it is only used for contrastive learning.

7. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 6, characterized in that The feature dictionary partitions and stores prototypes from labeled data according to semantic categories. The dictionary has a set length, and the data within it will be updated as the neural network is updated. When the feature dictionary is full, the old feature prototypes are deleted first.

8. The semi-supervised domain adaptation semantic segmentation method based on feature prototype according to claim 6, characterized in that The sliding average updates the target domain semantic feature prototype, including: With p i It represents the prototype of a semantic category in the target domain updated in the i-th iteration, use represents the feature prototype of this category obtained from the labeled target domain data in the i-th iteration, and the sliding update process is: α is a weight coefficient with a value range of [0,1], which adjusts the update speed of the target domain category feature prototype.

9. A semi-supervised domain adaptation semantic segmentation system based on feature prototypes, characterized by: include: The encoder module extracts features from the target domain style image; The decoder module generates or reconstructs the source-domain style image using the features extracted by the encoder; The classifier module uses the features extracted by the encoder to predict the segmentation results, while the features of the intermediate layers are used to calculate the feature prototypes; A style transfer module that accepts an image in the source domain style as input and outputs an image in the target domain style; Feature prototype module, which is used to calculate feature prototypes from the intermediate layer features of the classifier, update and maintain the feature dictionary, and update the target domain semantic prototype through the sliding average method; The calculation of the feature prototype includes two categories: For labeled source and target images, the model outputs the predicted correct regions in the segmentation using the ground truth annotations. Select the features of the correct area and average them in the length and width dimensions to obtain a prototype with labeled features; For unlabeled target domain images, set a probability threshold; According to the probability of each point in the segmentation prediction of the model, the area greater than the probability threshold is selected as the area for calculating the feature prototype; The semantic features of the region of the feature prototype are calculated to obtain the feature prototype.

Images