Image semantic segmentation active domain adaptation method and system based on segmentation all-in-one model

Abstract

The invention provides an image semantic segmentation active domain adaptation method and system based on a segmentation cutting model. The method comprises the following steps: generating a full-image mask for an image of a target domain by using a pre-trained segmentation cutting model; the images of the target domain are sampled in proportion, initial labeling is carried out on the full-image mask of each sampled image, an initial target domain labeling data set is obtained, and labeling is that a semantic category is given to a full-image mask area generated by each image; training the semantic segmentation model through semi-supervised learning; wherein in the training iteration of the semi-supervised learning, active learning is triggered according to a preset period, and an updated target domain labeling data set is obtained and used for continuous training of the semi-supervised learning. According to the method, the technical purpose of obtaining initial annotation data with complete semantics and clear structure at relatively low labor cost is achieved, and the technical problems that existing semantic segmentation training depends on large-scale pixel-level manual annotation, the annotation efficiency is low and the cost is high are solved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an active domain adaptation method and system for image semantic segmentation based on a segmentation-all model. Background Technology

[0002] Image semantic segmentation is a fundamental task in computer vision, aiming to assign a semantic category label to each pixel in an image. This task has wide applications in autonomous driving, robot navigation, intelligent security, and medical image analysis. In autonomous driving systems, semantic segmentation can accurately identify passable areas, various traffic participants (such as pedestrians, vehicles, and bicycles), and road facilities (such as traffic signs, traffic lights, and guardrails) in images. Figure 1 As shown, providing high-precision environmental understanding for vehicles is crucial input information for downstream positioning, path planning, and decision control.

[0003] Deep learning technology has developed rapidly in recent years, with semantic segmentation models based on convolutional neural networks or Transformers emerging and achieving significant performance improvements. However, these advanced models often rely on large-scale, high-quality pixel-level labeled data for training. In semantic segmentation tasks, manually labeling each image's pixels is an extremely time-consuming and labor-intensive task. Taking the widely used Cityscapes dataset for driving scenes as an example, labeling a 2048×1024 resolution street view image requires an average of 1.5 hours for professional labelers, and can take up to 3 hours under adverse weather conditions such as rain, snow, fog, or darkness. Such labeling costs significantly limit the rapid iteration and deployment of semantic segmentation technology in diverse real-world scenarios.

[0004] To reduce the cost of manual annotation, researchers have proposed various mitigation strategies. One approach utilizes a large number of synthetic images generated by virtual simulators such as CARLA, GTA5, and SYNTHIA as source domain training data, which presents a problem of inter-domain discrepancies with the target real images. To address this, domain adaptation techniques have been proposed to transfer the model from the synthetic domain to the real domain. While unsupervised domain adaptation methods in recent years (such as adversarial training, pseudo-label self-training, and prototype-based contrastive learning training) have narrowed the distribution gap between the source and target domains to some extent, their performance still faces significant bottlenecks when inter-domain differences are large or the target domain data is overly complex, making it difficult to meet the accuracy requirements of practical applications. Another approach uses a small amount of labeled data from the target domain combined with a large amount of unlabeled data to train a semantic segmentation model through semi-supervised learning. These methods include techniques based on pseudo-labels, self-supervised alignment, and consistency regularization, which can achieve certain accuracy improvements with limited labeled data, such as CN119313866A. However, most current semi-supervised semantic segmentation methods often randomly sample target domain samples for manual annotation or pseudo-label generation, neglecting the active evaluation and selection of sample information value. Meanwhile, most of these methods fail to make full use of the large amount of simulation image data available in autonomous driving scenarios, resulting in insufficient mining of source domain supervision information.

[0005] Therefore, there is an urgent need to develop an active domain adaptation method for image semantic segmentation, so as to achieve the technical goal of obtaining semantically complete and structurally clear initial labeled data with low human cost, and solve the technical problems of existing semantic segmentation training relying on large-scale pixel-level manual labeling, low labeling efficiency and high cost. Summary of the Invention

[0006] To address one of the shortcomings of existing technologies, the purpose of this application is to provide an active domain adaptation method and system for image semantic segmentation based on a segmentation-all-model approach.

[0007] A first aspect of this application provides an active domain adaptation method for image semantic segmentation based on a segmentation-all model, comprising:

[0008] A pre-trained all-segmentation model is used to generate a full-image mask for the target domain image;

[0009] The images of the target domain are sampled proportionally, and the full-image mask of each sampled image is initially labeled to obtain an initial target domain labeled dataset. The label is to assign a semantic category to the full-image mask region generated for each image.

[0010] Based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data, a semantic segmentation model is trained through semi-supervised learning.

[0011] In the training iteration of the semi-supervised learning, active learning is triggered according to a preset period. In each round of active learning, the segmentation model is used to generate a full-image mask, sample and manually annotate the unlabeled image in the target domain. The new annotated mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is used for the continued training of the semi-supervised learning.

[0012] Optionally, generating a full-image mask for the target domain image using a pre-trained segmentation model includes:

[0013] For all unlabeled images in the target domain, a pre-trained all-image segmentation model is used for inference, and its full-image segmentation interface is called to generate a high-quality mask set covering the entire image; wherein, the number of masks for each image ensures that the main semantic instances in the image are covered.

[0014] Optionally, the step of training a semantic segmentation model through semi-supervised learning based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data includes:

[0015] For unlabeled data in the target domain, the semantic segmentation model described above is used to generate prediction results and extract pseudo-labels.

[0016] A strong-weak enhancement consistency strategy is adopted for consistency training. The strong-weak enhancement consistency strategy applies image enhancement and feature perturbation to the same image and encourages the semantic segmentation model to maintain consistent prediction results under different perturbations.

[0017] Supervised cross-entropy loss and consistency loss are used as loss functions for the semi-supervised learning training, wherein the supervised cross-entropy loss is used for labeled images and the consistency loss is used for pseudo-labeled images.

[0018] Optionally, each round of active learning includes the following process:

[0019] Use the current trained model to predict all unlabeled target images and generate the semantic category and prediction confidence of each pixel as pseudo-labels;

[0020] Using the target domain pseudo-label as a cue, the segmentation model is invoked to generate a mask for the target image;

[0021] For each generated mask region, the EIS (Entropy-IoU-Sum) index is calculated based on its prediction results. Taking into account both class uncertainty and current class IoU performance, the EIS is used to measure the learning value of the mask. The higher the mask score, the more beneficial the mask region is to model training.

[0022] Based on the scores from high to low, the mask area is sampled, and a fixed proportion is manually labeled.

[0023] The newly labeled image and mask region are added to the target domain label data, and the corresponding image or mask is removed from the target domain label data to complete this round of active learning update. The target domain label data is the sum of the initial target domain label dataset and the target domain dataset obtained by subsequent active learning annotation.

[0024] Optionally, after each round of active learning, the method further includes: employing an IoU-based class weighting mechanism to assign training weights to each class, which is used to dynamically adjust for uneven distribution among classes.

[0025] Optionally, the IoU-based class weighting mechanism includes:

[0026] On all currently labeled data in the target domain, calculate the IoU value for each category, which represents the current model's ability to recognize that category;

[0027] The category weights are calculated using the inverse of IoU.

[0028] The class weights are applied to the class prediction of each pixel in the source domain supervision loss and the target domain pseudo-label loss.

[0029] Optionally, each round of active learning is executed continuously as a closed-loop cycle until the cumulative sampling and labeling ratio of active learning reaches a preset threshold. At this point, active learning and IoU weight updates are no longer performed, and only semi-supervised learning training continues until the number of training rounds reaches the maximum value.

[0030] A second aspect of this application provides an active domain adaptation system for image semantic segmentation based on a segmentation-all model, comprising:

[0031] Candidate mask generation module: Generates a full-image mask for the target domain image using a pre-trained segmentation model;

[0032] Initialization annotation module: Samples the images of the target domain proportionally, performs initial annotation on the full-image mask of each sampled image, and obtains an initial target domain annotation dataset. The annotation assigns a semantic category to the full-image mask region generated for each image.

[0033] Semi-supervised training module: Based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data, a semantic segmentation model is trained through semi-supervised learning;

[0034] Active learning module: In the training iteration of the semi-supervised learning, active learning is triggered according to a preset period. In each round of active learning, the segmentation model is used to generate a full-image mask, sample and manually annotate the unlabeled image in the target domain. The new annotated mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is used for the continued training of the semi-supervised learning.

[0035] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to perform the described active domain adaptation method for image semantic segmentation based on the segmentation-everything model.

[0036] The image semantic segmentation active domain adaptation method and system provided in this application, based on the "segmentation of everything" model, achieves the technical goal of obtaining semantically complete and structurally clear initial labeled data with low human cost by using high-quality masks generated based on the "segmentation of everything" model and manually labeled masks. This solves the technical problems of existing semantic segmentation training relying on large-scale pixel-level manual labeling, low labeling efficiency, and high cost. At the same time, by adopting a phased training mechanism that combines semi-supervised learning and active learning, it can fully explore the information value of unlabeled samples in the target domain under a limited labeling budget, and dynamically guide the model to learn key regions. This solves the technical problems of waste of supervision resources and insufficient model adaptability caused by random sampling of samples in existing methods.

[0037] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0038] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0039] Figure 1 This is a schematic diagram of existing image semantic segmentation technology;

[0040] Figure 2 The flowchart illustrates an active domain adaptation method for image semantic segmentation based on a segmentation-all model, according to an exemplary embodiment.

[0041] Figure 3 This is a schematic diagram illustrating an active domain adaptation method for image semantic segmentation based on a segmentation-all model, according to an exemplary preferred embodiment. Detailed Implementation

[0042] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0043] Current semantic segmentation training relies on large-scale pixel-level manual annotation, which is inefficient and costly. To further improve model performance in the target domain and reduce annotation costs, active learning techniques can be introduced into semantic segmentation tasks. Active learning evaluates the information content and diversity of unlabeled samples and selects the most valuable samples for model training for annotation, aiming to achieve performance as close as possible to fully supervised learning with minimal annotation costs. Current research has proposed various sample selection strategies for active learning, such as image-level sampling, pixel-level or region-level sampling, and superpixel-level sampling. While these strategies reduce the annotation burden to some extent, they still face the challenge of balancing sample selection granularity with annotation efficiency. Especially in real-world autonomous driving scenarios, where scene changes are complex and class distributions are extremely uneven, traditional sample-level selection methods are prone to missing key classes or failing to focus on effective regions.

[0044] Against this backdrop, this application provides a method framework that combines a segmentation model with an active learning mechanism. By introducing a mask-level sample selection mechanism and combining domain adaptation and semi-supervised training strategies, it can efficiently select candidate regions with rich information and quick annotation without relying on a large number of target domain annotations, and improve the generalization performance of the model in the target domain. This significantly improves the accuracy and efficiency of the semantic segmentation model in new scenarios while reducing manual costs.

[0045] Reference Figure 2 As shown in one embodiment of this application, an active domain adaptation method for image semantic segmentation based on a segmentation-everything model is provided, comprising the following steps:

[0046] S100 uses a pre-trained segmentation model to generate a full-image mask for the target domain image;

[0047] Existing semantic segmentation models rely on large-scale pixel-level manual annotation, which is inefficient and costly, limiting the rapid promotion and application of these models in multiple scenarios and cross-domain environments.

[0048] In this step, the Segment Anything Model (SAM) is employed. SAM possesses excellent versatility and zero-shot segmentation capabilities, automatically generating high-quality candidate masks for different images and scenes, covering semantically defined regions or instances in the image. Compared to traditional superpixel methods, the masks generated by SAM more closely resemble real-world semantic instances and require no additional training.

[0049] In some preferred embodiments of this application, this step of generating a full-image mask for the target domain image using a pre-trained segmentation model may include: performing inference on all unlabeled images in the target domain using the pre-trained segmentation model, calling its full-image segmentation interface, and generating a high-quality mask set covering the entire image. The number of masks for each image is typically in the tens to hundreds, ensuring that the main semantic instances in the image are covered.

[0050] S200: Sample the images of the target domain proportionally, perform initial annotation on the full-image mask of each sampled image to obtain the initial target domain annotation dataset. The annotation is to assign a semantic category to the full-image mask region generated for each image.

[0051] Existing image-level sampling requires labeling the entire image, which is costly. While pixel-level and region-level sampling reduce the amount of labeling, they are complex to operate and difficult to accurately locate manually. Superpixel methods suffer from semantic confusion and blurred boundaries, and require additional algorithms to divide regions. To overcome these problems, this embodiment uses a full-image mask generated by a segmentation model, and then samples proportionally. Simultaneously, a semantic category is assigned to the full-image mask region generated for each image during labeling. This allows for obtaining semantically complete and structurally clear initial labeled data with lower human cost, solving the technical problems of existing semantic segmentation training relying on large-scale pixel-level manual labeling, resulting in low labeling efficiency and high cost.

[0052] This step employs a low-cost strategy of "mask-level global classification." For example, approximately 5% of the images are randomly sampled from the target domain, and all mask regions generated in each image are manually labeled. This labeling method only requires assigning a semantic category to each mask as a whole, thus greatly reducing the workload of pixel-by-pixel labeling. After labeling is completed, an initial target domain labeled dataset is formed, providing supervision signals for the semi-supervised learning module.

[0053] S300 trains a semantic segmentation model using semi-supervised learning, based on an initial target domain labeled dataset, source domain labeled data, and unlabeled target domain data. In the training iterations of semi-supervised learning, active learning is triggered according to a preset cycle. In each round of active learning, the segmentation model is used to generate a full-image mask, sample, and manually label the unlabeled target domain images. The new labeled mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is then used for further training of semi-supervised learning.

[0054] In this step, based on the S100 Segmentation All Model (SAM) described above, using the mask regions generated by the Segmentation All Model (SAM) as candidate units in active learning has inherent advantages. On the one hand, each mask region has strong semantic consistency and can be directly used as a labeling unit; only one label needs to be assigned to the entire mask to complete the labeling, greatly reducing the workload. On the other hand, as an intermediate-granularity information unit, the mask naturally balances the needs of image-level structure and pixel-level accuracy, providing a more efficient and practical foundation for sample selection and label propagation in active learning.

[0055] The embodiments described above employ a phased training mechanism that combines semi-supervised learning and active learning. This mechanism can fully exploit the information value of unlabeled samples in the target domain under a limited labeling budget, dynamically guide the model to learn key regions, and solve the technical problems of waste of supervision resources and insufficient model adaptability caused by random sampling of samples in existing methods.

[0056] As is well known, semi-supervised learning is a machine learning method that lies between supervised and unsupervised learning. It utilizes a small amount of labeled data and a large amount of unlabeled data for model training. In semantic segmentation tasks, the goal is to classify each pixel in an image and identify its category. This method is particularly suitable for situations where it is difficult to obtain large-scale, high-quality labeled data. Specifically, an initial model is first trained using existing labeled data. Then, a large amount of unlabeled data is used to mine the internal structure or distribution information of the data, and a consistency strategy is used to make the model invariant to small perturbations in the input. The trained model is then used to predict the unlabeled data, and these predictions are used as pseudo-labels. Finally, the model is retrained by combining the original labeled data and these pseudo-labeled data.

[0057] In some embodiments of this application, in step S300, under limited target domain labeled data (i.e., the aforementioned initial target domain labeled dataset), a semi-supervised semantic segmentation model is trained by combining source domain and unlabeled target domain samples to improve the model's generalization ability to the target domain.

[0058] Specifically, a semi-supervised learning model is used to train the semantic segmentation model, which may include: generating prediction results and extracting pseudo-labels using the current semantic segmentation model for unlabeled data in the target domain; performing consistency training using a strong-weak enhancement consistency strategy, which applies different image enhancements and feature perturbations to the same image and encourages the segmentation model to maintain consistent prediction results under different perturbations; and using supervised cross-entropy loss and consistency loss as loss functions for semi-supervised learning training, where supervised cross-entropy loss is used for labeled images and consistency loss is used for pseudo-labeled images.

[0059] Reference Figure 3 As shown, to better understand the above training process, in a specific embodiment, it can be performed as follows:

[0060] S301, Initial Data Construction

[0061] Source domain labeled data is denoted as in Represents an RGB image. Represents the truth value of semantic segmentation;

[0062] The target domain annotation data consists of a small proportion of randomly selected initial annotations. in For a small number of randomly selected target domain RGB images, These are the semantic segmentation ground truth values ​​obtained through manual annotation.

[0063] Unlabeled data in the target domain

[0064] Regarding source domain labeled data, the semantic categories contained in the ground truth values ​​in the source domain dataset need to be consistent with the semantic categories that the target domain wants to segment. For example, for autonomous driving, the source domain labeled data can come from a driving simulator or from an open-source dataset.

[0065] In step 301, the source domain labeled data is initialized, and a small number of samples of a certain proportion are randomly selected from the target domain for manual labeling. The remaining unlabeled samples in the target domain are denoted as

[0066] S302, Pseudo-label generation and consistency training

[0067] Unlabeled data in the target domain Generate prediction results using the current model and extract pseudo-labels. A strong-weak enhancement consistency strategy is adopted, that is, for the same image Applying image enhancement techniques (such as CutMix, color dithering, Gaussian blur, etc.) and feature perturbation yields the following results: and It also encourages segmentation models to maintain consistent prediction results under different perturbations.

[0068] The confidence level of pseudo-labels is filtered by a threshold, and pixels with low confidence are not used for training to reduce pseudo-label noise.

[0069] S303, Loss Function Construction

[0070] We combine supervised cross-entropy loss (for labeled images) and consistency loss (for pseudo-labeled images). The final total loss function L total as follows:

[0071] L total =L sup +λ·L unsup

[0072] Where λ is the balance coefficient for consistency loss. L sup The supervised cross-entropy loss is the training loss applied to both the source domain labeled data and the target domain labeled data. unsup Consistency loss is the training loss applied to the unlabeled data in the target domain.

[0073]

[0074] Where N represents the number of pixels in the image, i and j represent the pixel and channel indices, respectively, C represents the number of categories, and p represents the semantic segmentation model's response to the input image. The prediction results These are extracted pseudo-tags. and It is the preprocessed, unlabeled image of the target domain. This is to initialize the target domain labeled dataset pairs obtained in S301 and subsequent active learning annotations.

[0075] Reference Figure 3 As shown, in some embodiments of this application, active learning is introduced into the training iteration of semi-supervised learning. When the training iteration reaches the specified active learning activation period, sample value evaluation and a new round of labeling are triggered.

[0076] Specifically, each round of active learning includes the following process:

[0077] S304, Model Inference and Pseudo-Label Generation

[0078] The current trained model is used to predict all unlabeled target images, generating a semantic category and prediction confidence score for each pixel as a pseudo-label.

[0079] S305, Target Domain Re-segmentation

[0080] Using target domain pseudo-labels as clues, the segmentation model is invoked to generate a mask for the target image. Since the model has been preliminarily adapted to the target domain, the mask generated at this stage is more accurate in terms of semantic boundaries and structural coverage.

[0081] S306, Mask-level Information Content Evaluation

[0082] For each generated mask region, the Entropy-IoU-Sum (EIS) metric is calculated based on its prediction results. This metric comprehensively considers class uncertainty (information entropy) and the current class IoU performance to measure the learning value of the mask. A higher mask score indicates that the region is more beneficial for model training.

[0083] The Entropy-IoU-Sum (EIS) metric is calculated as follows:

[0084]

[0085] Where N represents the number of pixels in the image, i and j represent the pixel and channel indices respectively, C represents the number of categories, p represents the prediction result of the semantic segmentation model for the target image, and IoU(p i ) represents the IoU value of the predicted category at pixel i by the semantic segmentation model.

[0086] S307, Mask Sampling and Manual Annotation

[0087] Sampling was performed on the masked areas according to the scores from high to low, and manual annotation was carried out at a fixed ratio.

[0088] For example, a fixed percentage (e.g., 2-5%) can be manually labeled. The labeling method still adopts the low-cost strategy of "mask-level overall classification" in S100. After labeling, pixels within the mask will share the same semantic label.

[0089] S308, Update the target domain annotation set

[0090] The newly labeled image and mask region are added to the target domain label data, and the corresponding image or mask is removed from the target domain label data to complete this round of active learning update. The target domain label data is the sum of the initial target domain label dataset and the target domain dataset obtained by subsequent active learning annotation.

[0091] Specifically, referring to S301-S307 above, add the newly annotated image and mask area. and from Remove the corresponding image or mask from the image to complete this round of active learning and updating.

[0092] Reference Figure 3As shown, in order to alleviate the problem of class imbalance, in some embodiments of this application, after each round of active learning, an IoU-based class weighting mechanism is adopted to assign training weights to each class, which is used to dynamically adjust the uneven distribution of classes and improve the learning ability of small sample classes.

[0093] Specifically, in a preferred embodiment, the class-weighted mechanism based on IoU may include the following steps:

[0094] S401, Category IoU Statistics

[0095] On all currently labeled data in the target domain, calculate the IoU value for each category, which represents the current model's ability to recognize that category;

[0096] In this step, all labeled data includes both image-level and mask-level annotations.

[0097] S402, Weighting function design;

[0098] Specifically, the category weights are calculated using the reciprocal of the IoU:

[0099] w c =(1-IoU) c )·(u-1)+1

[0100] Where w c This represents the class weights (c indicates the class number), and u is the maximum weight limit set (e.g., 2) to ensure training stability. IoU c It is the IoU value of category c calculated in S401.

[0101] S403, Weighted Loss Function

[0102] By applying class weights to the prediction of each pixel's class in the source domain supervision loss (supervised cross-entropy loss) and the target domain consistency loss, the model's attention to low-frequency classes is further enhanced.

[0103]

[0104] In this step, the weighted loss function means that, based on the supervised cross-entropy loss (for labeled images) and consistency loss (for pseudo-labeled images) mentioned above, a class weight w is added to these two losses. j In the formula, the subscript j represents the index of the category.

[0105] In the preferred embodiments described above, an IoU class weight mechanism based on dynamic estimation of prediction performance is adopted. This mechanism can dynamically adjust the uneven distribution of categories, improve the model's learning ability for small sample categories and the final segmentation accuracy, and solve the technical problem that the model tends to ignore low-frequency categories and cause prediction bias under traditional average weight training.

[0106] The above steps S302-S403 will be executed continuously as a closed loop until the cumulative sampling and labeling ratio of active learning reaches a preset threshold (such as 25%, 50%, etc.). At this point, the loop stops, active learning and IoU weight updates are no longer performed, and only semi-supervised training continues until the number of training epochs reaches the maximum value (Max Epoch).

[0107] Among them, the cumulative sampling and labeling ratio reaches a preset threshold, for example, the preset threshold is 25%. The labeling is carried out in five rounds of active learning. In each round, 5% of the data is selected and labeled, and the five rounds are accumulated to reach the final 25%.

[0108] The method described in the above embodiments of this application fully combines the structured knowledge of the basic segmentation model, a small amount of manual annotation of the target domain, and a mask-level active learning mechanism. While reducing manual costs, it effectively improves the model's adaptability to the target domain and the learning effect of long-tail categories.

[0109] Based on the same technical concept, in another embodiment of this application, an active domain adaptation system for image semantic segmentation based on a segmentation-everything model is also provided, comprising:

[0110] Candidate mask generation module: Generates a full-image mask for the target domain image using a pre-trained segmentation model;

[0111] Initialization annotation module: Samples the images of the target domain proportionally, performs initial annotation on the full-image mask of each sampled image, and obtains an initial target domain annotation dataset. The annotation assigns a semantic category to the full-image mask region generated for each image.

[0112] Semi-supervised training module: Based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data, a semantic segmentation model is trained through semi-supervised learning;

[0113] Active learning module: In the training iteration of the semi-supervised learning, active learning is triggered according to a preset period. In each round of active learning, the segmentation model is used to generate a full-image mask, sample and manually annotate the unlabeled image in the target domain. The new annotated mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is used for the continued training of the semi-supervised learning.

[0114] In the above embodiments of the active domain adaptation system for image semantic segmentation based on the segmentation of everything model, the specific implementation techniques of each module can be referred to the corresponding steps in the active domain adaptation method for image semantic segmentation based on the segmentation of everything model, and will not be repeated here.

[0115] Based on the same technical concept, in another embodiment of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, is used to perform the active domain adaptation method for image semantic segmentation based on the segmentation-everything model of any of the above embodiments.

[0116] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0117] The following examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0118] Application Example 1: Mask generation and initial annotation methods during the initialization phase

[0119] In autonomous driving image semantic segmentation tasks, given the lack of semantic segmentation annotation data for target city A, the following steps can be taken:

[0120] First, 1000 street view images were collected as the unlabeled sample set for the target domain. By calling the full-image segmentation interface provided by the Segment All Model (SAM), each image was processed offline to generate approximately 150-200 masks covering the semantic regions of the image.

[0121] Subsequently, 5% (50 images) of the 1000 images were randomly selected as the initial annotation set. Annotators only need to assign semantic category labels (such as "lane lines," "vehicles," "pedestrians," "buildings," etc.) to the mask regions generated for each image, without needing to select each pixel, greatly reducing the burden of manual annotation. This method was used to construct the initial annotation dataset for the target domain, supporting the subsequent semi-supervised training process.

[0122] This embodiment is applied to the semantic segmentation task of autonomous driving images, which can achieve low annotation cost.

[0123] Application Example 2: The Integration Process of Model Training and Active Learning

[0124] After initial annotation, the semi-supervised training phase begins. The training model uses a lightweight real-time semantic segmentation network (such as BiSeNet), with inputs including:

[0125] Source domain annotation data derived from simulators (such as GTA5);

[0126] The target domain annotation data obtained in the first step;

[0127] The remaining 95% of the target area was not labeled with images.

[0128] The model is trained using a pseudo-label consistency regularization method, and active learning sampling is triggered every 20 epochs (for a total of 3 rounds). In each round of active learning, the model first predicts the unlabeled image, generates pseudo-labels as prompts, and then calls SAM again to perform mask segmentation on the target image to obtain a new round of candidate masks.

[0129] Subsequently, the EIS score of each mask is calculated, and several masks with the highest scores are selected as candidate regions for annotation, which then enter the manual annotation process. The newly annotated masks are added to the training set. This process continues to iterate until the total number of annotations is controlled (e.g., the target annotation ratio is 25%), thereby achieving a gradual improvement in model performance.

[0130] Application Example 3: Specific Application of IoU Dynamic Weighting Mechanism in Model Training

[0131] After active learning of annotations is completed, the training set already contains target domain samples with masked annotations. To alleviate the class imbalance problem, IoU statistics are performed on all classes in the current labeled data, for example:

[0132] The IoU for the category "Buildings" is 0.85;

[0133] The IoU for the category "Traffic Signs" is 0.52;

[0134] The IoU for the category "electric poles" is 0.39;

[0135] The IoU for the category "buses" is 0.28.

[0136] The training weights for each class are calculated based on the IoU value, with the weight range set to [1, 2]. For example, the weight for the bus class will be 2. These new class weights will be used in the weighted cross-entropy loss and consistency function to guide the model to pay more attention to classes with low IoU and improve accuracy.

[0137] In the specific embodiments described above, the overall technical objective of constructing an active learning training loop with low annotation cost and high structural consistency, and significantly improving the adaptability and generalization performance of the semantic segmentation model in the target domain, can be achieved. This overcomes the prominent problems in the prior art, such as high annotation burden, unsuitable active learning granularity, and class imbalance affecting accuracy.

[0138] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An active domain adaptation method for image semantic segmentation based on a segmentation-all model, characterized in that, include: A pre-trained all-segmentation model is used to generate a full-image mask for the target domain image; The images of the target domain are sampled proportionally, and the full-image mask of each sampled image is initially labeled to obtain an initial target domain labeled dataset. The label is to assign a semantic category to the full-image mask region generated for each image. Based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data, a semantic segmentation model is trained through semi-supervised learning. In the training iteration of the semi-supervised learning, active learning is triggered according to a preset period. In each round of active learning, the segmentation model is used to generate a full-image mask, sample and manually annotate the unlabeled image in the target domain. The new annotated mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is used for the continued training of the semi-supervised learning.

2. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 1, characterized in that, The method of generating a full-image mask for the target domain image using a pre-trained segmentation model includes: For all unlabeled images in the target domain, a pre-trained all-image segmentation model is used for inference, and its full-image segmentation interface is called to generate a high-quality mask set covering the entire image. The number of masks for each image ensures that the main semantic instances in the image are covered.

3. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 1, characterized in that, The step of training a semantic segmentation model through semi-supervised learning based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data includes: For unlabeled data in the target domain, the semantic segmentation model described above is used to generate prediction results and extract pseudo-labels. A strong-weak enhancement consistency strategy is adopted for consistency training. The strong-weak enhancement consistency strategy applies different image enhancements and feature perturbations to the same image and encourages the semantic segmentation model to maintain consistent prediction results under different perturbations. Supervised cross-entropy loss and consistency loss are used as loss functions for the semi-supervised learning training, wherein the supervised cross-entropy loss is used for labeled images and the consistency loss is used for pseudo-labeled images.

4. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 3, characterized in that, The confidence level of the pseudo-labels is filtered by a threshold, and pixels with a confidence level higher than the threshold are used for training.

5. The active domain adaptation method for image semantic segmentation based on the segmentation of everything model according to claim 1, characterized in that, Each round of active learning includes the following process: Use the current trained model to predict all unlabeled target images and generate the semantic category and prediction confidence of each pixel as pseudo-labels; Using the target domain pseudo-label as a cue, the segmentation model is invoked to generate a mask for the target image; For each generated mask region, the EIS index is calculated based on its prediction results. Taking into account both class uncertainty and current class IoU performance, the EIS index is used to measure the learning value of the mask. The higher the mask score, the more beneficial the mask region is to model training. Based on the scores from high to low, the mask area is sampled, and a fixed proportion is manually labeled. The newly labeled image and mask region are added to the target domain label data, and the corresponding image or mask is removed from the target domain label data to complete this round of active learning update. The target domain label data is the sum of the initial target domain label dataset and the target domain dataset obtained by subsequent active learning annotation.

6. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 5, characterized in that, Following each round of active learning, the following is also included: An IoU-based class weighting mechanism is adopted to assign training weights to each class, which is used to dynamically adjust the uneven distribution among classes.

7. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 6, characterized in that, The class-weighted mechanism based on IoU includes: On all currently labeled data in the target domain, calculate the IoU value for each category, which represents the current model's ability to recognize that category; The category weights are calculated using the inverse of IoU. The class weights are applied to the class prediction of each pixel in the source domain supervision loss and the target domain pseudo-label loss.

8. The image semantic segmentation active domain adaptation method based on the segmentation of everything model according to claim 6, characterized in that, Each round of active learning is executed continuously as a closed-loop cycle until the cumulative sampling and labeling ratio of active learning reaches a preset threshold. At this point, active learning and IoU weight updates are no longer performed, and only semi-supervised learning training continues until the number of training rounds reaches the maximum value.

9. An active domain adaptation system for image semantic segmentation based on a segmentation-everything model, characterized in that, include: Candidate mask generation module: Generates a full-image mask for the target domain image using a pre-trained segmentation model; Initialization annotation module: Samples the images of the target domain proportionally, performs initial annotation on the full-image mask of each sampled image, and obtains an initial target domain annotation dataset. The annotation assigns a semantic category to the full-image mask region generated for each image. Semi-supervised training module: Based on the initial target domain labeled dataset, source domain labeled data, and target domain unlabeled data, a semantic segmentation model is trained through semi-supervised learning; Active learning module: In the training iteration of the semi-supervised learning, active learning is triggered according to a preset period. In each round of active learning, the segmentation model is used to generate a full-image mask, sample and manually annotate the unlabeled image in the target domain. The new annotated mask is added to the target domain labeled dataset of the previous round of active learning to obtain an updated target domain labeled dataset, which is used for the continued training of the semi-supervised learning.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, this program is used to perform the method described in any one of claims 1-8.

Images