Active domain adaptive semantic segmentation method, system and equipment and storage medium

Abstract

The invention discloses an active domain adaptive semantic segmentation method, system and device, and a storage medium, and is characterized in that selection labeling is carried out by taking a superpixel as a unit, different from an image level labeling method and a pixel level labeling method, the superpixel level labeling greatly improves the labeling efficiency by only distributing one semantic category for each superpixel; besides, different from an uncertainty-based selection labeling strategy of an existing scheme, the method focuses on difficult samples in a domain adaptation scene, proposes a selection strategy based on domain information amount, and selects superpixels most valuable for domain adaptation learning to label; by adopting the super-pixel level labeling method and the selection strategy based on the domain information amount, the labeling quality is improved while the labeling cost is greatly reduced, and the performance of domain adaptive semantic segmentation is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to an active domain adaptation semantic segmentation method, system, device and storage medium. Background Art

[0002] In recent years, deep learning has made remarkable progress in the field of computer vision. However, deep learning models usually require a large amount of high-quality annotated data for training. For pixel-level prediction tasks such as semantic segmentation, obtaining large-scale annotated datasets requires huge time and human resources, so manually annotating datasets becomes impractical.

[0003] To solve this problem, researchers have proposed an unsupervised domain adaptation learning method. This method aims to use the knowledge learned in the source domain with a large number of labeled samples to help the model learn in the target domain that is related to the source domain but lacks annotations. By reducing the difference between the domains, the performance of the model in the target domain is improved, thereby reducing the annotation cost in the target domain. Although the current unsupervised domain adaptation learning methods have achieved good results by reducing the difference between the source domain and the target domain, the domain adaptation problem has not been completely solved due to the lack of guidance from the target domain annotation information.

[0004] In order to further improve the performance of the model in the target domain, an active domain adaptation learning method is proposed. This method selects a small number of the most valuable target domain samples for annotation through active learning, thereby further guiding the domain adaptation learning of the model. By selecting annotated samples in a targeted manner, the model can better learn the characteristics of the target domain. This method can significantly improve the domain adaptation performance by introducing a small amount of annotation cost.

[0005] The commonly used active domain adaptation method is to select the most valuable target domain samples for image-level or pixel-level annotation based on the uncertainty of model prediction. In the Chinese invention patent "A method for well logging lithology identification based on active domain adaptation" with authorization announcement number CN111767674B, two differentiated neural networks are trained using source domain samples, and the prediction results of the two neural networks for each sample in the target sample set are obtained; based on the prediction results, the method selects samples with high uncertainty in the target domain samples for image-level annotation, and uses these annotated samples for target domain learning of the model. In the Chinese invention patent application "A cost-effective scene text detection method and system" with publication number CN114220086A, a scene text detection network with an entropy-aware global alignment module and a text area alignment module is proposed for pre-training to reduce differences between domains; active learning based on uncertainty measurement is performed using the pre-trained model, and several target domain samples with the highest uncertainty are selected for image-level annotation, and these annotated samples are used to fine-tune the pre-trained model to adapt to target domain learning. In the Chinese invention patent application "Remote sensing image classification method based on multi-classifier adversarial active transfer learning" with publication number CN115620160A, the adversarial transfer learning method is used to train the feature extractor and two classifiers; the active learning query strategy selects the target domain data that are inconsistently classified by multiple classifiers for image-level annotation, and uses it for the target domain learning of the model. In the Chinese invention patent application "Image classification method and system based on source domain-independent domain adaptation and active learning" with publication number CN116863186A, the source domain model is used to initialize the feature extractor and classifier of the target domain model, and the features of all samples are extracted; the features of all samples are clustered to obtain pseudo labels for all samples, and the value of neighbor uncertainty of each sample is calculated; the target domain samples are selected for image-level annotation according to the value of neighbor uncertainty. In the Chinese invention patent application "Domain Adaptive Interactive Semantic Segmentation Method, Device and Equipment Based on Active Learning" with publication number CN116758541A, the integration of the two tasks of human-computer interaction and semantic segmentation is accelerated by weak supervision; the model is trained multiple times using source domain data and annotation sampling strategies in the warm-up phase; the annotation sampling strategy is to use a look-before-question annotation sampling strategy to annotate the target domain samples at the pixel level (only a small number of the most valuable pixels are annotated for each image) based on a preset value evaluation function and a preset score threshold. In the Chinese invention patent application "Semi-supervised Domain Adaptive Semantic Segmentation Method Based on Dual-level Alignment Active Learning" with publication number CN117058373A, the model is pre-trained using source domain data; the features of the target domain samples are extracted using the pre-trained model, and the features are clustered; an equal number of difficult target domain samples with the highest entropy are selected from each cluster for image-level annotation, and these annotated data are used for adversarial and comparative learning, and the domain adaptation performance of the model is improved through dual-level alignment active learning.

[0006] However, in the above solutions for image-level or pixel-level annotation, both the annotation efficiency and the annotation quality need to be improved.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide an active domain adaptation semantic segmentation method, system, device and storage medium, which improves the annotation efficiency of active domain adaptation, while improving the quality of annotation and ensuring the performance of domain adaptation semantic segmentation.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] An active domain adaptation semantic segmentation method, comprising:

[0011] The entropy map of the target domain image is extracted through the domain adaptation model, and the superpixels of the target domain image are extracted through the superpixel extraction network. The average entropy of each superpixel is determined by combining the entropy map, and the superpixels are divided into two categories according to the size of the average entropy, one is called high uncertainty superpixel, and the other is called low uncertainty superpixel. For low uncertainty superpixels, the set conditions are used for fusion;

[0012] Use the superpixel annotations involved in the fusion to obtain the first-stage annotations, and use the first-stage annotations to train the target domain model to obtain the first-stage target domain model;

[0013] The features of high-uncertainty superpixels are extracted through the domain adaptation model and the first-stage target domain model respectively, and several high-uncertainty superpixels with the largest feature differences are selected for annotation to obtain the second-stage annotations. The second-stage annotations are then used to train the first-stage target domain model to obtain the second-stage target domain model.

[0014] The target domain image is semantically segmented through the target domain model in the second stage.

[0015] An active domain adaptation semantic segmentation system, comprising:

[0016] The low-uncertainty superpixel fusion module is used to extract the entropy map of the target domain image through the domain adaptation model, extract the superpixels of the target domain image through the superpixel extraction network, determine the average entropy of each superpixel in combination with the entropy map, and divide them into two categories according to the size of the average entropy, one is called high-uncertainty superpixels, and the other is called low-uncertainty superpixels; for low-uncertainty superpixels, set conditions are used for fusion;

[0017] The first-stage annotation and training module is used to obtain the first-stage annotations using the superpixel annotations involved in the fusion, and to train the target domain model using the first-stage annotations to obtain the first-stage target domain model;

[0018] The second-stage annotation and training module is used to extract the features of high-uncertainty superpixels through the domain adaptation model and the first-stage target domain model, and select several high-uncertainty superpixels with the largest feature differences for annotation to obtain the second-stage annotations, and then use the second-stage annotations to train the first-stage target domain model to obtain the second-stage target domain model;

[0019] The semantic segmentation module is used to perform semantic segmentation on the target domain image through the target domain model in the second stage.

[0020] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0022] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0023] It can be seen from the technical solution provided by the present invention that superpixels are used as units for selection and annotation. Unlike image-level and pixel-level annotation methods, superpixel-level annotation greatly improves the annotation efficiency by assigning only one semantic category to each superpixel. In addition, unlike the existing solution's selection and annotation strategy based on uncertainty, the present invention comprehensively considers the two aspects of the number of annotations and the difficulty of annotation, and proposes a selection strategy based on the amount of domain information to select the superpixels that are most valuable for domain adaptation learning for annotation. The most valuable here is mainly reflected in two aspects: first, the fused large-size superpixels have more annotated pixels, so their annotation value is higher, and the fused superpixels are selected to be annotated first; second, for small-size superpixels that are not involved in the fusion, the ones with the largest domain difference are selected for annotation, because samples with large domain differences are often difficult examples and need more annotation learning; in general, the selection strategy based on the amount of domain information is mainly reflected in the two aspects of the number and quality of pixels; by adopting the superpixel-level annotation method and the selection strategy based on the amount of domain information, the present invention greatly reduces the annotation cost while improving the quality of annotation, thereby ensuring the performance of domain adaptation semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 A flowchart of an active domain adaptation semantic segmentation method provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram comparing various annotation schemes for active domain adaptation provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of superpixel fusion provided in an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of an overall solution of active domain adaptation based on superpixel level annotation provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a model training process provided by an embodiment of the present invention;

[0030] Figure 6 An active domain adaptation semantic segmentation system provided by an embodiment of the present invention

[0031] Figure 7 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0033] First, the terms that may be used in this article are explained as follows:

[0034] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0035] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than those explicitly listed, except for conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in that clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0036] The AAAA method provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, the conditions are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers of the reagents or instruments used in the embodiments of the present invention are not specified, they are all conventional products that can be purchased commercially.

[0037] Embodiment 1

[0038] The embodiment of the present invention also provides an active domain adaptation semantic segmentation method, such as Figure 1 As shown, it mainly includes the following steps:

[0039] Step 1: Low uncertainty superpixel fusion.

[0040] In an embodiment of the present invention, an entropy map of a target domain image is extracted through a domain adaptation model, and superpixels of the target domain image are extracted through a superpixel extraction network. The average entropy of each superpixel is determined in combination with the entropy map, and the superpixels are divided into two categories according to the size of the average entropy, one of which is called high-uncertainty superpixels and the other is called low-uncertainty superpixels. For low-uncertainty superpixels, fusion is performed using set conditions.

[0041] In the embodiment of the present invention, the domain adaptation model is pre-trained using source domain annotated data and target domain unannotated data; and the superpixel extraction network is pre-trained using source domain annotated data.

[0042] In the embodiment of the present invention, superpixel division is expressed as:

[0043]

[0044] Among them, ent s represents the average entropy of superpixel s, τ is the threshold, LUSP represents low uncertainty superpixel, and HUSP represents high uncertainty superpixel.

[0045] In the embodiment of the present invention, the setting conditions for fusion are:

[0046] d JS (f(k)||f(n))<∈

[0047] Among them, d JS Represents Jensen-Shannon divergence. The symbol || is a commonly used symbol when calculating divergence, which means relative to . Here, it is recorded as the Jensen-Shannon divergence of f(k) and f(n). f(k) and f(n) are the average features of low uncertainty superpixel k and low uncertainty superpixel n, and ∈ is the threshold.

[0048] Step 2: First phase labeling and training.

[0049] In the embodiment of the present invention, all superpixels involved in the fusion are first labeled to obtain the first-stage labeling results. Since the number of superpixels after fusion is small but the size is large, a large number of labeled pixels can be obtained with only a small amount of labeling cost. Then, the target domain model is trained using the labeled data of the first stage, and finally the target domain model of the first stage is obtained.

[0050] Step 3: Second stage labeling and training.

[0051] In an embodiment of the present invention, features of high-uncertainty superpixels that are not involved in fusion are extracted respectively through a domain adaptation model and a first-stage target domain model, and a number of high-uncertainty superpixels with the largest feature differences are selected for annotation according to an annotation budget to obtain second-stage annotations, and then the first-stage target domain model is trained using the second-stage annotations to obtain a second-stage target domain model.

[0052] In the embodiment of the present invention, the characteristic difference is calculated as follows:

[0053]

[0054] Among them, DomainGp represents the feature difference, Represents the average features of high-uncertainty superpixels m extracted by the target domain model and domain adaptation model in the first stage.

[0055] In an embodiment of the present invention, the number of high-uncertainty superpixels to be selected may be set according to actual conditions. For example, when the second-stage labeling budget is set to 30, 30 high-uncertainty superpixels with the largest feature differences may be selected for labeling.

[0056] Step 4: Semantic segmentation.

[0057] In the embodiment of the present invention, semantic segmentation is performed on the target domain image through the target domain model of the second stage.

[0058] The above solution provided by the embodiment of the present invention mainly has the following advantages:

[0059] 1) The present invention uses a superpixel-level annotation method to reduce the annotation cost, and further improves the annotation efficiency of active learning through a low-uncertainty superpixel fusion module.

[0060] 2) The present invention proposes a selection and annotation strategy based on the amount of domain information, and selects the most valuable samples for domain adaptation learning for annotation.

[0061] In general, the present invention utilizes the superpixel level annotation method to effectively improve the annotation efficiency of active domain adaptation, improve the quality of annotation, and ensure the performance of domain adaptation semantic segmentation.

[0062] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0063] 1. General overview of the plan.

[0064] The present invention proposes an active domain adaptation semantic segmentation method based on superpixel-level annotation, which selects annotations in units of superpixels. Unlike image-level and pixel-level annotation methods, superpixel-level annotation greatly improves annotation efficiency by assigning only one semantic category to each superpixel. In addition, unlike the uncertainty-based selection annotation strategy of the above-mentioned work, the present invention focuses on difficult samples in domain adaptation scenarios, and proposes a selection strategy based on domain information volume to select the superpixels that are most valuable for domain adaptation learning for annotation. By adopting a superpixel-level annotation method and a selection strategy based on domain information volume, the present invention greatly reduces the annotation cost while improving the quality of annotation.

[0065] The core innovations of the present invention can be summarized into three aspects: 1) A superpixel fusion scheme based on uncertainty is proposed, which further reduces the labeling cost through superpixel fusion, identifies easily confused superpixels through uncertainty, and excludes them from the fusion process to ensure the fusion quality; 2) A two-stage selection and labeling strategy based on domain information is designed. In the first stage, superpixels participating in the fusion with large information are selected for labeling; in the second stage, several superpixels that are not involved in the fusion and have the largest domain differences are selected for labeling; by selecting samples with the largest amount of domain information for labeling, efficient active learning is achieved to maximize the labeling value; 3) A domain adaptation learning method suitable for superpixel-level labeling is proposed; the domain differences of source domain data are alleviated through cross-domain hybrid technology, and the label noise of superpixel-level labeling is alleviated through pseudo-label consistency technology.

[0066] 2. Detailed introduction of the plan.

[0067] The core idea of ​​the present invention is to use a superpixel-level annotation method to select and annotate in units of superpixels, thereby reducing the waste of annotation resources and improving annotation efficiency. To this end, the present invention first compares various annotation methods. Figure 2 The image-level annotation, pixel-level annotation and super-pixel-level annotation methods are compared. Figure 2 As shown in the figure, image-level annotation annotates every pixel of the entire image, resulting in a waste of annotation budget for redundant areas within the annotated object, thereby reducing the annotation efficiency. In contrast, pixel-level annotation uses pixels as units and only selects a small number of the most valuable pixels for annotation. However, the amount of information in pixel-level annotation is usually low, and more than 10,000 to 20,000 pixels need to be annotated for each image to provide sufficient training information, resulting in a high annotation cost. Superpixel-level annotation uses superpixels as units, and only a semantic category needs to be assigned to each superpixel to annotate all pixels within the superpixel, greatly improving the annotation efficiency. In order to further reduce the waste of annotation resources, the superpixel fusion method can be used to merge superpixels that may have the same semantic category together to improve the annotation efficiency.

[0068] Therefore, the present invention considers using a superpixel-level annotation method and further improving the annotation efficiency through superpixel fusion. However, simple fusion based on feature similarity is not suitable for domain adaptation scenarios. Figure 3 As shown in part (b) of the figure, superpixel fusion in domain adaptation scenarios may face the challenge of fusion errors. Due to the differences between the source domain and the target domain, features in the target domain may be confused between classes, such as the similar features of roads and sidewalks, and the similar features of buses and trains. Therefore, in domain adaptation scenarios, directly using fusion methods based on feature similarity may not be applicable. In order to solve this problem, the present invention proposes a superpixel fusion scheme based on uncertainty. Figure 3 As shown in part (c) of the paper, superpixels with easily confused features are first accurately identified and excluded during the fusion process. Only superpixels that are not easily confused participate in the fusion process, which can effectively solve the problem of superpixel fusion errors in domain adaptation scenarios.

[0069] Specifically, entropy maps are used to identify superpixels with confusing features. Figure 4 As shown in FIG. 1 , the principle of the solution of the present invention is demonstrated. When low uncertainty superpixels are fused, they are divided into high uncertainty superpixels (HUSP) and low uncertainty superpixels (LUSP) according to their average entropy:

[0070]

[0071] Among them, s is a basic superpixel, ent sis the average entropy of superpixel s, and τ is the threshold. HUSP remains unchanged and does not participate in the fusion, while only LUSP is fused based on feature similarity. The condition for the fusion of two low uncertainty superpixels k and n is:

[0072] d JS (f(k)||f(n))<∈

[0073] Among them, d JS represents Jensen-Shannon divergence, f(k) and f(n) are the average features of low uncertainty superpixel k and low uncertainty superpixel n, and ∈ is the threshold. After superpixel fusion, two-stage selection annotation is performed.

[0074] Exemplarily, τ∈[0.01,0.10] can be set, for example, τ=0.05; ∈∈[0.01,0.20] can be set, for example, ∈=0.10; of course, this is only an example, and the present invention does not limit the specific numerical value. In actual application, the user can set it according to the actual situation or experience.

[0075] Considering that the size of the merged low uncertainty superpixel (MLUSP) is much larger than that of the HUSP that is not involved in the fusion, the MLUSP with large information content is selected for annotation in the first stage, so as to obtain a large amount of annotation information with a very small annotation cost. After completing the first stage of selection annotation, use the annotation information A target domain model 1 is initially trained with the designed training scheme. This model uses the labeled target domain information to learn the characteristics and semantics of the target domain, providing a basis for the subsequent superpixel annotation selection. In the second stage, the remaining annotation budget is used to select the HUSP with the largest domain difference for annotation, so as to provide the most valuable annotation information for domain adaptation learning. Here, the prediction difference between the domain adaptation model biased towards the source domain and the target domain model 1 is used to evaluate the domain difference size of the HUSP:

[0076]

[0077] Among them, DomainGap represents the feature difference, It represents the average features of the high-uncertainty superpixel m extracted by the target domain model and the domain adaptation model in the first stage. Obviously, the greater the difference between the two models’ predictions, the greater the pixel domain difference (DomainGap) contained in the superpixel s.

[0078] The above two-stage annotation selection method effectively utilizes annotation resources and gives priority to superpixels with larger information content and larger domain differences for annotation.

[0079] Exemplarily, DeepLabv2 can be used as the image semantic segmentation model (ie, the target domain model), DACS can be used as the domain adaptation model to extract the entropy map, and SSN can be used as the superpixel network to extract superpixels for each target domain image.

[0080] Figure 4 A stands for Acquisition Function, which is the strategy for selecting annotations; Indicates the superpixels involved in fusion selected in the first stage; Indicates several superpixels with the largest domain differences selected in the second stage.

[0081] For model training, since superpixels are not completely accurate, superpixel-level annotations inevitably contain noise. In addition, the annotation information of the source domain will also bring about domain differences. Therefore, the present invention introduces label denoising and domain adaptation technology in model training. Figure 5 As shown in Figure 1, when using the first-stage annotations to train the target domain model and the second-stage annotations to train the first-stage target domain model, cross-domain mixing and pseudo-label consistency techniques are used. The cross-domain mixing technique refers to the label Y of the source domain data. s , randomly select the original image area corresponding to the set number of categories (for example, half of the categories) in the annotation, and paste it to the target domain image I t , thus obtaining the cross-domain mixed image I st The pseudo-label consistency technique is to extract the target domain pseudo-label According to cross-domain mixing, the source domain label Y s A set number of category regions are attached to the target domain pseudo-label to obtain a cross-domain mixed pseudo-label And use the data enhancement method A to force the enhanced A(I st ) is consistent with the prediction (pseudo-label) before enhancement; when the target domain model is trained using the first-stage annotations, When the target domain model of the first phase is annotated using the second phase, I t with A(I st ) are used as input data for the two-stage training.

[0082] Figure 5 The Network in it refers to the segmentation network, P is the prediction, I is the image, the subscript s is the source domain, t is the target domain, and st is the cross-domain mixture (source mix target); Figure 4 The model training of the target domain network in the two stages is in accordance with Figure 5 The training method shown mainly lies in the difference in labeled data.

[0083] Finally, the loss of the active domain adaptation learning method is:

[0084]

[0085] Among them, L total is the loss of the active domain adaptation learning method, L CE is the cross entropy loss, I t is the target domain image, I st is the mixed data of the source domain and the target domain (i.e., cross-domain mixed images), is the target domain superpixel-level active annotation (i.e., target domain pseudo-label), is a mixed label of the source domain label and the target domain pseudo label (i.e., a cross-domain mixed pseudo label), and A represents the st Data augmentation on .

[0086] Considering that the solution of using loss to update model parameters can be implemented in the conventional way, it will not be elaborated here.

[0087] 3. Example introduction of the solution.

[0088] This example introduces the process of the active domain adaptation semantic segmentation method based on a specific model, which mainly includes:

[0089] Step S1: Prepare the training data set with labeled source domain and the training set and test set of the target domain. For the training set images of the source domain and the target domain, ordinary random cropping is used online. After image processing, the image size is scaled to 512×1024, then cropped to 512×512, and then numerically normalized.

[0090] Step S2, using the Pytorch deep learning framework, specifically using DeepLabv2 based on ResNet101 as the segmentation network, using the domain adaptation classic method DACS model as the domain adaptation model, and using the superpixel classic network SSN as the superpixel extraction network.

[0091] Step S3: Use the source domain labeled data and the target domain unlabeled data to train the DACS model.

[0092] Step S4: Use source domain labeled data to train the SSN network.

[0093] Step S5: Use the trained DACS and SSN to extract the entropy map and superpixels of the target domain data respectively, and use the low-uncertainty superpixel fusion module to perform superpixel fusion.

[0094] Step S6: After fusion, the first stage selection annotation is performed, and the MLUSP participating in the fusion is selected for annotation to obtain the first stage annotation

[0095] Step S7: Use the first stage annotation Perform model training and obtain target domain model 1 (i.e., the target domain model of the first stage); specifically, use the loss provided in the previous article for training.

[0096] Step S8: Use the remaining annotation budget to perform the second stage selection annotation. According to the prediction difference between the trained domain adaptation model DACS and the target domain model 1, several HUSPs with the largest domain differences are selected for annotation to obtain the second stage annotation.

[0097] Step S9: Use the second stage annotation Perform model training and obtain target domain model 2 (i.e., the target domain model of the second stage); specifically, use the loss provided in the previous article for training.

[0098] Step S10: input the test data set and calculate the segmentation accuracy of the target domain model 2.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0100] Embodiment 2

[0101] The present invention also provides an active domain adaptation semantic segmentation system, which is mainly used to implement the method provided in the above embodiment, such as Figure 6 As shown, the system mainly includes:

[0102] The low-uncertainty superpixel fusion module is used to extract the entropy map of the target domain image through the domain adaptation model, extract the superpixels of the target domain image through the superpixel extraction network, determine the average entropy of each superpixel in combination with the entropy map, and divide them into two categories according to the size of the average entropy, one is called high-uncertainty superpixels, and the other is called low-uncertainty superpixels; for low-uncertainty superpixels, set conditions are used for fusion;

[0103] The first-stage annotation and training module is used to obtain the first-stage annotations using the superpixel annotations involved in the fusion, and to train the target domain model using the first-stage annotations to obtain the first-stage target domain model;

[0104] The second-stage annotation and training module is used to extract the features of high-uncertainty superpixels through the domain adaptation model and the first-stage target domain model, and select several high-uncertainty superpixels with the largest feature differences for annotation to obtain the second-stage annotations, and then use the second-stage annotations to train the first-stage target domain model to obtain the second-stage target domain model;

[0105] The semantic segmentation module is used to perform semantic segmentation on the target domain image through the target domain model in the second stage.

[0106] Considering that the main technical details involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0107] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0108] Embodiment 3

[0109] The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided by the aforementioned embodiments.

[0110] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0111] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0112] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0113] The output device may be a display terminal;

[0114] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0115] Embodiment 4

[0116] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0117] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0118] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An active domain adaptation semantic segmentation method, characterized in that: include: The entropy map of the target domain image is extracted through the domain adaptation model, and the superpixels of the target domain image are extracted through the superpixel extraction network. The average entropy of each superpixel is determined by combining the entropy map, and the superpixels are divided into two categories according to the size of the average entropy, one is called high uncertainty superpixel, and the other is called low uncertainty superpixel. For low uncertainty superpixels, the set conditions are used for fusion; Use the superpixel annotations involved in the fusion to obtain the first-stage annotations, and use the first-stage annotations to train the target domain model to obtain the first-stage target domain model; The features of high-uncertainty superpixels are extracted through the domain adaptation model and the first-stage target domain model respectively, and several high-uncertainty superpixels with the largest feature differences are selected for annotation to obtain the second-stage annotations. The second-stage annotations are then used to train the first-stage target domain model to obtain the second-stage target domain model. The target domain image is semantically segmented through the target domain model in the second stage.

2. The active domain adaptation semantic segmentation method according to claim 1, characterized in that: The domain adaptation model is pre-trained using source domain annotated data and target domain unannotated data; the superpixel extraction network is pre-trained using source domain annotated data.

3. The active domain adaptation semantic segmentation method according to claim 1, characterized in that: The combined entropy map determines the average entropy of each superpixel, and is divided into two categories according to the size of the average entropy: Among them, ent s represents the average entropy of superpixel s, τ is the threshold, LUSP represents low uncertainty superpixel, and HUSP represents high uncertainty superpixel.

4. The active domain adaptation semantic segmentation method according to claim 1, characterized in that: The setting conditions of the fusion are: d JS (f(k)||f(n))<∈ Among them, d JS represents Jensen-Shannon divergence, f(k) and f(n) are the average features of low uncertainty superpixel k and low uncertainty superpixel n, and ∈ is the threshold.

5. The active domain adaptation semantic segmentation method according to claim 1, characterized in that: The feature difference is calculated as: Among them, DomainGap represents the feature difference, Represents the average features of high-uncertainty superpixels m extracted by the target domain model and domain adaptation model in the first stage.

6. The active domain adaptation semantic segmentation method according to claim 1, characterized in that: Use cross-domain mixing and pseudo-label consistency techniques when training the target domain model with the first-stage annotations and the first-stage target domain model with the second-stage annotations; Among them, cross-domain hybrid technology refers to the label Y of the source domain data s , randomly select the original image area corresponding to the set number of categories in the annotation and paste it to the target domain image I t , get the cross-domain mixed image I st ; Pseudo-label consistency technology is to extract the target domain pseudo-label According to cross-domain mixing, the source domain label Y s A set number of category regions are attached to the target domain pseudo-label to obtain a cross-domain mixed pseudo-label And use the data enhancement method A to make the enhanced A(I st ) is consistent with the prediction before enhancement; when the target domain model is trained using the first stage annotations, When the target domain model of the first phase is annotated using the second phase, I t with A(I st ) are used as input data for the two-stage training.

7. The active domain adaptation semantic segmentation method according to claim 6, characterized in that: The active domain adaptation loss during training is: Among them, L total is the active domain adaptation loss, L CE is the cross entropy loss.

8. An active domain adaptation semantic segmentation system, characterized in that: include: The low-uncertainty superpixel fusion module is used to extract the entropy map of the target domain image through the domain adaptation model, extract the superpixels of the target domain image through the superpixel extraction network, determine the average entropy of each superpixel in combination with the entropy map, and divide them into two categories according to the size of the average entropy, one is called high-uncertainty superpixels, and the other is called low-uncertainty superpixels; for low-uncertainty superpixels, set conditions are used for fusion; The first-stage annotation and training module is used to obtain the first-stage annotations using the superpixel annotations involved in the fusion, and to train the target domain model using the first-stage annotations to obtain the first-stage target domain model; The second-stage annotation and training module is used to extract the features of high-uncertainty superpixels through the domain adaptation model and the first-stage target domain model, and select several high-uncertainty superpixels with the largest feature differences for annotation to obtain the second-stage annotations, and then use the second-stage annotations to train the first-stage target domain model to obtain the second-stage target domain model; The semantic segmentation module is used to perform semantic segmentation on the target domain image through the target domain model in the second stage.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images