Domain adaptation interactive semantic segmentation method, device and equipment based on active learning

Abstract

The present application relates to a domain-adaptive interactive semantic segmentation method, device and equipment based on active learning. The method integrates interactive segmentation with semantic segmentation to construct domain-adaptive interactive semantic segmentation. This model can not only be used as a labeling tool to improve the labeling efficiency of experts, but also make it possible to correct the prediction mask of the model; the labeling / sampling strategy that integrates interactive sampling and active learning enables both obviously erroneous pixels with high model confidence and pixels that the model is unsure of to be labeled / sampled, making more efficient use of limited labeling resources; constructs a click loss function to accelerate the fusion of human-computer interaction and semantic segmentation tasks in a weakly supervised manner; constructs an interactive loss function so that the model can correctly respond to human interaction; uses a small amount of labeled data in the target domain and an interactive semantic segmentation model to generate positive pseudo-labeling for the target domain, providing more supervisory signals for the target domain. This method can improve the semantic segmentation accuracy of images.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision semantic segmentation, and in particular to a domain adaptation interactive semantic segmentation method, apparatus, and device based on active learning. Background Art

[0002] Semantic segmentation is a crucial research and application area in computer vision. It automatically divides an image into distinct regions based on specific rules, labeling each pixel with a distinct category. With the rise of deep learning technology, semantic segmentation has achieved impressive results in various scenarios. However, deep learning-based semantic segmentation still faces difficulties and challenges. First, the superior performance of deep learning-based models relies heavily on large amounts of high-precision pixel-level annotated data. Acquiring this pixel-level annotated data is expensive and time-consuming, especially in highly specialized fields such as medical imaging diagnosis. Furthermore, in real-world applications, real data often comes from a different distribution than the training data, resulting in domain shift, which often results in unsatisfactory performance of deep learning-based models in real-world applications. Furthermore, deep learning-based semantic segmentation models are often uninterpretable and non-interpretable. This makes it difficult for humans to intervene and guide the model to correct basic errors that are obvious to humans, thus promoting improvement.

[0003] To address the challenges of obtaining labeled data and domain shift, domain-adaptive semantic segmentation has become a hot research topic. Domain-adaptive semantic segmentation processes data from two different distributions: a source domain dataset with abundant pixel-level annotations and a target domain dataset with limited annotations. Its goal is to leverage this abundantly labeled source and limitedly labeled target datasets to transfer models to the target domain. Mainstream domain-adaptive semantic segmentation methods include unsupervised (UAD), semi-supervised (SSDA), weakly supervised (WDA), and active learning (Learning Domain Adaptation). Unsupervised domain-adaptive semantic segmentation methods utilize only labeled source domain data and an unlabeled target domain dataset for training, achieving adaptation to the target domain by reducing the distributional discrepancy between the two domains. However, due to the lack of supervisory signals from labeled target domain data, the performance of these methods lags far behind supervised methods. Unlike unsupervised methods, semi-supervised domain-adaptive semantic segmentation methods incorporate a small amount of labeled target domain data into training, achieving superior performance. To reduce the need for pixel-level annotated data, weakly supervised domain adaptation semantic segmentation methods use image-level or click-level annotated data for the target domain. However, the annotation resources obtained by semi-supervised and weakly supervised methods consume manpower and material resources, which is not the most efficient for model learning. Therefore, active learning methods formulate reasonable value assessment functions, calculate the annotation value of different pixels in the image, and then annotate these pixels with high annotation value, thus achieving efficient model training with limited annotation costs. Because active learning methods provide targeted annotated data, they also have the most satisfactory domain adaptation performance.

[0004] Although existing domain adaptation semantic segmentation methods based on active learning use certain strategies to select high-value labeled objects, existing methods are still not efficient enough. On the one hand, the active learning labeling strategy requires experts to inform the category of the questioned pixels one by one, and can only label one pixel at a time, which is not efficient. On the other hand, due to the scarcity of labeled data in the target domain, neural networks may make high-confidence but obvious errors during domain adaptation. The value assessment function used by existing active learning methods generally treats pixels that the model is unsure of as high-value labeled pixels. These high-confidence erroneous pixels are difficult to be selected by existing methods, making it difficult to correct the model's obvious errors. Finally, existing methods still cannot intervene in the model's output, and the model's predictions are not under human control, resulting in image semantic segmentation accuracy that does not meet requirements. Summary of the Invention

[0005] Based on this, it is necessary to provide a domain-adaptive interactive semantic segmentation method, device and equipment based on active learning to address the above technical problems.

[0006] A domain adaptation interactive semantic segmentation method based on active learning, the method comprising:

[0007] A different domain training dataset is obtained, where the training dataset includes a source domain dataset with annotated masks and an unlabeled target domain dataset.

[0008] The training dataset is used to train the domain-adapted interactive semantic segmentation model to obtain a trained domain-adapted interactive semantic segmentation model; the training stage includes: a warm-up stage and a domain adaptation stage. In the warm-up stage, the domain-adapted interactive semantic segmentation model is trained for multiple rounds according to a source domain dataset with annotated masks and an annotation / sampling strategy; in the domain adaptation stage, the model trained in the warm-up stage is trained for multiple rounds according to the source domain dataset, the target domain dataset and the annotation / sampling strategy to obtain a trained domain-adapted interactive semantic segmentation model; the annotation / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question annotation / sampling strategy, and uses the obtained new annotations as the annotated pixels of the target domain dataset for supervised learning of the model, or inputs the obtained sampled interaction points into the model as interaction information.

[0009] The image to be segmented is input into the trained domain-adapted interactive semantic segmentation model to obtain the semantic segmentation result.

[0010] A domain-adaptive interactive semantic segmentation device based on active learning, comprising:

[0011] The training data acquisition module is used to obtain training data sets of different domains, wherein the training data sets include a source domain data set with annotated masks and an unannotated target domain data set.

[0012] A domain-adaptive interactive semantic segmentation model training module is used to train the domain-adaptive interactive semantic segmentation model using the training dataset to obtain a trained domain-adaptive interactive semantic segmentation model; the training stages include: a warm-up stage and a domain adaptation stage. In the warm-up stage, the domain-adaptive semantic segmentation model is trained for multiple rounds based on a source domain dataset with annotated masks and an annotation / sampling strategy; in the domain adaptation stage, the model trained in the warm-up stage is trained for multiple rounds based on the source domain dataset, the target domain dataset, and the annotation / sampling strategy to obtain a trained domain-adaptive interactive semantic segmentation model; the annotation / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question annotation / sampling strategy, using the obtained new annotations as the annotated pixels of the target domain dataset for supervised learning of the model, or inputting the obtained sampled interaction points into the model as interaction information.

[0013] The domain adaptation interactive semantic segmentation module is used to input the image to be segmented into the trained domain adaptation interactive semantic segmentation model to obtain the semantic segmentation result.

[0014] A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program:

[0015] The aforementioned domain-adaptive interactive semantic segmentation method, apparatus, and device based on active learning fuse interactive segmentation with semantic segmentation to construct a domain-adaptive interactive semantic segmentation model. This model not only improves expert annotation efficiency as a labeling tool but also enables correction of the model's predicted masks. Its look-before-question labeling / sampling strategy ensures that both clearly erroneous pixels with high model confidence and pixels about which the model is unsure are labeled / sampled, making more efficient use of limited labeling resources. A click-through loss function is constructed to accelerate the fusion of human-computer interaction and semantic segmentation tasks through weak supervision. An interactive loss function is constructed to enable the model to correctly respond to human interaction. This method utilizes a small amount of labeled data from the target domain and the interactive semantic segmentation model to generate positive and false labels for the target domain, providing more supervisory signals for the target domain. This method can improve the semantic segmentation accuracy of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flowchart of a domain adaptation interactive semantic segmentation method based on active learning in one embodiment;

[0017] Figure 2 A workflow for domain-adaptive interactive semantic segmentation in another embodiment;

[0018] Figure 3Schematic diagram of the model training strategy principle in one embodiment, where (a) is a training strategy for pre-adaptive semantic segmentation based on active learning, (b) is an iterative training strategy for interactive segmentation, and (c) is a training strategy for the pre-adaptive interactive semantic segmentation model based on active learning in this application.

[0019] Figure 4 The architecture of a domain-adaptive interactive semantic segmentation model in another embodiment;

[0020] Figure 5 Schematic diagram of the labeling strategy principle in another embodiment, where (a) is the labeling strategy for active learning, (b) is the sampling strategy for interactive models, and (c) is the labeling / sampling strategy that integrates active learning and interactive sampling;

[0021] Figure 6 1 is a structural block diagram of a domain adaptation interactive semantic segmentation device based on active learning in one embodiment;

[0022] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0024] In one embodiment, Figure 1 As shown, a domain adaptation interactive semantic segmentation method based on active learning is provided, which includes the following steps:

[0025] Step 100: Obtain training datasets in different domains. The training datasets include a source domain dataset with labeled masks and an unlabeled target domain dataset.

[0026] Specifically, the source domain dataset is the labeled source domain remote sensing image, and the target domain dataset is the target domain remote sensing image.

[0027] Step 102: The domain-adapted interactive semantic segmentation model is trained using the training dataset to obtain a trained domain-adapted interactive semantic segmentation model. The training phase includes a warm-up phase and a domain adaptation phase. In the warm-up phase, the domain-adapted interactive semantic segmentation model is trained for multiple rounds based on the source domain dataset with annotated masks and the annotation / sampling strategy. In the domain adaptation phase, the model trained in the warm-up phase is trained for multiple rounds based on the source domain dataset, the target domain dataset, and the annotation / sampling strategy to obtain a trained domain-adapted interactive semantic segmentation model. The annotation / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question annotation / sampling strategy. The obtained new annotations are used as the annotated pixels of the target domain dataset for supervised learning of the model, or the obtained sampled interaction points are input into the model as interaction information.

[0028] Specifically, domain-adaptive interactive semantic segmentation is an iterative process. The first iteration of interactive semantic segmentation is automatic semantic segmentation. The domain-adaptive interactive semantic segmentation model automatically generates an initial mask with semantic meaning after receiving the image. Subsequent iterations are interactive processes. The model needs to receive interactive points provided by the user or sampled by the computer to modify the initial mask. The workflow of domain-adaptive interactive semantic segmentation is as follows: Figure 2 shown.

[0029] The domain-adapted interactive semantic segmentation model is

[0030] Pred i =Model(image,mask,point) (1)

[0031] Among them, image is an image, Model is a domain-adapted interactive semantic segmentation model, and Pred i is the predicted mask of the i-th iteration; mask is the mask to be corrected, which can be the output Pred of the last iteration of the model i-1 , or it can be an existing mask with insufficient accuracy; point is an interaction point provided by the user or automatically sampled by the computer based on the error, which contains position and annotation information and is used to indicate the error of the mask to be corrected. Both point and mask belong to interaction information. In this paradigm, image input is required, while point and mask can be empty. When both point and mask are empty, formula (1) is equivalent to Pred = Model (image), which is consistent with the paradigm of automatic semantic segmentation; when point is not empty, the model will output a higher-precision predicted mask based on the position and annotation information of the interaction point. Similarly, when mask is not empty, the model outputs a higher-precision predicted mask based on the prior annotation information provided by the mask to be corrected.

[0032] A new training strategy is developed for the domain-adaptive interactive semantic segmentation model based on active learning. The basic principle is as follows Figure 3 As shown, (a) is the training strategy of pre-adaptive semantic segmentation based on active learning, (b) is the iterative training strategy of interactive segmentation and (c) is the training strategy of the pre-adaptive interactive semantic segmentation model based on active learning in this application. Figure 3 A circle in the figure represents a batch of samples, and the same color represents the same batch of samples. The red model represents the model performing parameter updates, and the silver-white model represents the model performing inference only without updating. The red dotted arrow represents the gradient backpropagation based on the labeled data of a specific domain for model update. The green arrow represents the new labeled data generated by the labeling strategy. The purple arrow represents the interaction points generated by the sampling strategy, which are input into the model for interaction. Each dotted box represents a round, that is, the process of updating the model using a batch of samples. The training process consists of multiple rounds.

[0033] The quality of the pixels selected by the value evaluation function in active learning depends on the performance of the model. The more mature the model, the higher the quality of the pixels selected. For this reason, the training process of the pre-adaptive semantic segmentation method of active learning usually includes a warm-up phase and a domain adaptation phase. Figure 3 As shown in (a), during the warm-up phase, only labeled data from the source domain is used to update the model, pre-conditioning the model's semantic understanding capabilities. During the domain adaptation phase, a defined value assessment function is used to query experts for high-value pixel annotations from the target domain, resulting in a small number of target domain annotations. During the domain adaptation phase, model updates are jointly supervised by labeled data from both the source and target domains.

[0034] In order to allow the model to adjust its prediction output according to the interaction points, the training of the interactive model is iterative. Figure 3 As shown in (b), each round consists of multiple iterations. Each iteration generates new interaction points based on the sampling strategy and feeds them into the model in the next iteration to correct the predicted mask. The number of iterations N in each round is random, and the model is updated only on the final iteration. This training strategy enables the model to learn information from varying numbers of interaction points at different locations while using relatively few computational resources.

[0035] The training strategy of the domain-adapted interactive semantic segmentation model based on active learning is shown in (c) in 3, which includes a warm-up phase and a domain adaptation phase, and each round is also a multiple-iteration process.

[0036] The warm-up phase not only leverages source domain annotated data to enhance the model's semantic understanding capabilities, but also enables the model to be interactive through multiple iterations. The first iteration of interactive semantic segmentation is automatic segmentation, and subsequent iterations are interactive. During training, model updates are required during both the first and last iterations. During training, interaction points are automatically generated by the computer using a sampling strategy based on the ground truth and predicted masks.

[0037] After the model has acquired a certain level of semantic understanding and interactive functionality, it enters the domain adaptation phase. In the domain adaptation phase, a small number (specific number is manually defined) of labeled pixels in the target domain are obtained through labeling strategies in certain rounds (defined manually), and supervised training is performed based on these labeled pixels. At the same time, these labeled pixels are also used as interactive point inputs for interactive semantic segmentation, and interactive semantic segmentation is performed. Predicted pixels with higher confidence are taken as positive pseudo-labels for the target domain data for semi-supervised training. The update of the model in one round of the domain adaptation phase involves both the first and last iterations of the source domain data and the target domain data.

[0038] Step 104: Input the image to be segmented into the trained domain-adapted interactive semantic segmentation model to obtain a semantic segmentation result.

[0039] The aforementioned domain-adaptive interactive semantic segmentation method based on active learning integrates interactive segmentation with semantic segmentation to construct a domain-adaptive interactive semantic segmentation model. This model not only improves expert annotation efficiency as a labeling tool but also enables correction of the model's predicted masks. Its look-before-question labeling / sampling strategy ensures that both clearly erroneous pixels with high model confidence and pixels about which the model is unsure are labeled / sampled, making more efficient use of limited labeling resources. A click-through loss function is constructed to accelerate the fusion of human-computer interaction and semantic segmentation tasks through weak supervision. An interactive loss function is constructed to enable the model to correctly respond to human interaction. This method leverages a small amount of labeled data from the target domain and the interactive semantic segmentation model to generate positive and false labels for the target domain, providing more supervisory signals for the target domain. This method can improve the semantic segmentation accuracy of images.

[0040] In one embodiment, Figure 4The architecture of the domain-adaptive interactive semantic segmentation model shown in step 102 includes: an interactive encoding module, an interactive information preprocessing module, an image information preprocessing module, a feature fusion module and a semantic segmentation network; the interactive encoding module is used to convert the interactive information into an interactive information feature map; the interactive information preprocessing module and the image information preprocessing module are used to adjust the size and number of channels of the interactive information feature map and the image information feature to be consistent; the feature fusion module is used to fuse the preprocessed interactive information features and image information features using an element-by-element addition operation to obtain a fused feature; the semantic segmentation network is used to perform interactive semantic segmentation based on the fused feature to obtain a semantic segmentation prediction result.

[0041] Specifically, the domain-adaptive interactive semantic segmentation model needs to be able to receive image information and additional interactive information. The architecture of the model is as follows: Figure 4 As shown. Before entering the semantic segmentation network, the interaction information (mask and point) needs to be converted into a graph through the interaction encoding module. Before entering the interaction encoding module, although the mask is a graph of dimension H×W×C (H, W are the length and width of the image (unit: pixel); C is the number of categories of semantic segmentation (unit: piece), but point is a vector of dimension N×3 (N is the total number of interaction points, unit: piece; the second dimension of 3 is used to indicate the horizontal coordinate (unit: pixel) and vertical coordinate (unit: pixel) of the interaction point in the image coordinate system, and the interaction point represents the category label c, c∈[1,C], indicating that the interaction point belongs to the cth category). The output of the interaction encoding module is an interaction information graph of dimension H×W×2C, and its working principle is Among them, encoding is a click encoding module, which transforms the point with dimension N×3 into a point feature map with dimension H×W×C. Common transformation methods include Gaussian transformation, distance transformation, disk transformation and dynamic disk transformation; Represents the splicing operation, which splices the mask and point feature map in the third dimension and outputs the interaction information map f with dimensions H×W×2C inter .

[0042] The image information and interaction information are preprocessed by the image information preprocessing module and the interaction information preprocessing module respectively, and the size (H and W) and number of channels (3 channels for the image and 2C channels for the interaction information graph) of the two information streams are adjusted to be consistent for subsequent information fusion. inter ) works as follows:

[0043] N(iamge)=MaxPool 3 / 2 (ReLu(BN(Conv 7,3→64 / 2(image))) (2)

[0044] M(f inter )=Conv 3,32→64 / 2 (ReLu(Conv 3,2C→32 / 2 (ReLu(Conv 3,2C→2C / 1 (f inter )))) (3)

[0046] Among them, MaxPool 3 / 2 Represents a 3×3 maximum pooling with a step size of 2; Conv 7,3→64 / 2 Represents a 7×7 convolution with 3 input channels, 64 output channels, and a stride of 2.

[0047] The preprocessed image information and interaction information need to be fused. This paper uses element-by-element addition for this fusion. When the interaction information is empty, that is, when performing automatic semantic segmentation, the interaction branch does not perform any operations, and no information fusion is required. When the interaction information is not empty, that is, when at least one of the mask and the point exists, information fusion is performed, and the empty interaction information is encoded as an all-zero feature map.

[0048] The semantic segmentation network converts the input fusion information into a prediction mask. The semantic segmentation network does not need to be designed from scratch, and any effective semantic segmentation network can be used.

[0049] In one embodiment, step 102 includes: setting the number of warm-up phase training rounds M1, the number of low-noise training rounds M′1, and the threshold fn_score_thres; setting the current warm-up training round to 1; extracting a batch of samples from the source domain dataset as input data and annotation masks for the current warm-up training round; setting a preset number of iterations N in one round of training; in the first iteration, inputting the input data into the domain-adapted interactive semantic segmentation model to obtain an initial prediction mask; calculating the semantic segmentation loss of the first iteration based on the annotation mask and the initial prediction mask; in the second to N-1th iterations, automatically generating a preset amount of interactive information using the annotation / sampling strategy based on the initial prediction mask and the annotation mask; when the current warm-up training round is less than the low-noise training round, setting the mask to None, Perform low-noise training; when the current warm-up training round is greater than or equal to the low-noise training round, continue training the model after the previous iteration according to the input data, the predicted mask obtained in the previous iteration, and the interactive information until the N-1th iteration is completed. Calculate the interactive semantic segmentation loss of the N-1th iteration based on the labeled mask and the predicted mask of the N-1th iteration, and take the sum of the semantic segmentation loss of the first iteration and the interactive semantic segmentation loss of the N-1th iteration as the total loss of the current round in the warm-up stage; and update the domain-adapted interactive semantic segmentation model according to the total loss of the current round in the warm-up stage; increase the current warm-up training round by 1, and perform the next round of warm-up training until the current warm-up training round is greater than the number of training rounds in the warm-up stage, and obtain the domain-adapted interactive semantic segmentation model after preheating training.

[0050] The specific training strategy pseudo code for the warm-up phase is as follows:

[0051] Input: Source domain dataset (X s ,Y s ), number of warm-up training rounds M1, low-noise training rounds M′1, threshold fn_score_thres;

[0052]

[0053] Output: model model

[0054] Step 1 means training M1 rounds, and step 2 x s,i represents the sample x taken from the source domain dataset in the i-th round. s,i,1 Represents the first iteration prediction of the source domain data sample in the i-th round. The loss function BCE in step 5 is seg The subscript seg represents automatic semantic segmentation, and CE represents the cross entropy loss function, which is calculated as follows:

[0055]

[0056] where y k,i,j,c and p k,i,j,c They represent the true value and predicted value of the k sample (i, j) pixel position for the c category.

[0057] The 6th line indicates N interactive iterations. In the early stages of training, the model's predicted mask quality is poor and noisy. To prevent the model from ignoring the interactive information, the mask is set to None in the first M′1 rounds and low-noise training is performed, as shown in lines 10 and 11. In addition, to force the model to pay attention to the interactive information, the click cross entropy loss CE is proposed. point , ensure that the model predicts the correct pixel at the point position and use the interactive semantic segmentation loss CE iseg Force the prediction results of interactive semantic segmentation to be close to the true mask, as shown in row 12.

[0058] In one embodiment, the input data is fed into the domain-adapted interactive semantic segmentation model, and the initial predicted mask is:

[0059] pred s,i,1 =SEgNet(N(x s,i )) (5)

[0060] Among them, pred s,i,1 is the initial prediction mask of the source domain data in the i-th round, x s,i is the source domain data input of the i-th round, SEGNet() is the semantic segmentation network, and N() is the image information preprocessing module.

[0061] In one embodiment, a preset amount of interactive information is automatically generated using a labeling / sampling strategy based on the initial mask. The labeling / sampling strategy in the step is: based on the predicted mask and the labeled mask, the proportion of incorrectly segmented areas in the predicted mask of each class is calculated as:

[0062] fn_score c =CalFn(pred c ,gt c )=((1-pred c )∩gt c ) / (pred c ∪gt c ) (6)

[0063] Among them, fn_score c is the proportion of incorrectly segmented areas in the predicted mask of class c, pred c is the predicted mask of the cth class, CalFn() is the calculation function of the proportion of incorrect segmentation areas in the predicted mask, gt c is the annotation mask of class c.

[0064] The proportion is compared with the preset threshold. When the proportion is greater than the preset threshold, the method of observation is used to mark / sample near the center of the error area with the largest area. When the proportion is less than the preset threshold, the method of questioning is used to select pixels with high marking value for marking / sampling according to the value evaluation function. The value evaluation function is:

[0065]

[0066] Among them, value i,j The pixel label value of coordinate (i, j); pred i,j,c is the predicted value of the cth class at coordinate (i, j) in the predicted mask, C is the set of all classes in the predicted mask, and CalVal() is the value evaluation function.

[0067] Specifically, when training a domain-adaptive semantic segmentation model based on active learning, it is necessary to formulate a labeling strategy so that the most valuable pixels in the target domain are labeled. When training an interactive model, it is necessary to input interaction points into the model to help it learn interaction information. Therefore, it is necessary to formulate a sampling strategy to allow the computer to automatically generate interaction points to achieve fully automatic interactive learning. This application proposes a new look-before-question labeling / sampling strategy that integrates active learning labeling and interactive sampling. Its basic principle is as follows: Figure 5 As shown in the figure, (a) is the active learning annotation strategy, (b) is the interactive model sampling strategy, and (c) is the annotation / sampling strategy that combines active learning and interactive sampling. The green box represents the predicted mask, and the red box represents the new annotation / sampling map generated based on the predicted mask.

[0068] There is no essential difference between active learning annotation and interactive sampling. Their common point is that new annotation points / interaction points are obtained through experts. The difference is that active learning uses the obtained new annotations as true annotations for supervised training of model prediction masks, while interactive sampling inputs interaction points into the model, and the model outputs a more accurate mask based on the error correction information it provides.

[0069] Active learning uses a "questioning" approach, using a value assessment function to determine the locations of pixels most worthy of labeling, and then asking experts for their category labels. Active learning often queries pixels about which the model is unsure, but obvious errors with high confidence levels go uncorrected. Interactive models use a "visualization" approach, where experts visually observe and label the corresponding pixels near the center of the largest error area with their corresponding category. This approach can correct obvious errors in model predictions, but because "visualization" requires experts to determine the locations of the labeled pixels, it can increase the burden on experts when the error areas in the prediction mask are small and highly accurate.

[0070] This application proposes a look-first, ask-later annotation / sampling strategy, which is used regardless of whether the model is provided with target domain annotation data or interaction points as input. The core idea of ​​the look-first, ask-later annotation / sampling strategy is: when there are obvious errors, let the experts make corrections to the corresponding areas through visual observation. Otherwise, ask the experts for high-value pixel annotations through the value evaluation function. During the training process, we use the score fn_score to determine the boundary between "look" and "ask", and its calculation expression is as follows:

[0071] fn_score c =CalFn(pred c ,gt c )=((1-pred c )∩gt c ) / (pred c ∪gt c ) (8)

[0072] Among them, pred c It represents the prediction mask of class c (after 01 binarization), gt c It represents the true mask of class c. c It represents the proportion of incorrect segmentation areas in the prediction mask of class c. Set a threshold fn_score_thres, when fn_score c When the value is greater than the threshold, the error of the predicted mask of category C is large and the prediction accuracy is not high. The “look” method is used to sample / mark near the center of the largest error area. When fn_score c When the value is less than the threshold, the accuracy of the predicted mask of category c is high. By "asking", pixels with high annotation value are selected for sampling / annotation based on the value evaluation function. The value evaluation function is defined based on information entropy, and its calculation expression is as follows

[0073]

[0074] In actual applications, the boundary between "looking" and "asking" during the annotation process is determined by experts. When the expert believes that there are no obvious errors in the predicted mask, he can click the "Ask" button, and the model will provide high-value pixels that need to be annotated through the value evaluation function.

[0075] In summary, the labeling / sampling strategy proposed in this application can be expressed as

[0076] y / point=G(mask pred , mask gt ,fn_score_thres,Num) (10)

[0077] Among them, the output is the annotation point y or the interaction point point, mask pred To predict the mask, mask gt is the real mask, fn_score_thres is the threshold, and Num is the number of labels / samples required. gt When not empty, the labeling / sampling position of the "look" method can be based on the mask gt With mask pred The difference is automatically determined, and the annotation / sampling point annotation of the "look" and "ask" strategies can be based on the mask gt In other words, mask gt The existence of can make the labeling / sampling automatic. gt When it is empty, the above information will be given by experts.

[0078] In one embodiment, in the domain adaptation stage, the model trained in the warm-up stage is trained for multiple rounds according to the source domain dataset, the target domain dataset, and the look-before-question annotation / sampling strategy to obtain a trained domain-adapted interactive semantic segmentation model, including: setting the number of training rounds M2 in the domain adaptation stage, annotating the round set [m1, m2, ..., m n ], number of annotations N p ; Use None to initialize the annotation mask of the target domain dataset; set the current domain adaptation training round to 1; extract a batch of samples from the source domain dataset and the target domain dataset respectively as the current source domain input dataset and annotation mask and the current target domain input dataset and annotation mask; set the preset number of iterations N1 in a round of domain adaptation training; in the first iteration: input the current source domain input data and the current target domain input data into the domain adaptation interactive semantic segmentation model to obtain the initial prediction mask of the source domain input data and the target domain input; if the annotation mask of the current target domain input data is not None, then input the annotated pixels therein as interactive information into the domain adaptation interactive semantic segmentation model to obtain the corrected mask;

[0079] Based on the annotation masks, initial prediction masks and correction masks of the source domain input data and the target domain input data, the total loss of the first iteration of the current round of the domain adaptation stage is determined; the domain adaptation interactive semantic segmentation model is interactively iterated N1-1 times according to the initial prediction mask of the source domain, the current source domain input dataset, and the annotation / sampling strategy; the loss value generated by the N1-1th interactive process is multiplied by the scaling factor β and added to the total loss of the first iteration to obtain the loss of the current domain adaptation training round; and the domain adaptation interactive semantic segmentation model is updated according to the loss of the current domain adaptation training round; if the current domain adaptation training round is an element of the annotation round set, a preset number of new target domain annotation pixels are generated through the annotation / sampling strategy, and the new annotation pixels are merged with the existing annotation pixels through a union operation to update the target domain annotation mask; the current domain adaptation training round is increased by 1, and the next round of domain adaptation training is performed until the current domain adaptation training round is greater than the number of training rounds in the domain adaptation stage, and a trained domain adaptation interactive semantic segmentation model is obtained.

[0080] In one embodiment, based on the labeled masks, initial predicted masks, and corrected masks of the source domain input data and the target domain input data, the total loss of the first iteration of the current round of the domain adaptation phase is determined as:

[0081] Loss TotaL1 =Loss1+Loss2 (11)

[0082]

[0083]

[0084] Among them, Loss TotaL1 is the total loss of the first iteration of the current round in the domain adaptation stage, Loss1 is the total loss of semantic segmentation, CE seg (pred s,i,1 ,y s,i ) is the semantic segmentation cross entropy loss of source domain data, CE seg (pred t,i,1 ,y t,i ) is the cross entropy loss of semantic segmentation in the target domain; Loss2 is the positive and negative pseudo-labeling loss, PL(pred t,i,1 , pred t,i,2 ) is the positive pseudo-label loss, NL(pred t,i,1 ) is the negative pseudo-labeling loss. pred s,i,1 is the initial prediction mask of the source domain, pred t,i,1 is the initial predicted mask of the target domain, y s,i is the annotation mask of the source domain data for the i-th iteration, y t，i is the annotation mask of the target domain data in the i-th iteration.

[0085] In one embodiment, the positive-false labeling loss is:

[0086]

[0087]

[0088] Among them, τ p is the threshold for determining the positive and false labels, p k,i,j,c represents the predicted value of the k-sample (i, j) pixel position for the c-category, N, H, W, C are the number of samples, sample size length, sample size width, and sample channel number, respectively, p′ k,i,j,c is the predicted value of the k-th sample (i, j) pixel position for the c-th class, p and p′ are the predicted values ​​of the initial mask and the final mask, respectively.

[0089] The negative pseudo-annotation loss is:

[0090]

[0091]

[0092] where τ n is the threshold for determining negative pseudo-annotations.

[0093] Specifically, the loss function in the domain adaptation stage is divided into three parts: the total semantic segmentation loss, the total interactive semantic segmentation loss, and the positive and negative pseudo-labeling loss.

[0094] (1) The total loss of semantic segmentation is used to supervise the initial prediction mask of the first iteration to improve the performance of semantic segmentation. The total loss of semantic segmentation includes the cross entropy loss CE of the source domain data seg (pred s,i,1 ,y s,i ) and the cross entropy loss CE of the target domain data seg (pred t,i,1 ,y t,i ), where the subscripts s and t refer to the source domain and target domain respectively, and the subscript seg represents semantic segmentation.

[0095] However, since the target domain does not necessarily have annotations, it is necessary to determine whether the annotation mask of the target domain is None, and then determine whether to calculate the cross entropy loss CE of the target domain data. seg (pred t，i，1 ,y t，i ).

[0096] (2) The total loss of interactive semantic segmentation is used to maintain the interactive function of the interactive semantic segmentation model. It has two functions: in terms of functional application, it allows the output of the model to be manually intervened, controlled and corrected. Through a few simple interactions, high-precision semantic masks can be quickly generated, which plays an important role in scenarios such as data annotation and image editing. In terms of domain adaptation, the labeled pixels of the target domain can be input into the interactive semantic segmentation model as interaction points to generate a higher-precision mask, and the pixels with higher confidence in the predicted mask are taken as additional labeled pixels, i.e., positive pseudo-annotations. The initial predicted mask of the target domain is supervised by the positive pseudo-annotation loss, thereby enhancing the performance of the domain-adapted interactive semantic segmentation model in the target domain. That is, users can enhance the network's automatic semantic segmentation performance by correcting errors in the network, thereby achieving fast and efficient domain adaptation.

[0097] The interactive semantic segmentation loss is the same as the semantic segmentation loss, and both use the cross entropy loss function. Since the target domain does not have a complete annotation mask, the interactive semantic segmentation loss is only calculated using the source domain data. In order to accelerate the fusion of the two tasks of human-computer interaction and semantic segmentation, a click loss function is constructed, CE point (pred s，i，N+1 , point). The click loss function uses the point of the input model as a supervision signal to supervise the predicted output of the last interaction iteration, forcing the model to pay attention to the interaction information of the input.

[0098] In order to enable the model to correctly respond to human interaction, an interactive semantic segmentation cross entropy loss is constructed to supervise the prediction output of the last iteration. The interactive semantic segmentation loss is: CE iseg (pred s，i，N+1 ,y s，i ), where the subscript iseg represents interactive semantic segmentation, y s，i is the annotation mask of the source domain data at the i-th iteration.

[0099] Since the interactive function of the training model is not the focus in the domain adaptation stage (it has been trained in the warm-up stage and only needs to be maintained in the domain adaptation stage), the loss function generated by the interaction process will be multiplied by a scaling factor β.

[0100] (3) The positive and negative pseudo-labeling losses are used to add supervisory signals to the target domain.

[0101] Positive and Pseudo-labeling: One of the functions of maintaining the interactive function of the interactive semantic segmentation model through source domain data is to generate more accurate prediction masks in the target domain through interaction. The predicted pixels that the model is more confident in are taken as the labeled pixels, that is, the positive and pseudo-labeling, and the loss function is calculated.

[0102] Negative pseudo-labels: Negative pseudo-labels can be computed without requiring interactive iteration. For some pixels in the initial target domain prediction mask, the network may be unsure of their class, but it can be certain that they are not. We can use the fact that a pixel is not of a certain class as a supervisory signal to monitor the initial target domain prediction mask.

[0103] Among them, "positive" refers to telling the model which category a pixel belongs to, "negative" refers to telling the model that a pixel is not that category, and "pseudo" is because the label is generated by the model prediction and is not always correct.

[0104] Specifically, the pseudo code of the training strategy in the domain adaptation stage is as follows:

[0105] Input: Source domain dataset (X s , Y s ), target domain dataset X t , the number of training rounds in the domain adaptation phase is M2, and the number of labeling rounds [m1, m2, ..., m n ], number of annotations N p ;

[0106]

[0107] Output: model model

[0108] Since there is no labeled data in the target domain data at the beginning, the first line initializes the true mask of all target domain data with None. The second line indicates training M2 rounds. The x in the third line s，i ,y s，i represents the sample x and the annotation mask y taken from the source domain data in the i-th round, and x t，i ,y t，i The 5th line generates the initial prediction mask of the source domain and the target domain, where the parameter pred s，i，1 Represents the first iteration prediction of the source domain data sample in the i-th round. The sixth line calculates the semantic segmentation loss CE of the source domain seg (pred s,i,1 ,y s,i ), since the negative pseudo-label loss function only needs the initial predicted mask of the target domain to be calculated, the negative pseudo-label loss function NL(pred t,i,1 ), where CE seg represents the cross entropy function for automatic semantic segmentation of source domain data, and NL represents the negative pseudo-annotation loss function. The negative pseudo-annotation loss function treats the predicted values ​​with low confidence in the predicted mask as negative pseudo-annotations, and uses the fact that the pixel at that location is not of this type as a supervisory signal to guide model updates.

[0109] Step 7 determines whether the target domain data is labeled as None. If not, go to step 8 and input the labeled pixels into the model as interaction points to obtain the corrected mask pred t，i，2 , the accuracy of the corrected mask is higher, and this is used to calculate the positive pseudo-label loss function PL (pred t,i,1 , pred t,i,2 ), used to supervise the initial prediction mask of the target domain. In step 9, the semantic segmentation loss CE of the target domain is also added seg (pred t,i,1 ,y t,i ).

[0110] Lines 10 to 12 are the interactive iterative process of the source domain data, which is used to maintain the interactive function of the model. Line 13 multiplies the loss function value generated by the interactive process by the scaling factor β (β≤1) and adds it to the total loss, maintaining the interactive function of the model through the click loss and the interactive semantic segmentation cross entropy loss. Lines 15 to 16 indicate that the annotation / sampling strategy needs to be used to generate the annotation round. New target domain annotated data is generated and the new annotated data is merged with the existing annotated data through a union (U) operation.

[0111] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0112] In a confirmatory example, the Potsdam dataset and the Vaihingen dataset were used to conduct a comparison experiment between the method of the present application and other similar methods.

[0113] (1) Experimental materials:

[0114] Potsdam Dataset: The Potsdam dataset contains 38 remote sensing images taken in Potsdam, Germany. Each image is 6000×6000×3 and has a resolution of 5 em. Potsdam contains six target categories: buildings, roads, vehicles, trees, low vegetation, and other. After removing one incorrectly labeled image, we selected the first 24 images as the training set and the last 13 images as the test set. To construct the training set, we used a sliding window of size 960×960 with a stride of 480 to tile the images, resulting in 3456 960×960×3 remote sensing tiles for training. To construct the test set, we used a sliding window of size 960×960 with a stride of 960 to tile the images, resulting in 637 960×960×3 remote sensing tiles for testing.

[0115] Vaihingen Dataset: The Vaihingen dataset contains 33 remote sensing images taken in Vaihingen, Germany. Each image has a varying size and a resolution of 9 cm. Vaihingen contains the same six target categories as Potsdam. We selected the first 26 images as the training set and the last 7 images as the test set. To construct the training set, we used a sliding window of size 512×512 with a stride of 256 to tile the images, resulting in 1784 512×512×3 remote sensing training tiles. To construct the test set, we used a sliding window of size 512×512 with a stride of 512 to tile the images, resulting in 155 512×512×3 remote sensing test tiles.

[0116] Because Potsdam has a larger dataset, domain adaptation from Potsdam to Vaihingen is more suitable for practical applications and is a common benchmark for domain adaptation semantic segmentation. Therefore, this paper's experiments will use the images and annotations from the Potsdam training set and the images from Vaihingen as training data, the annotations from Vaihingen as the training medium, and the images and annotations from the Vaihingen test set as test data.

[0117] (2) Experimental configuration

[0118] The model is updated using the SGD optimizer, where the image information preprocessing module and the segmentation network use the pre-trained weights of deeplabv2-resnet101, and the initial learning rate is set to 1×10 -4 , the interactive information preprocessing module is randomly initialized, and the initial learning rate is set to 1×10 -3The training rounds M1 in the warm-up phase are set to 3000, the low-noise training rounds M′1 are set to 500, and the threshold fn_score_thres is set to 0.3. The training rounds M2 in the domain adaptation phase are set to 7000, and the labeling rounds [m1, m2, ..., m n ] is set to [1, 501, 1001, 1501, 2001], and the number of annotations N p Set to 40. The threshold τ of the negative sample loss function n Set to 0.05, the coefficient λ is set to 1.0. The threshold τ of the positive sample loss function p The coefficient γ is set to 0.05 and 1.0. The scaling factor β of the interaction loss function is set to 0.1. To reduce the impact of the resolution of the two datasets on domain adaptation, the input size of the Potsdam dataset is 960×960×3, and the input size of the Vaihingen dataset is 512×512×3.

[0119] (3) Evaluation indicators

[0120] F1: The F1 score is an evaluation indicator for multi-classification problems, and its calculation expression is:

[0121] Precision = TP / (TP+FP)

[0122] Recall = TP / (TP+FN)

[0123]

[0124] Where TP is the true positive example of each class, FP is the false positive example of each class, and FN is the false negative example of each class.

[0125] mIoU: The ratio of the intersection and union of the model's predicted mask and the true mask, which is calculated as:

[0126]

[0127] Where TP is the true positive of each class, FP is the false positive of each class, FN is the false negative of each class, and C is the number of classes.

[0128] (4) Experimental results

[0129] The results of the domain adaptation experiment from Potsdam to Vaihingen are shown in Table 1. The value under each category in the table is the F1 score, and the overall accuracy is measured by mIoU. The bold value is the optimal value.

[0130] Table 1 Comparative experiment of Potsdam → Vaihingen

[0131]

[0132] Experimental results show that our proposed interactive semantic segmentation model can improve the model's mIoU by 4.85% overall. Our method also achieves higher F1 scores on almost all categories, with the score on the more difficult "Other" class more than doubling. This demonstrates that our interactive semantic segmentation model, through continuous error correction, enables the model to gain a more accurate understanding of each category, and that the positive pseudo-annotations generated through error correction can also add more effective supervision information to the target domain dataset.

[0133] We adjust the value of Fn_score_thres to verify the effectiveness of the fusion of the interactive sampling and active learning annotation strategies, that is, to verify the effectiveness of the annotation / sampling strategy. The experimental results are shown in Table 2. The bold one is the best and the one with the lower line is the second best.

[0134] Table 2 Verification of the effectiveness of the labeling / sampling strategy

[0135]

[0136]

[0137] The algorithm shows that as the value of Fn_score_thres increases, the proportion of active learning annotation in the labeling / sampling strategy increases. Experimental results show that when Fn_score_thres < 0.5, the F1 score of the more difficult "other" class is very high, remaining above 50. However, it drops sharply when Fn_score_thres = 0.5. This demonstrates that interactive sampling can identify high-confidence but obvious errors in the model, allowing the model to gain a clearer understanding of each class, especially challenging ones. As Fn_score_thres decreases, the mIoU does not increase monotonically, indicating that interactive sampling struggles to select pixels with high labeling value that the model is unsure of. The optimal Fn_score_thres value of 0.3 is achieved, demonstrating that both active learning annotation and interactive sampling have their own advantages and disadvantages. Our proposed labeling / sampling strategy effectively leverages the strengths of both while mitigating their respective drawbacks.

[0138] In one embodiment, Figure 6 As shown, a domain adaptation interactive semantic segmentation device based on active learning is provided, comprising: a training data acquisition module, a domain adaptation interactive semantic segmentation model training module and a domain adaptation interactive semantic segmentation module, wherein:

[0139] The training data acquisition module is used to obtain training data sets in different domains. The training data sets include source domain data sets with labeled masks and unlabeled target domain data sets.

[0140] The domain adaptation interactive semantic segmentation model training module is used to train the domain adaptation interactive semantic segmentation model using the training dataset to obtain a trained domain adaptation interactive semantic segmentation model; the training stages include: a warm-up stage and a domain adaptation stage. In the warm-up stage, the domain adaptation semantic segmentation model is trained for multiple rounds based on the source domain dataset with labeled masks and the labeling / sampling strategy; in the domain adaptation stage, the model trained in the warm-up stage is trained for multiple rounds based on the source domain dataset, the target domain dataset and the labeling / sampling strategy to obtain a trained domain adaptation interactive semantic segmentation model; the labeling / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question labeling / sampling strategy, and uses the obtained new labels as the labeled pixels of the target domain dataset for supervised learning of the model, or inputs the obtained sampled interaction points into the model as interaction information.

[0141] The domain adaptation interactive semantic segmentation module is used to input the image to be segmented into the trained domain adaptation interactive semantic segmentation model to obtain the semantic segmentation result.

[0142] In one embodiment, the domain-adaptive interactive semantic segmentation model in the domain-adaptive interactive semantic segmentation model training module includes: an interactive encoding module, an interactive information preprocessing module, an image information preprocessing module, a feature fusion module and a semantic segmentation network; the interactive encoding module is used to convert the interactive information into an interactive information feature map; the interactive information preprocessing module and the image information preprocessing module are used to adjust the size and number of channels of the interactive information feature map and the image information feature to be consistent; the feature fusion module is used to fuse the preprocessed interactive information features and image information features using an element-by-element addition operation to obtain a fused feature; the semantic segmentation network is used to perform interactive semantic segmentation based on the fused feature to obtain a semantic segmentation prediction result.

[0143] In one embodiment, the domain adaptation interactive semantic segmentation model training module is further used to set the number of warm-up training rounds M1, the number of low-noise training rounds M′1, and the threshold fn_score_thres; set the current warm-up training round to 1; extract a batch of samples from the source domain data set as the input data and annotation mask of the current warm-up training round; set a preset number of iterations N in one round of training; during the first iteration, input the input data into the domain adaptation interactive semantic segmentation model to obtain an initial prediction mask; calculate the semantic segmentation loss of the first iteration based on the annotation mask and the initial prediction mask; during the second to N-1th iterations, automatically generate a preset amount of interactive information based on the initial prediction mask and the annotation mask using the annotation / sampling strategy; when the current warm-up training round is less than the low-noise training round, set the mask to If it is None, low-noise training is performed; when the current warm-up training round is greater than or equal to the low-noise training round, the model after the previous iteration is continued to be trained according to the input data, the predicted mask obtained in the previous iteration, and the interactive information until the N-1th iteration is completed. The interactive semantic segmentation loss of the N-1th iteration is calculated according to the labeled mask and the predicted mask of the N-1th iteration, and the sum of the semantic segmentation loss of the first iteration and the semantic segmentation loss of the interactive N-1th iteration is taken as the total loss of the current round in the warm-up stage; and the domain-adapted interactive semantic segmentation model is updated according to the total loss of the current round in the warm-up stage; the current warm-up training round is increased by 1, and the next round of warm-up training is performed until the current warm-up training round is greater than the number of training rounds in the warm-up stage, and the domain-adapted interactive semantic segmentation model after preheating training is obtained.

[0144] In one embodiment, the domain adaptation interactive semantic segmentation model training module is further used to input input data into the domain adaptation interactive semantic segmentation model to obtain an initial prediction mask as shown in formula (5).

[0145] In one embodiment, the domain adaptation interactive semantic segmentation model training module is also used to automatically generate a preset amount of interactive information based on the initial mask using a labeling / sampling strategy. The labeling / sampling strategy in the step is: based on the predicted mask and the labeled mask, calculate the proportion of the incorrectly segmented area in the predicted mask of each class as shown in formula (6); compare the proportion with the preset threshold, and when the proportion is greater than the preset threshold, use a visual approach to label / sample near the center of the error area with the largest area; when the proportion is less than the preset threshold, use a questioning approach to select pixels with high labeling value for labeling / sampling according to a value evaluation function; the value evaluation function is shown in formula (7).

[0146] In one embodiment, the domain adaptation interactive semantic segmentation model training module is further used to set the number of training rounds M2 in the domain adaptation stage, and mark the round set [m1, m2, ..., m n], number of annotations N p ; Use None to initialize the annotation mask of the target domain dataset; set the current domain adaptation training round to 1; extract a batch of samples from the source domain dataset and the target domain dataset respectively as the current source domain input dataset and annotation mask and the current target domain input dataset and annotation mask; set the preset number of iterations N1 in a round of domain adaptation training; in the first iteration: input the current source domain input data and the current target domain input data into the domain adaptation interactive semantic segmentation model to obtain the initial prediction masks of the source domain input data and the target domain input; if the annotation mask of the current target domain input data is not None, then input the annotated pixels therein as the interactive information into the domain adaptation interactive semantic segmentation model to obtain the correction mask; according to the annotation mask, initial prediction mask and correction mask of the source domain input data and the target domain input data, determine the total loss of the first iteration of the current round of the domain adaptation stage. loss; perform N1-1 interactive iterations on the domain adaptation interactive semantic segmentation model according to the initial prediction mask of the source domain, the current source domain input dataset, and the annotation / sampling strategy; multiply the loss value generated by the N1-1th interactive process by the scaling factor β and add it to the total loss of the first iteration to obtain the loss of the current domain adaptation training round; and update the domain adaptation interactive semantic segmentation model according to the loss of the current domain adaptation training round; if the current domain adaptation training round is an element of the annotation round set, generate a preset number of new target domain annotation pixels through the annotation / sampling strategy, and merge the new annotation pixels with the existing annotation pixels through a union operation to update the target domain annotation mask; increase the current domain adaptation training round by 1, and perform the next round of domain adaptation training until the current domain adaptation training round is greater than the number of training rounds in the domain adaptation stage, and obtain a trained domain adaptation interactive semantic segmentation model.

[0147] In one embodiment, the domain adaptation interactive semantic segmentation model training module is further used to determine the total loss of the first iteration of the current round of the domain adaptation stage according to the labeled masks, initial predicted masks, and corrected masks of the source domain input data and the target domain input data, as shown in Equations (11) to (13).

[0148] In one embodiment, the positive pseudo-labeling loss in the domain adaptation interactive semantic segmentation model training module is shown in Formulas (14) and (15); the negative pseudo-labeling loss is shown in Formulas (16) and (17).

[0149] For the specific definition of the domain adaptation interactive semantic segmentation device based on active learning, please refer to the definition of the domain adaptation interactive semantic segmentation method based on active learning above, which will not be repeated here. The various modules in the above-mentioned domain adaptation interactive semantic segmentation device based on active learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0150] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a domain-adaptive interactive semantic segmentation method based on active learning is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0151] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0152] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in any of the above method embodiments when executing the computer program.

[0153] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A domain-adaptive interactive semantic segmentation method based on active learning, characterized by: The method comprises: Obtaining a different domain training dataset, wherein the training dataset includes a source domain dataset with annotated masks and an unlabeled target domain dataset; The domain-adapted interactive semantic segmentation model is trained using the training dataset to obtain a trained domain-adapted interactive semantic segmentation model; the training stage includes: a warm-up stage and a domain adaptation stage. In the warm-up stage, the domain-adapted interactive semantic segmentation model is trained for multiple rounds based on a source domain dataset with annotated masks and an annotation / sampling strategy; in the domain adaptation stage, the model trained in the warm-up stage is trained for multiple rounds based on the source domain dataset, the target domain dataset, and the annotation / sampling strategy to obtain a trained domain-adapted interactive semantic segmentation model; the annotation / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question annotation / sampling strategy, using the obtained new annotations as the annotated pixels of the target domain dataset for supervised learning of the model, or inputting the obtained sampled interaction points into the model as interaction information; The image to be segmented is input into the trained domain-adapted interactive semantic segmentation model to obtain the semantic segmentation result.

2. The method according to claim 1, characterized in that The domain adaptation interactive semantic segmentation model includes: an interactive encoding module, an interactive information preprocessing module, an image information preprocessing module, a feature fusion module and a semantic segmentation network; The interaction encoding module is used to convert the interaction information into an interaction information feature map; The interaction information preprocessing module and the image information preprocessing module are used to adjust the size and number of channels of the interaction information feature map and the image information feature map to be consistent; The feature fusion module is used to fuse the preprocessed interactive information features and image information features by element-by-element addition to obtain fused features; The semantic segmentation network is used to perform interactive semantic segmentation based on the fusion features to obtain a semantic segmentation prediction result.

3. The method according to claim 1, characterized in that In the warm-up phase, the domain-adapted interactive semantic segmentation model is trained for multiple rounds based on the source domain dataset with annotated masks and the annotation / sampling strategy, including: Set the number of training rounds in the warmup phase , low-noise training rounds , threshold ; Set the current warm-up training round to 1; Extracting a batch of samples from the source domain dataset as input data and annotation masks for the current warm-up training round; Set the preset number of iterations N in one round of training; In a first iteration, the input data is input into a domain-adapted interactive semantic segmentation model to obtain an initial prediction mask; and a semantic segmentation loss of the first iteration is calculated based on the labeled mask and the initial prediction mask. In the second to N-1th iterations, a preset amount of interactive information is automatically generated using a labeling / sampling strategy based on the initial prediction mask and the labeling mask; when the current warm-up training round is less than the low-noise training round, mask Set to None to perform low-noise training; when the current warm-up training round is greater than or equal to the low-noise training round, continue training the model after the previous iteration according to the input data, the prediction mask obtained in the previous iteration, and the interactive information until the N-1th iteration is completed, calculate the interactive semantic segmentation loss of the N-1th iteration according to the labeled mask and the prediction mask of the N-1th iteration, and take the sum of the semantic segmentation loss of the first iteration and the semantic segmentation loss of the interactive N-1th iteration as the total loss of the current round in the warm-up phase; and update the domain-adapted interactive semantic segmentation model according to the total loss of the current round in the warm-up phase; The current warm-up training round is increased by 1, and the next warm-up training round is performed until the current warm-up training round is greater than the number of training rounds in the warm-up stage, and the domain-adapted interactive semantic segmentation model after warm-up training is obtained.

4. The method according to claim 3, characterized in that The input data is fed into the domain-adapted interactive semantic segmentation model, and the initial predicted mask is obtained as: in, For the i The initial prediction mask of the source domain data for rounds, For the i Rounds of source domain data input, is the semantic segmentation network, It is the image information preprocessing module.

5. The method according to claim 3, characterized in that A preset amount of interactive information is automatically generated using a labeling / sampling strategy based on the initial prediction mask and the labeling mask. The labeling / sampling strategy in the step is: According to the predicted mask and the annotated mask, the proportion of incorrectly segmented areas in the predicted mask of each class is calculated as: in, is the proportion of incorrectly segmented areas in the prediction mask of category c, For the c The predicted mask of the class, Calculate the function to predict the proportion of incorrectly segmented areas in the mask, For the c The annotation mask of the class; The proportion is compared with a preset threshold. When the proportion is greater than the preset threshold, a visual approach is adopted to mark / sample pixels near the center of the largest error area. When the proportion is less than the preset threshold, a questioning approach is adopted to select pixels with high marking value for marking / sampling according to a value evaluation function. The value evaluation function is: in, For coordinates The pixel annotation value of To predict the coordinates in the mask No. c Class prediction value, C is the set of all classes in the predicted mask, Value assessment function.

6. The method according to claim 1, characterized in that In the domain adaptation phase, the model trained in the warm-up phase is trained for multiple rounds based on the source domain dataset, the target domain dataset, and the look-before-question annotation / sampling strategy to obtain a trained domain-adapted interactive semantic segmentation model, including: Set the number of training rounds in the domain adaptation phase , labeling round set , mark the quantity ; Use None to initialize the annotation mask of the target domain dataset; Set the current domain adaptation training round to 1; Extracting a batch of samples from the source domain dataset and the target domain dataset as the current source domain input dataset and annotation mask and the current target domain input dataset and annotation mask respectively; Set the preset number of iterations N1 in one round of domain adaptation training; In a first iteration: inputting the current source domain input data and the current target domain input data into the domain-adapted interactive semantic segmentation model to obtain initial prediction masks of the source domain input data and the target domain input; If the annotation mask of the current target domain input data is not None, the annotated pixels therein are input into the domain-adaptive interactive semantic segmentation model as interactive information to obtain a corrected mask; Determine the total loss of the first iteration of the current round of the domain adaptation phase based on the labeled masks and the initial predicted masks of the source domain input data and the target domain input data and the corrected masks; Performing N1-1 interactive iterations on the domain-adapted interactive semantic segmentation model based on the initial predicted mask of the source domain, the current source domain input dataset, and the annotation / sampling strategy; Multiply the loss value generated by the N1-1th interactive process by the scaling factor Adding the loss to the total loss of the first iteration to obtain the loss of the current domain adaptation training round; and updating the domain adaptation interactive semantic segmentation model according to the loss of the current domain adaptation training round; If the current domain adaptation training round is an element of the annotation round set, a preset number of new target domain annotated pixels are generated through the annotation / sampling strategy, and the new annotated pixels are merged with the existing annotated pixels through a union operation to update the target domain annotation mask; The current domain adaptation training round is increased by 1, and the next round of domain adaptation training is performed until the current domain adaptation training round is greater than the number of training rounds in the domain adaptation stage, and a trained domain adaptation interactive semantic segmentation model is obtained.

7. The method according to claim 6, characterized in that According to the labeled masks and initial predicted masks of the source domain input data and the target domain input data and the correction mask, the total loss of the first iteration of the current round in the domain adaptation stage is determined as: in, is the total loss of the first iteration of the current round in the domain adaptation stage, is the total loss of semantic segmentation, is the semantic segmentation cross entropy loss of the source domain data, is the cross entropy loss for semantic segmentation of the target domain; is the positive and negative pseudo-labeling loss, is the positive-false labeling loss, is the negative pseudo-labeling loss; is the initial prediction mask of the source domain, is the initial prediction mask of the target domain, The source domain data i The annotation mask of the iteration, The target domain data i The annotation mask of the iteration.

8. The method according to claim 7, characterized in that The positive and false labeling loss is: in, is the threshold for determining the positive and false labels, express sample Pixel position pair The predicted value of the category, are the number of samples, sample size length, sample size width, number of sample channels, is the predicted value of the k-sample (i, j) pixel position for the c-th class, are the predicted values ​​of the initial mask and the final mask respectively; The negative pseudo-annotation loss is: in is the threshold for determining negative pseudo-annotations.

9. A domain-adaptive interactive semantic segmentation device based on active learning, characterized in that: The device comprises: A training data acquisition module is used to acquire training data sets of different domains, wherein the training data sets include a source domain data set with annotated masks and an unannotated target domain data set; A domain-adaptive interactive semantic segmentation model training module is configured to train the domain-adaptive interactive semantic segmentation model using the training dataset to obtain a trained domain-adaptive interactive semantic segmentation model. The training phases include a warm-up phase and a domain adaptation phase. In the warm-up phase, the domain-adaptive semantic segmentation model is trained for multiple rounds based on a source domain dataset with annotated masks and an annotation / sampling strategy. In the domain adaptation phase, the model trained in the warm-up phase is trained for multiple rounds based on the source domain dataset, the target domain dataset, and the annotation / sampling strategy to obtain a trained domain-adaptive interactive semantic segmentation model. The annotation / sampling strategy is based on a preset value evaluation function and a preset score threshold, and adopts a look-before-question annotation / sampling strategy. The obtained new annotations are used as the annotated pixels of the target domain dataset for supervised learning of the model, or the obtained sampled interaction points are input into the model as interaction information. The domain adaptation interactive semantic segmentation module is used to input the image to be segmented into the trained domain adaptation interactive semantic segmentation model to obtain the semantic segmentation result.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images