A semi-supervised instance segmentation method and system for pseudo label quality decoupling and correction

Abstract

The application discloses a kind of semi-supervised instance segmentation method and system of pseudo label quality decoupling and correction, method includes the following steps: teacher model processes unlabelled image and generates prediction result;By the filtering mechanism of decoupled double threshold, reserve the prediction result that meets condition as instance pseudo label;Dynamic instance class correction is carried out to instance pseudo label, and modified pseudo label is obtained;Loss function is constructed in combination with prediction result and modified pseudo label, and student model for carrying out instance segmentation is trained.The application is filtered by the filtering mechanism of decoupled double threshold, dynamic instance class correction module and pixel-level mask uncertainty perception mechanism, and the class of pseudo label and mask quality are independently controlled, effectively reduce the influence brought by pseudo label noise in semi-supervised instance segmentation task;The application is especially suitable for semi-supervised scene needing to utilize large-scale unlabelled data to improve the effectiveness of image understanding, and the performance of semi-supervised instance segmentation is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instance segmentation, in particular to a semi-supervised instance segmentation method and system for pseudo-label quality decoupling and correction. BACKGROUND

[0002] In the field of object instance segmentation, artificial intelligence technology has made significant progress in recent years, especially with the rapid development of deep learning, which has driven the improvement of image understanding capabilities. Large-scale labeled datasets such as COCO, LVIS, Cityscapes, and BDD100K are widely used in fully supervised instance segmentation (FSIS) research, significantly improving the accuracy of pixel-level image segmentation. However, the high cost and complexity of pixel-level labeling severely limit the promotion of FSIS in practical applications. For this reason, semi-supervised learning has gradually gained attention, reducing the dependence on labeled data by utilizing large-scale unlabeled data, thereby improving performance in image classification and object detection tasks. However, the instance segmentation task is more challenging than image classification and object detection, as it not only requires learning of semantic-level classes and instance-level coordinates, but also requires pixel-level classification and grouping. Therefore, semi-supervised instance segmentation (SSIS) lags far behind other semi-supervised learning tasks in development. The main problems faced by existing technologies include accurate evaluation of pseudo-label quality, class prediction confusion, and pixel-level mask pseudo-label noise, which severely affect the training effect and application performance of the model. To address these issues, there is an urgent need for a semi-supervised instance segmentation method that can effectively evaluate pseudo-label quality, reduce class confusion, and improve model robustness. SUMMARY

[0003] The purpose of the present application is to solve the problems caused by unstable pseudo-label quality, class confusion, and pixel-level pseudo-label noise in the prior art.

[0004] The technical solution adopted by the present application to solve its technical problems is to provide a semi-supervised instance segmentation method for pseudo-label quality decoupling and correction, comprising the following steps:

[0005] The teacher model processes unlabeled images and generates prediction results;

[0006] The prediction results are filtered through a decoupled double-threshold mechanism, and the prediction results that meet the conditions are retained as instance pseudo-labels;

[0007] The instance pseudo-labels are dynamically corrected for instance class, and corrected pseudo-labels are obtained;

[0008] A loss function is constructed combining the prediction results and the corrected pseudo-labels, and a student model for instance segmentation is trained.

[0009] Preferably, the teacher model processes unlabelled images and generates prediction results, the unlabelled images are input into the teacher model after weak augmentation, and the weak augmentation includes random horizontal flip and scaling jitter.

[0010] Preferably, the prediction results include class quality and mask quality, which are respectively represented as:

[0011]

[0012] wherein c k represents the class quality of the kth instance, x k represents the logit value of the class prediction of the kth instance, and N represents the number of classes; m k represents the mask quality of the kth instance, and HW represents the total number of all pixels in the mask, represents the mask logit value of each pixel in the kth instance, and σ represents the sigmoid function, represents the pixels whose mask probability is greater than 0.5.

[0013] Preferably, the prediction results are filtered through a decoupled double-threshold mechanism, and the filtering is realized by setting a class quality threshold c t and a mask quality threshold m t , which is represented as:

[0014]

[0015] Only instances whose pseudo labels (C i , M i ) meet the conditions are retained for training of the student model.

[0016] Preferably, the instance pseudo labels are dynamically corrected in terms of instance class, and the class correction for each pseudo label (C i , M i ) includes the following steps:

[0017] extracting an image block from the mask M i ; inputting the image block into the visual encoder CLIP V of CLIP;

[0018] inputting the text description of the class into the text encoder CLIP T of CLIP;

[0019] calculating the similarity probability distribution of the class based on the output of CLIP, which is represented as:

[0020] ​

[0021] a similarity probability distribution of the classes a class probability distribution generated by the teacher model perform weighted fusion to obtain a final class correction probability distribution is expressed as:

[0022]

[0023] wherein the weight w decays with the number of training iterations;

[0024] based on the final class correction probability distribution select a correction class of the pseudo label is expressed as:

[0025]

[0026] wherein argmax represents the class with the maximum probability value in the probability distribution .

[0027] Preferably, the prediction result and the corrected pseudo label are combined to construct a loss function, and the loss function includes an unlabeled data loss L unsup , which is expressed as:

[0028]

[0029]

[0030] wherein, represents a classification loss, Q represents the number of instances predicted in the training process, represents the best matching pseudo label selected by the Hungarian algorithm matching k predictions, is the predicted class probability of the i-th class of the k-th instance by the student model, represents the correction class of the pseudo label selected by the Hungarian matching; represents a mask loss, is the predicted mask probability of the i-th pixel of the k-th instance by the student model, is the optimal mask pseudo label selected by the Hungarian algorithm matching; represents the uncertainty of the mask prediction probability of each pixel by the teacher model, represents the uncertainty selected by the Hungarian algorithm matching, represents the mask prediction probability of each pixel by the teacher model.

[0031] Preferably, the student model trained for instance segmentation is trained using unlabeled images, and the unlabeled images are input into the student model after being subjected to strong enhancement, and the strong enhancement includes horizontal flipping, scaling jitter, color jitter, grayscale, Gaussian blur, and CutOut.

[0032] Preferably, the teacher model and the student model both use the basic instance segmentation network Mask2Former.

[0033] The application also provides a semi-supervised instance segmentation system for pseudo label quality decoupling and correction, comprising:

[0034] A prediction module, the teacher model processes the unlabeled images and generates a prediction result;

[0035] A filtering module, the prediction result is filtered through a decoupled double-threshold mechanism, and the prediction result meeting the condition is reserved as an instance pseudo label;

[0036] A correction module, the instance pseudo label is dynamically corrected for instance class to obtain a corrected pseudo label;

[0037] A training module, a loss function is constructed in combination with the prediction result and the corrected pseudo label, and a student model for instance segmentation is trained.

[0038] The application has the following beneficial effects:

[0039] (1) The application designs a decoupled double-threshold filtering mechanism, independently evaluates the class quality and mask quality of the pseudo label, thereby ensuring the high-quality performance of the instance pseudo label in classification and grouping, and avoiding the problem of inaccurate pseudo label caused by coupled filtering;

[0040] (2) The application introduces a dynamic instance class correction module based on a visual-linguistic alignment model (such as CLIP), dynamically adjusts the probability distribution of the class pseudo label by using a visual-linguistic model pre-trained from a large-scale image-text pair, corrects the confusion problem in class prediction, and thereby improves the accuracy of instance classification;

[0041] (3) Since pixel-level pseudo labels are more susceptible to noise than instance-level pseudo labels, the application proposes a pixel-level mask uncertainty perception loss function. The loss function assigns a smaller weight to the area with higher uncertainty and a larger weight to the area with lower uncertainty according to the uncertainty of the mask pseudo label, thereby reducing the influence of noise on model training and enhancing the robustness and segmentation accuracy of the model.

[0042] The application will be further described in detail below in combination with the drawings and embodiments, but the application is not limited to the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Method step diagram of the embodiment of the present application;

[0044] Figure 2 Flowchart of the embodiment of the present application;

[0045] Figure 3 System structure diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application proposes a pseudo-label quality decoupling and correction framework (PL-DC) for a semi-supervised instance segmentation task, aiming to solve the problems of unstable pseudo-label quality, class confusion and pixel-level pseudo-label noise in semi-supervised instance segmentation. The framework adopts a teacher-student network architecture, generates pseudo-labels by a teacher model, and learns by a student model through pseudo-labels and labeled data, finally improves the segmentation performance.

[0047] Referring to Figure 1 and Figure 2 , the method step diagram and the flowchart of the embodiment of the present application, comprising the following steps:

[0048] S101, the teacher model processes the unlabeled image and generates a prediction result;

[0049] S102, filtering the prediction result by a decoupled dual-threshold mechanism, and retaining the prediction result meeting the condition as an instance pseudo-label;

[0050] S103, dynamically correcting the instance pseudo-label to obtain a corrected pseudo-label;

[0051] S104, combining the prediction result and the corrected pseudo-label to construct a loss function, and training a student model for instance segmentation.

[0052] The content of the embodiment of the present application is described in detail as follows.

[0053] (1) Decoupled dual-threshold filtering mechanism (DDTF).

[0054] In the traditional pseudo-label filtering method, the class quality and mask quality of the instance are usually evaluated by a coupled score. However, the quality of the class and the mask is not always improved synchronously, and a single threshold filtering is easy to cause deviation in one aspect, thereby affecting the accuracy of the pseudo-label. Therefore, the present application designs a decoupled dual-threshold filtering mechanism, which independently controls the filtering process of the pseudo-label by respectively evaluating the class and mask quality of the instance. The teacher model will process the unlabeled image after weak enhancement and generate a prediction result. The class quality c kand mask quality m k are calculated by the following equations, respectively.

[0055] category quality c k is represented as:

[0056]

[0057] where N is the number of categories, x k is the logit value of the category prediction of the kth instance.

[0058] mask quality m k is represented as:

[0059]

[0060] where HW represents the total number of pixels in the mask, is the logit value of the mask of each pixel of the kth instance, and σ represents the sigmoid function, is the pixel whose mask probability is greater than 0.5.

[0061] Subsequently, through the decoupled double-threshold mechanism, the teacher model will filter the pseudo-labels according to the category quality threshold c t and the mask quality threshold m t

[0062]

[0063] Only the instances whose pseudo-labels (C i , M i ) meet the conditions will be retained for the training of the student model.

[0064] (2) Dynamic Instance Category Correction (DICC) module.

[0065] Due to the similarity or frequent co-occurrence of some categories, the problem of category confusion may occur. To solve this problem, the present application introduces a dynamic instance category correction module based on a visual-linguistic alignment model (such as CLIP). This module reduces category confusion by dynamically correcting the category pseudo-labels.

[0066] For each pseudo-label (C i , M i ), first, the image block is extracted from the mask M i · Then, the image block is input into the visual encoder CLIP V of CLIP, and the category text description is input into the text encoder CLIP​T , compute the similarity probability distribution of the category

[0067]

[0068] The probability distribution calculated by CLIP The category probability distribution generated by the teacher model Weighted fusion to obtain the final category correction probability distribution

[0069]

[0070] Wherein, the weight w decays with the number of training iterations, the initial value is 0.5, and gradually decreases to 0, to reflect the gradual improvement of the accuracy of the teacher model.

[0071] Finally, the modified category of the pseudo label is selected based on the highest probability

[0072]

[0073] (3) Pixel-level mask uncertainty-aware loss (Pixel-Level Mask Uncertainty-Aware, PMUA).

[0074] In semi-supervised learning, the noise of pixel-level pseudo labels may affect the training of the model, especially in areas with object boundaries or object overlaps. Therefore, the present application proposes an uncertainty-aware loss function that reweights the loss based on the pixel-level uncertainty of the mask pseudo label.

[0075] The mask prediction probability of each pixel of the teacher model is Its uncertainty is defined as:

[0076]

[0077] The higher the uncertainty, the more unreliable the prediction of the pixel. When training the student model, the pixel-level mask loss for unlabeled data is defined as follows:

[0078]

[0079] Wherein, is the predicted mask probability of the student model for the i-th pixel of the k-th instance, is the optimal mask pseudo label assigned by the Hungarian algorithm. Through this uncertainty-aware loss, pixels with high uncertainty contribute less to the loss, thereby reducing the impact of noise on model training and enhancing the robustness of the model.

[0080] The student model overall loss function L of the embodiment of the application is represented as:

[0081] L = L sup + λL unsup ;

[0082] Wherein, L sup represents the labeled data loss, L unsup represents the unlabeled data loss, and λ represents a weight coefficient.

[0083] The effect of the application is further illustrated by the following simulation experiment.

[0084] The application is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the application is Python. In the simulation experiment of the application, a large number of tests are carried out on the COCO and Cityscapes data sets to verify the effectiveness of the PL-DC framework in the semi-supervised instance segmentation task.

[0085] On the COCO data set, the train2017 subset of COCO is used, which contains 118,000 labeled images, 5,000 validation images and 123,000 unlabeled images. 1%, 2%, 5% and 10% of the labeled images are randomly selected for training, and the remaining data is regarded as unlabeled data. In addition, the full amount of labeled data is also used in the experiment for comparison. The Cityscapes data set contains 2,975 training images and 500 validation images, and the image size is 1024x2048, which is labeled with 8 semantic instance categories. In the Cityscapes experiment, 5%, 10%, 20% and 30% of the labeled data are randomly selected for training, and the remaining data is regarded as unlabeled data.

[0086] The basic instance segmentation network used is Mask2Former, and is based on the ResNet-50 backbone network. All experiments are carried out on four 3090 GPUs with 24GB of video memory, using the AdamW optimizer, and the learning rate and weight decay are both set to 0.0001. The teacher model is weakly augmented (including random horizontal flip and scaling jitter), and the student model is strongly augmented (including horizontal flip, scaling jitter, color jitter, grayscale, Gaussian blur and CutOut). The filtering threshold of the pseudo label is set to the mask quality threshold mt=0.9 and the class quality threshold ct=0.85. The parameters of the teacher model are updated by exponential moving average (EMA), and the decay rate of EMA is α=0.9996.

[0087] Table 1 compares different latest SSIS technology methods on COCO.

[0088]

[0089] Table 2 Comparison of different state-of-the-art SSIS techniques on Cityscapes.

[0090]

[0091] Referring to Figure 3 As shown in the system structure diagram of the embodiment of the application, comprising:

[0092] The prediction module 301 processes the unlabeled images and generates a prediction result by the teacher model;

[0093] The filtering module 302 filters the prediction result by a decoupled double threshold mechanism, and retains the prediction result meeting the condition as an instance pseudo label;

[0094] The correction module 303 performs dynamic instance class correction on the instance pseudo label to obtain a corrected pseudo label;

[0095] The training module 304 combines the prediction result and the corrected pseudo label to construct a loss function, and trains a student model for instance segmentation.

[0096] It can be seen that the application provides a pseudo label quality decoupling and correction method, referred to as PseudoLabelQuality Decoupling and Correction (PL-DC). The framework independently controls the class and mask quality of the pseudo label through a decoupled double threshold filtering mechanism, a dynamic instance class correction module and a pixel-level mask uncertainty perception mechanism, effectively reducing the influence of pseudo label noise in the semi-supervised instance segmentation task. PL-DC is especially suitable for semi-supervised scenarios that need to use large amounts of unlabeled data to improve the effectiveness of image understanding, and significantly improves the performance of semi-supervised instance segmentation.

[0097] The above is only a preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A semi-supervised instance segmentation method for decoupling and correcting pseudo-label quality, characterized in that, Includes the following steps: The teacher model processes unlabeled images and generates predictions. The prediction results are filtered by a decoupled dual threshold mechanism, and the prediction results that meet the conditions are retained as instance pseudo-labels. Dynamically correct the instance category of the instance pseudo-tags to obtain corrected pseudo-tags; A loss function is constructed by combining the prediction results and corrected pseudo-labels, and a student model for instance segmentation is trained. The prediction results include category quality and mask quality, which are expressed as follows: Among them, C k x represents the class quality of the k-th instance. k This represents the logit value of the class prediction for the k-th instance, where N represents the number of classes; M k HW represents the mask quality of the k-th instance, and HW represents the total number of pixels in the mask. This represents the mask logit value for each pixel in the k-th instance, where σ represents the sigmoid function. This represents pixels with a mask probability greater than 0.5; The decoupled dual-threshold mechanism for filtering prediction results specifically involves setting a category quality threshold c. t and mask quality threshold m t To achieve filtering, it is represented as: Only instance pseudo-tags that meet the conditions (C) i M i Only those that are selected will be retained for training student models; The dynamic instance category correction of the instance pseudo-tags is described above, for each pseudo-tag (C i M i The category correction includes the following steps: From mask M i Extracting image patches The image block is input into CLIP's visual encoder. V ; Text description of the category CLIP text encoder for input CLIP T ; The similarity probability distribution of categories is calculated based on CLIP output. Represented as: Similarity probability distribution of categories Class probability distribution generated by the teacher model Weighted fusion is performed to obtain the final category-corrected probability distribution. Represented as: Among them, the weight w decays with the number of training iterations; Based on the final category-corrected probability distribution Select the correction category for pseudo-labels. Represented as: Where argmax represents the probability distribution to be obtained. The category with the highest probability value; The loss function is constructed by combining the prediction results and the corrected pseudo-labels. The loss function includes the unlabeled data loss L. unsup , represented as: in, Let Q represent the classification loss, and let Q represent the number of instances predicted during training. This represents the pseudo-label of the best match for the k-th instance in the Hungarian algorithm's matching selection prediction. The student model predicts the class probability for the i′ class of the k-th instance. This indicates the pseudo-label correction category selected through Hungarian matching; Indicates mask loss. It is the predicted mask probability of the student model for the i-th pixel of the k-th instance. It is the optimal mask pseudo-label selected through Hungarian algorithm matching; This represents the uncertainty in the teacher model's mask prediction probability for each pixel. This indicates the uncertainty in the selection process using the Hungarian algorithm. This represents the mask prediction probability of the teacher model for each pixel.

2. The semi-supervised instance segmentation method for decoupling and correcting pseudo-label quality according to claim 1, characterized in that, The teacher model processes unlabeled images and generates prediction results. The unlabeled images are then input into the teacher model after undergoing weak enhancement processing, which includes random horizontal flipping and scaling jitter.

3. The semi-supervised instance segmentation method for decoupling and correcting pseudo-label quality according to claim 1, characterized in that, The training is used to train a student model for instance segmentation. During training, unlabeled images are used. The unlabeled images are then strongly enhanced and input into the student model. The strong enhancements include horizontal flipping, scaling jitter, color jitter, grayscale, Gaussian blur, and CutOut.

4. The semi-supervised instance segmentation method for decoupling and correcting pseudo-label quality according to claim 1, characterized in that, Both the teacher and student models use the basic instance segmentation network Mask2Former.

5. A semi-supervised instance segmentation system for decoupling and correcting pseudo-label quality, used to implement the semi-supervised instance segmentation method for decoupling and correcting label quality as described in any one of claims 1 to 4, characterized in that, include: In the prediction module, the teacher model processes unlabeled images and generates prediction results. The filtering module filters the prediction results through a decoupled dual threshold mechanism, retaining the prediction results that meet the conditions as instance pseudo-labels; The correction module dynamically corrects the instance category of the instance pseudo-tags to obtain corrected pseudo-tags. The training module combines the prediction results and corrected pseudo-labels to construct a loss function and trains a student model for instance segmentation.

Images