Semi-supervised instance segmentation method and system for pseudo label quality decoupling and correction

Abstract

The invention discloses a semi-supervised instance segmentation method and system for pseudo label quality decoupling and correction. The method comprises the following steps: a teacher model processes an unlabeled image and generates a prediction result; filtering prediction results through a decoupled dual-threshold mechanism, and reserving prediction results meeting conditions as instance pseudo labels; performing dynamic instance category correction on the instance pseudo-tag to obtain a corrected pseudo-tag; and constructing a loss function in combination with the prediction result and the corrected pseudo tag, and training a student model for instance segmentation. According to the invention, through a decoupled dual-threshold filtering mechanism, a dynamic instance category correction module and a pixel-level mask uncertainty perception mechanism, the category and mask quality of a pseudo tag are independently controlled, and the influence caused by pseudo tag noise in a semi-supervised instance segmentation task is effectively reduced; the method is especially suitable for a semi-supervised scene which needs to use large-scale unlabeled data to improve the effectiveness of the image understanding aspect, and the performance of semi-supervised instance segmentation is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of instance segmentation, and in particular to a semi-supervised instance segmentation method and system for pseudo-label quality decoupling and correction. Background Art

[0002] In the field of object instance segmentation, artificial intelligence technology has made significant progress in recent years, especially with the rapid development of deep learning, which has promoted the improvement of image understanding ability. Large-scale annotated datasets such as COCO, LVIS, Cityscapes and BDD100K are widely used in the research of fully supervised instance segmentation (FSIS), which significantly improves the accuracy of pixel-level image segmentation. However, the high cost and complexity of pixel-level annotation seriously limit the promotion of FSIS in practical applications. To this end, semi-supervised learning has gradually attracted attention. By utilizing large-scale unlabeled data and reducing the dependence on labeled data, it can improve the performance in image classification and object detection tasks. However, the instance segmentation task is more challenging than image classification and object detection because it requires not only the learning of semantic-level categories and instance-level coordinates, but also pixel-level classification and grouping. Therefore, semi-supervised instance segmentation (SSIS) lags far behind other semi-supervised learning tasks in development. The main problems faced by existing technologies include: accurate evaluation of pseudo-label quality, category prediction confusion, and pixel-level mask pseudo-label noise, which seriously affect the training effect and application performance of the model. To address these problems, there is an urgent need for a semi-supervised instance segmentation method that can effectively evaluate the quality of pseudo-labels, alleviate category confusion, and improve model robustness. Summary of the invention

[0003] The purpose of the present invention is to solve the problems of unstable pseudo-label quality, category confusion and pixel-level pseudo-label noise in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a semi-supervised instance segmentation method for pseudo-label quality decoupling and correction, comprising the following steps:

[0005] The teacher model processes the unlabeled images and generates predictions;

[0006] The prediction results are filtered through a decoupled dual threshold mechanism, and the prediction results that meet the conditions are retained as instance pseudo labels;

[0007] Dynamically correct the instance category of the instance pseudo label to obtain the corrected pseudo label;

[0008] The loss function is constructed by combining the prediction results and the corrected pseudo labels to train the student model for instance segmentation.

[0009] Preferably, the teacher model processes unlabeled images and generates prediction results, and the non-standard images are input into the teacher model after being weakly enhanced, and the weak enhancement includes random horizontal flipping and scaling jitter.

[0010] Preferably, the prediction result includes category quality and mask quality, which are respectively expressed as:

[0011]

[0012] Among them, c k represents the category quality of the kth instance, x k represents the logit value of the category prediction of the kth instance, N represents the number of categories; m k represents the mask quality of the kth instance, HW represents the total number of all pixels in the mask, represents the mask logit value of each pixel in the kth instance, σ represents the sigmoid function, Indicates pixels with mask probability greater than 0.5.

[0013] Preferably, the prediction results are filtered by the decoupled dual threshold mechanism, specifically by setting the category quality threshold c t and the mask quality threshold m t To achieve filtering, it is expressed as:

[0014]

[0015] Only the pseudo labels of instances that meet the conditions (C i ,M i ) will be retained for the training of the student model.

[0016] Preferably, the instance pseudo-label is dynamically modified to the instance category, and for each pseudo-label (C i ,M i ) category modification includes the following steps:

[0017] From the mask M i Extract image blocks This image patch is fed into CLIP’s visual encoder V ;

[0018] The text description of the category Input CLIP text encoder CLIP T ;

[0019] Calculate the similarity probability distribution of categories based on the output of CLIP It is expressed as:

[0020]

[0021] The similarity probability distribution of categories Class probability distribution generated by the teacher model Perform weighted fusion to obtain the final category correction probability distribution It is expressed as:

[0022]

[0023] Among them, the weight w decays with the number of training iterations;

[0024] Based on the final category correction probability distribution Select the correction category for pseudo-labels It is expressed as:

[0025]

[0026] Among them, argmax means obtaining the probability distribution The category with the highest probability value.

[0027] Preferably, the loss function is constructed by combining the prediction results and the corrected pseudo-labels, and the loss function includes the unlabeled data loss L unsup , expressed as:

[0028]

[0029]

[0030] in, represents the classification loss, Q represents the number of instances predicted during training, The Hungarian algorithm matches the best matching pseudo labels selected by k predictions. is the predicted category probability of the student model for the i-th category of the k-th instance, represents the pseudo-label correction category after Hungarian matching selection; represents the mask loss, is the predicted mask probability of the student model for the i-th pixel of the k-th instance, It is the optimal mask pseudo-label selected by Hungarian algorithm matching; represents the uncertainty of the teacher model’s mask prediction probability for each pixel, represents the uncertainty of the matching selection after the Hungarian algorithm, represents the teacher model's mask prediction probability for each pixel.

[0031] Preferably, the training is used for a student model for instance segmentation, and unlabeled images are used for training. The unlabeled images are input into the student model after strong enhancement, and the strong enhancement includes horizontal flipping, scaling jittering, color jittering, grayscale, Gaussian blurring and CutOut.

[0032] Preferably, both the teacher model and the student model adopt the basic instance segmentation network Mask2Former.

[0033] The present invention also provides a semi-supervised instance segmentation system with pseudo-label quality decoupling and correction, comprising:

[0034] Prediction module, where the teacher model processes unlabeled images and generates prediction results;

[0035] The filtering module filters the prediction results through a decoupled dual threshold mechanism and retains the prediction results that meet the conditions as instance pseudo labels;

[0036] The correction module dynamically corrects the instance category of the instance pseudo label to obtain the corrected pseudo label;

[0037] The training module combines the prediction results and the corrected pseudo labels to construct a loss function and train the student model for instance segmentation.

[0038] The present invention has the following beneficial effects:

[0039] (1) The present invention designs a decoupled dual-threshold filtering mechanism to independently evaluate the category quality and mask quality of pseudo labels, thereby ensuring high-quality performance of instance pseudo labels in classification and grouping, and avoiding the problem of inaccurate pseudo labels caused by coupled filtering;

[0040] (2) The present invention introduces a dynamic instance category correction module based on a visual-language alignment model (such as CLIP). By using a visual-language model pre-trained from large-scale image-text pairs, the probability distribution of category pseudo-labels is dynamically adjusted to correct the confusion problem in category prediction, thereby improving the accuracy of instance classification.

[0041] (3) Since pixel-level pseudo-labels are often more susceptible to noise than instance-level pseudo-labels, the present invention proposes a pixel-level mask uncertainty-aware loss function. This loss function assigns smaller weights to regions with higher uncertainty and larger weights to regions with lower uncertainty according to the uncertainty of the mask pseudo-label, thereby reducing the impact of noise on model training and enhancing the robustness and segmentation accuracy of the model.

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A method step diagram of an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a flow chart of an embodiment of the present invention;

[0045] Figure 3 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] This paper proposes a pseudo-label quality decoupling and correction framework (PL-DC) for semi-supervised instance segmentation tasks, aiming to solve the problems of unstable pseudo-label quality, category confusion and pixel-level pseudo-label noise in semi-supervised instance segmentation. The framework adopts a teacher-student network architecture, in which the teacher model generates pseudo-labels, and the student model learns through pseudo-labels and labeled data, ultimately improving the segmentation performance.

[0047] See also Figure 1 and Figure 2 The method step diagram and flow chart of an embodiment of the present invention are shown, which include the following steps:

[0048] S101, the teacher model processes the unlabeled image and generates prediction results;

[0049] S102, filtering the prediction results through a decoupled dual threshold mechanism, and retaining the prediction results that meet the conditions as instance pseudo labels;

[0050] S103, dynamically correcting the instance category of the instance pseudo label to obtain a corrected pseudo label;

[0051] S104, combining the prediction results and the corrected pseudo labels to construct a loss function and train a student model for instance segmentation.

[0052] The following describes the contents of the embodiments of the present invention in detail point by point.

[0053] (1) Decoupled Dual-Threshold Filtering (DDTF).

[0054] In traditional pseudo-label filtering methods, the category quality and mask quality of instances are usually evaluated uniformly using a coupled score. However, the quality of categories and masks often do not improve synchronously, and a single threshold filtering can easily lead to deviations in one aspect, thereby affecting the accuracy of pseudo-labels. To this end, the present invention designs a decoupled dual-threshold filtering mechanism that independently controls the pseudo-label filtering process by separately evaluating the category and mask quality of instances. The teacher model processes unlabeled images that have been weakly enhanced. And generate prediction results. The category quality c of the instance kand the mask quality m k They are calculated using the following formulas respectively.

[0055] Category Quality c k It is expressed as:

[0056]

[0057] Where N is the number of categories, x k is the logit value of the class prediction for the kth instance.

[0058] Mask quality m k It is expressed as:

[0059]

[0060] Where HW represents the total number of all pixels in the mask, is the mask logit value of each pixel of the kth instance, σ represents the sigmoid function, Refers to pixels with mask probability greater than 0.5.

[0061] Then, through the decoupled dual threshold mechanism, the teacher model will be based on the category quality threshold c t and the mask quality threshold m t Filter pseudo labels:

[0062]

[0063] Only the pseudo labels of instances that meet the conditions (C i ,M i ) will be retained for the training of the student model.

[0064] (2) Dynamic Instance Category Correction (DICC) module.

[0065] Since some categories look similar or often co-occur, category confusion may occur. To solve this problem, the present invention introduces a dynamic instance category correction module based on a visual-language alignment model (such as CLIP). This module reduces category confusion by dynamically correcting category pseudo-labels.

[0066] For each pseudo label (C i ,M i ), first from the mask M i Extract image blocks This image patch is then fed into CLIP’s visual encoder V , while using the category text description Input CLIP text encoder CLIPT , calculate the similarity probability distribution of categories

[0067]

[0068] The probability distribution calculated by CLIP Class probability distribution generated by the teacher model Perform weighted fusion to obtain the final category correction probability distribution

[0069]

[0070] Among them, the weight w decays with the number of training iterations. The initial value is 0.5 and gradually decreases to 0 to reflect the gradual improvement of the accuracy of the teacher model.

[0071] Finally, the corrected category of the pseudo-label is selected based on the highest probability

[0072]

[0073] (3) Pixel-Level Mask Uncertainty-Aware (PMUA).

[0074] In semi-supervised learning, the noise of pixel-level pseudo-labels may affect the training of the model, especially in the object boundaries or object overlapping areas. To this end, this paper proposes an uncertainty-aware loss function to reweight the loss according to the pixel-level uncertainty of the mask pseudo-labels.

[0075] The teacher model predicts the probability of the mask for each pixel as Its uncertainty Defined as:

[0076]

[0077] The higher the uncertainty, the less reliable the prediction result for that pixel. When training the student model, the pixel-level mask loss for unlabeled data is defined as follows:

[0078]

[0079] in, is the predicted mask probability of the student model for the i-th pixel of the k-th instance, is the optimal mask pseudo-label assigned by the Hungarian algorithm. Through this uncertainty-aware loss, pixels with higher uncertainty contribute less to the loss, thereby reducing the impact of noise on model training and enhancing the robustness of the model.

[0080] The overall loss function L of the student model in the embodiment of the present invention is expressed as:

[0081] L=L sup +λL unsup ;

[0082] Among them, L sup represents the loss of labeled data, L unsup represents the unlabeled data loss, and λ represents the weight coefficient.

[0083] The effect of the present invention is further illustrated by the following simulation experiment.

[0084] The present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python. In the simulation experiment of the present invention, a large number of tests are carried out on the COCO and Cityscapes datasets to verify the effectiveness of the PL-DC framework in the semi-supervised instance segmentation task.

[0085] On the COCO dataset, the train2017 subset of COCO was used, which contains 118,000 annotated images, 5,000 validation images, and 123,000 unlabeled images. 1%, 2%, 5%, and 10% of the annotated images were randomly selected for training, and the rest of the data was considered unlabeled. In addition, the full amount of annotated data was used in the experiment for comparison. The Cityscapes dataset contains 2,975 training images and 500 validation images, with an image size of 1024x2048 and 8 semantic instance categories annotated. In the Cityscapes experiment, 5%, 10%, 20%, and 30% of the annotated data were randomly selected for training, and the rest of the data was considered unlabeled.

[0086] The basic instance segmentation network used is Mask2Former, which is based on the ResNet-50 backbone network. All experiments are performed on four 3090 GPUs with 24GB of video memory, using the AdamW optimizer, with the learning rate and weight decay set to 0.0001. The teacher model is weakly enhanced (including random horizontal flip and scale jitter), while the student model is strongly enhanced (including horizontal flip, scale jitter, color jitter, grayscale, Gaussian blur, and CutOut). The filtering thresholds for pseudo labels are set to the mask quality threshold mt=0.9 and the category quality threshold ct=0.85. The parameters of the teacher model are updated by the exponential moving average (EMA), and the decay rate of EMA is α=0.9996.

[0087] Table 1 Comparison with different latest SSIS technology methods on COCO.

[0088]

[0089] Table 2 Comparison with different latest SSIS technology methods on Cityscapes.

[0090]

[0091] See also Figure 3 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0092] Prediction module 301, the teacher model processes the unlabeled images and generates prediction results;

[0093] A filtering module 302 filters the prediction results through a decoupled dual threshold mechanism and retains the prediction results that meet the conditions as instance pseudo labels;

[0094] A correction module 303 performs dynamic instance category correction on the instance pseudo label to obtain a corrected pseudo label;

[0095] The training module 304 constructs a loss function by combining the prediction results and the corrected pseudo labels, and trains a student model for instance segmentation.

[0096] It can be seen that the present invention provides a method for decoupling and correcting pseudo label quality, called PseudoLabelQuality Decoupling and Correction (PL-DC). This framework independently controls the category and mask quality of pseudo labels through a decoupled dual threshold filtering mechanism, a dynamic instance category correction module, and a pixel-level mask uncertainty perception mechanism, effectively reducing the impact of pseudo label noise in semi-supervised instance segmentation tasks. PL-DC is particularly suitable for semi-supervised scenarios that require the use of large-scale unlabeled data to improve the effectiveness of image understanding, and it significantly improves the performance of semi-supervised instance segmentation.

[0097] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-supervised instance segmentation method with pseudo-label quality decoupling and correction, characterized in that: The following steps are involved: The teacher model processes the unlabeled images and generates predictions; The prediction results are filtered through a decoupled dual threshold mechanism, and the prediction results that meet the conditions are retained as instance pseudo labels; Dynamically correct the instance category of the instance pseudo label to obtain the corrected pseudo label; The loss function is constructed by combining the prediction results and the corrected pseudo labels to train the student model for instance segmentation.

2. The method for semi-supervised instance segmentation with pseudo-label quality decoupling and correction according to claim 1, characterized in that: The teacher model processes the unlabeled image and generates a prediction result, and the unstandardized image is input into the teacher model after being weakly enhanced, and the weak enhancement includes random horizontal flipping and scaling jitter.

3. The semi-supervised instance segmentation method with pseudo-label quality decoupling and correction according to claim 1, characterized in that: The prediction results include category quality and mask quality, which are respectively expressed as: Among them, c k represents the category quality of the kth instance, x k represents the logit value of the category prediction of the kth instance, N represents the number of categories; m k represents the mask quality of the kth instance, HW represents the total number of all pixels in the mask, represents the mask logit value of each pixel in the kth instance, σ represents the sigmoid function, Indicates pixels with mask probability greater than 0.

5.

4. The semi-supervised instance segmentation method with pseudo-label quality decoupling and correction according to claim 3, characterized in that: The prediction results are filtered by the decoupled dual threshold mechanism, specifically by setting the category quality threshold c t and the mask quality threshold m t To achieve filtering, it is expressed as: Only the pseudo labels of instances that meet the conditions (C i ,M i ) will be retained for the training of the student model.

5. The semi-supervised instance segmentation method with pseudo-label quality decoupling and correction according to claim 4, characterized in that: The instance pseudo-label is dynamically modified for each pseudo-label (C i ,M i ) category modification includes the following steps: From the mask M i Extract image blocks This image patch is fed into CLIP’s visual encoder V ; The text description of the category Input CLIP text encoder CLIP T ; Calculate the similarity probability distribution of categories based on the output of CLIP It is expressed as: The similarity probability distribution of categories Class probability distribution generated by the teacher model Perform weighted fusion to obtain the final category correction probability distribution It is expressed as: Among them, the weight w decays with the number of training iterations; Based on the final category correction probability distribution Select the correction category for pseudo-labels It is expressed as: Among them, argmax means obtaining the probability distribution The category with the highest probability value.

6. The method for semi-supervised instance segmentation with pseudo-label quality decoupling and correction according to claim 5, characterized in that: The loss function is constructed by combining the prediction results and the corrected pseudo labels. The loss function includes the unlabeled data loss L unsup , expressed as: in, represents the classification loss, Q represents the number of instances predicted during training, The Hungarian algorithm matches the best matching pseudo labels selected by k predictions. is the predicted category probability of the student model for the i-th category of the k-th instance, represents the pseudo-label correction category after Hungarian matching selection; represents the mask loss, is the predicted mask probability of the student model for the i-th pixel of the k-th instance, It is the optimal mask pseudo-label selected by Hungarian algorithm matching; represents the uncertainty of the teacher model’s mask prediction probability for each pixel, represents the uncertainty of the matching selection after the Hungarian algorithm, represents the teacher model's mask prediction probability for each pixel.

7. The semi-supervised instance segmentation method with pseudo-label quality decoupling and correction according to claim 1, characterized in that: The training is used for a student model for instance segmentation. Unlabeled images are used for training. The unlabeled images are input into the student model after strong enhancement. The strong enhancement includes horizontal flipping, scaling jittering, color jittering, grayscale, Gaussian blurring and CutOut.

8. The semi-supervised instance segmentation method with pseudo-label quality decoupling and correction according to claim 1, characterized in that: Both the teacher model and the student model use the basic instance segmentation network Mask2Former.

9. A semi-supervised instance segmentation system with pseudo-label quality decoupling and correction, characterized in that: include: Prediction module, where the teacher model processes unlabeled images and generates prediction results; The filtering module filters the prediction results through a decoupled dual threshold mechanism and retains the prediction results that meet the conditions as instance pseudo labels; The correction module dynamically corrects the instance category of the instance pseudo label to obtain the corrected pseudo label; The training module combines the prediction results and the corrected pseudo labels to construct a loss function and train the student model for instance segmentation.

Images