A semi-supervised instance segmentation method and system based on multi-scale learning and a storage medium

Abstract

The application relates to a kind of semi-supervised instance segmentation method, system and storage medium based on multi-scale learning, the angle of the classification label predicted under two scales and the angle of the mask are mutually referenced and corrected in the application, and the final pseudo label that should be reserved is determined by comprehensive score, so that the quantity of pseudo label on single scale is well supplemented and the quality is improved.The application also designs an area adaptive additional scale learning strategy, which allows the model to learn large area and small area pseudo label reserved according to the prediction in additional scale, effectively improving the learning effect of the model for pseudo label under additional down sampling.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image segmentation in computer vision and the field of semi-supervised learning in machine learning, and is a combined task of semi-supervised learning and instance segmentation, in particular to a semi-supervised instance segmentation method based on multi-scale learning, a system and a storage medium. BACKGROUND

[0002] The task setting of instance segmentation is to output pixel-level segmentation prediction for all instances in the picture. At present, the instance segmentation method based on direct segmentation has made great progress in speed and performance. Semi-supervised learning is to train the model through part of the labeled pictures and a large number of unlabeled pictures, and to further improve the performance of the model through the unlabeled pictures. However, the training of these models requires a large amount of pixel-level annotation information, which is time-consuming. Semi-supervised learning can well alleviate this problem.

[0003] At present, there are two basic methods of semi-supervised learning: consistency learning method and pseudo-label method. Recent work has well combined semi-supervised learning into the target detection task. For the semi-supervised instance segmentation task, one work has constructed a multi-stage framework, which better utilizes the information of the unlabeled pictures to improve the performance of semi-supervised instance segmentation by adding a more noise-resistant mask branch and a weight that preserves the edge information of the mask. Another work with the best performance so far has constructed an end-to-end framework, and the designed loss function can dynamically adjust the weights of the classification loss and the mask loss according to the quality of different pseudo-labels. These weights are the values learned by the pixel-level information of each mask.

[0004] In the task setting of semi-supervised instance segmentation, the teacher model reasons the unlabeled pictures to generate pseudo-labels, which are used to guide the training of the student model. Therefore, compared with the real labels of the pictures, the pseudo-labels generated by the teacher model in a single scale inevitably contain noise at the pixel level and the classification level, resulting in low quality of the pseudo-labels, which affects the convergence of consistency learning in semi-supervised learning and limits the improvement of the instance segmentation performance of the model. SUMMARY

[0005] In view of the problems existing in the prior art, the purpose of the present application is to provide a semi-supervised instance segmentation method based on multi-scale learning, a system and a storage medium, which introduces information of an additional down-sampling scale, and corrects and fuses the information generated in multiple scales according to the pseudo-labels, to improve the quality of the pseudo-labels.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is:

[0007] A semi-supervised instance segmentation method based on multi-scale learning, which adds a multi-scale pseudo label correction module and an area adaptive additional scale learning strategy; the method specifically comprises the following steps:

[0008] Step 1, randomly initialize student model and teacher model;

[0009] Step 2, input the labeled picture of the original scale into the student model to obtain the prediction result ;

[0010] Step 3, calculate the loss function by the label and the prediction result of the marked picture;

[0011] Step 4, weak data augmentation is performed on the unlabeled picture of the original scale and the down-sampled 0.5 times scale, and the prediction result of the weak data augmented unlabeled picture is obtained from the teacher model, finally obtaining the pseudo label of the original scale and the pseudo label of the down-sampled 0.5 times scale ;

[0012] Step 5, input the pseudo label of the original scale and the pseudo label of the down-sampled 0.5 times scale into the multi-scale pseudo label correction module to obtain the optimized pseudo label ;

[0013] In the step 5, after the pseudo label of the original scale and the pseudo label of the down-sampled 0.5 times scale are input into the multi-scale pseudo label correction module, the processing of the pseudo label correction module includes three steps: multi-scale pseudo label matching, cross-scale optimization and mixed pseudo label selection;

[0014] Step 6, strong data augmentation is performed on the unlabeled picture of the original scale, and the prediction result of the student model is obtained by inputting the strong data augmented unlabeled picture into the student model ;

[0015] Step 7, calculate the loss function of the unlabeled picture by the prediction result of the student model for the strong data augmented unlabeled picture and the optimized pseudo label ;

[0016] Step 8, strong data augmentation is performed on the unlabeled picture of the down-sampled 0.5 times scale, and the prediction result of the student model is obtained by inputting the strong data augmented unlabeled picture into the student model ;

[0017] Step 9, calculate the loss function of the unlabeled picture by using the area adaptive extra scale learning strategy Downsample 0.5 times scale to get pseudo labels , the prediction result of the student model for the data augmentation picture under the downsample 0.5 times scale And pseudo labels Calculate the loss function of the unlabeled picture by using the area adaptive extra scale learning strategy ;

[0018] In step 9, the loss function of the unlabeled picture is calculated by using the area adaptive extra scale learning strategy, which is as follows:

[0019] Through the area threshold All pseudo labels are divided into large area pseudo labels and small area pseudo labels, and the small area pseudo labels are matched with the prediction results under the downsample 0.5 times scale Do bipartite graph matching, and calculate the IoU of the small area pseudo label and the matching prediction mask corresponding to it, if the IoU is less than the threshold , discard the small area pseudo label, and greater than it is retained;

[0020] The large area and the retained small area pseudo labels will supervise the student model at extra scales respectively, and the losses are And , the formula is as follows:

[0021] (10)

[0022] (11)

[0023] (12)

[0024] Wherein, And The number of large area and retained small area pseudo labels respectively; Is the prediction of the model under the downsample 0.5 times scale, Is the pseudo label optimized by MPC after corresponding downsample, Is the Dice loss calculated by the mask, Is the cross entropy loss calculated by classification, Is the total loss under the unsupervised downsample resolution;

[0025] Step 10, weight the loss values obtained in steps 3, 7 and 9 to get the total loss , train the student model by gradient back propagation algorithm;

[0026] Step 11, integrate the student models in multiple time steps by EMA method exponential moving average, and update the parameters of the teacher model;

[0027] Step 12: Repeat steps 2 to 11 to continuously train the instance segmentation performance of the student model and the teacher model until the preset number of iterations is achieved;

[0028] Step 13: After training is complete, either the teacher model or the student model can be randomly selected to perform inference and prediction on the input image to obtain the instance segmentation result.

[0029] In step 3, the loss function The calculation is as follows:

[0030] (1)

[0031] in, It is the cross-entropy loss calculated for classification. It is the Dice loss calculated using masking.

[0032] The pseudo-label and Each includes a mask and Category tags and Classification confidence and IoU score and ;

[0033] Step 5.1: Multi-scale pseudo-label matching;

[0034] First, given the pseudo-labels at the original scale. The i-th mask in Pseudo-labels at a downsampling scale of 0.5 All masks Compare to find the pseudo-label at the j-th downsampling scale with the maximum IoU. This allows for the discovery of matching mask pairs across two scales.

[0035] If the maximum intersection-union ratio is greater than the threshold This is considered as the prediction masks at two different scales pointing to the same instance, forming Pairing;

[0036] Pseudo-labels that do not form pairs are retained; pseudo-labels that exist at the original scale but not at the downsampling scale are denoted as... ; Pseudo-labels that exist at the downsampling scale but not at the original scale are denoted as ;

[0037] Step 5.2, cross-scale optimization;

[0038] For the mask pairing formed in step 5.1 In pseudo-tags and There are corresponding category tags in the middle. And classification confidence level First, the pseudo-labels are optimized at the classification level, that is, the classification categories with low confidence are corrected to the classification categories with higher predicted classification scores at another scale.

[0039] Mask pairing Correspondingly, there are also IoU scores. Next, the mixed score is calculated by multiplying the classification confidence score and the IoU score. To evaluate the quality of the pseudomask; when the blending score of the pseudo-label masks at both scales is below a threshold. At the same time, the probability prediction maps corresponding to multi-scale masks are subjected to IoU score-guided weighted averaging and pixel-level fusion to obtain optimized mask-level pseudo-labels. ;

[0040] Step 5.3: Selecting mixed pseudo-tags;

[0041] Selective retention is performed on the mixed set composed of multi-scale pseudo-labels; there are three cases:

[0042] The first type is for pairing. The optimized mask-level pseudo-label obtained by pixel fusion in step 5.2 Preserve the optimized mask-level pseudo-labels And the category corresponding to the highest classification confidence is used as the classification label to obtain the corresponding pseudo-label. ;

[0043] The second case is for pairing. No pixel fusion mask was used in step 5.2, which means that at least one of the paired masks is of high quality and has a high blending score. Greater than the threshold Choose which mixed score to keep. Higher pseudo-tags yield corresponding pseudo-tags. ;

[0044] The last scenario is when no paired pseudo-tags are formed in step 5.1. and All are retained, and supplementation is made from the perspective of the number of pseudo-tags;

[0045] Finally, the optimized pseudo-tags were obtained. Recorded as:

[0046] (8).

[0047] In step 5.1, the pseudo label at the jth down-sampling scale with the maximum IoU is found is expressed as:

[0048] (2)

[0049] wherein, is a function of binarizing the predicted mask likelihood;

[0050] In step 5.2, the modification of the classification label pair and is expressed as:

[0051] (3)

[0052] In step 5.2, the mixed score of the pseudo label mask of the two scales is calculated as follows:

[0053] (4)

[0054] (5)

[0055] In step 5.2, the optimized mask-level pseudo label is:

[0056] (6)

[0057] In step 5.3, the single pseudo label in the pseudo label is

[0058] (7).

[0059] In step 7, the loss function is calculated as follows:

[0060] (9)

[0061] wherein, is the cross-entropy loss of the classification calculation, is the Dice loss of the mask calculation.

[0062] In step 10, the total loss is:

[0063] (13)

[0064] wherein, and is a hyperparameter for balancing the loss term.

[0065] The parameter update of the teacher model is as follows:

[0066] (14)

[0067] wherein, denotes the parameter of the teacher model in the n-th iteration, denotes the parameter of the student model in the n-th iteration, denotes the parameter of the teacher model in the (n-1)-th iteration, is a hyperparameter for adjusting the amplitude of the exponential moving average.

[0068] A semi-supervised instance segmentation system based on multi-scale learning includes a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the steps of a semi-supervised instance segmentation method based on multi-scale learning as described above.

[0069] A computer-readable storage medium has a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of a semi-supervised instance segmentation method based on multi-scale learning as described above.

[0070] After the above scheme, the present application designs a multi-scale pseudo label correction module to solve the noise problem of the pseudo label in the single scale of the training process in the prior art at the pixel level and the classification level. The pseudo label generated under the additional down-sampling scale is introduced to provide a reference for information combination. Specifically, the angles of the classification labels predicted under the two scales and the angles of the masks are mutually referenced and corrected, and the final pseudo label to be retained is determined through the hybrid score, so that the pseudo label in the single scale is well supplemented in quantity and optimized in quality. In addition, in order to make the generation quality of the pseudo label under the additional down-sampling scale higher, the present application adaptively trains the model under multiple scales and designs an area adaptive additional scale learning strategy. Specifically, to solve the incompatibility problem of small area pseudo labels in the down-sampling scale, the pseudo labels are divided into small area and large area according to the mask area, the small area pseudo label is matched with the prediction under the down-sampling scale, and the intersection over union of the mask is calculated. We discard part of the small area pseudo labels with low intersection over union, and retain the small area pseudo labels and all large area pseudo labels under the down-sampling scale to supervise the model respectively.

[0071] Based on the above, the present application has the following outstanding advantages:

[0072] (1) The present application optimizes the pseudo label with more quantity and better quality by information correction and fusion of the pseudo label generated under multiple scales during semi-supervised training.

[0073] (2) The application also designs an area-adaptive extra scale learning strategy, which effectively improves the learning effect of the model for pseudo labels under extra down sampling by respectively learning large area and small area pseudo labels reserved according to the prediction under the extra scale.

[0074] (3) The application conducts extensive experiments on COCO, Cityscapes and BDD100K data sets, and obtains that the method of the application achieves the most advanced performance of 23.4, 26.7, 30.7 and 33.0 mAP on the COCO data set under the settings of 1%, 2%, 5% and 10%. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is the overall framework diagram of the application;

[0076] Figure 2 is the principle diagram of area-adaptive extra scale learning;

[0077] Figure 3 is the optimization effect of the multi-scale pseudo label correction module for pseudo labels;

[0078] Figure 4 is a visual effect comparison diagram of the prior art and the application. DETAILED DESCRIPTION

[0079] The application discloses a semi-supervised instance segmentation method based on multi-scale learning, which adds a multi-scale pseudo label correction (MPC) module and an area-adaptive extra scale learning (AAL) strategy, and effectively improves the quality of pseudo labels generated by the model under extra down sampling by mutually correcting and combining information of pseudo labels under multi-scale.

[0080] The method of the application specifically includes the following steps:

[0081] Step 1, randomly initialize the student model and the teacher model.

[0082] Step 2, input the labeled picture of the original scale into the student model to obtain the prediction result .

[0083] Step 3, calculate the loss function by the label and the prediction result of the labeled picture:

[0084] (1)

[0085] wherein, is the cross-entropy loss of classification calculation, is the mask computation Dice loss.

[0086] Step 4, weak data augmentation is performed on the original scale and the down-sampled 0.5 times scale unlabeled pictures, the prediction results of the weak augmented unlabeled pictures are obtained from the teacher model, and finally the pseudo labels under the original scale and the pseudo labels under the down-sampled 0.5 times scale are obtained and the pseudo labels under the down-sampled 0.5 times scale respectively. and respectively contain mask and , classification label and , classification confidence and , and IoU score and .

[0087] Step 5, the pseudo labels under the original scale and the pseudo labels under the down-sampled 0.5 times scale are input into the multi-scale pseudo label correction (MPC) module to obtain the optimized pseudo labels .

[0088] For the pseudo labels generated under the original scale, the pixel-level mask may be incomplete, and the classification label may also be misclassified. However, under the corresponding down-sampled 0.5 times scale, the corresponding pixel-level mask and classification are better. In addition, in terms of the quality of the pseudo labels, for the same instance segmented under different scales, there are a considerable number of masks that are more complete under the down-sampled 0.5 times scale; in terms of the number of pseudo labels, a considerable number of masks that cannot be segmented under the original scale can be segmented under the down-sampled 0.5 times scale. In order to fully exploit the information of the pseudo labels under the down-sampled 0.5 times scale to optimize the pseudo labels under the original scale, the present application adds a multi-scale pseudo label correction (MPC) module. After the pseudo labels under the original scale and the pseudo labels under the down-sampled 0.5 times scale are input into the multi-scale pseudo label correction module, the processing of the pseudo label correction module includes three main steps: multi-scale pseudo label matching, cross-scale optimization and mixed pseudo label selection.

[0089] Step 5.1, multi-scale pseudo label matching.

[0090] The existing pseudo labels under the original scale are , wherein a single pseudo label is denoted as , ; the pseudo labels under the down-sampled 0.5 times scale are , wherein a single pseudo label is denoted as , ; the mask in the pseudo label at original scale where the single mask is denoted as , ; the mask in the pseudo label at down-sampled 0.5 times scale where the single mask is denoted as , .

[0091] Given the pseudo label at original scale the i-th mask in is compared with all masks in the pseudo label at down-sampled 0.5 times scale to find the j-th pseudo label at down-sampled 0.5 times scale with the largest Intersection over Union (IoU) , so that the matched mask pair in two scales can be found, which can be expressed as:

[0092] (2)

[0093] where (. ) is a function to binarize the predicted mask likelihood.

[0094] If the largest IoU is greater than a threshold , it is considered that the predicted masks in two scales point to the same instance, forming a pair of .

[0095] For the pseudo labels that do not form a pair, it means that these pseudo labels can be segmented in one of the scales, but not in the other scale. Among them, the pseudo labels that have in the original scale but not in the down-sampled 0.5 times scale are denoted as ; the pseudo labels that have in the down-sampled 0.5 times scale but not in the original scale are denoted as , which are reserved for subsequent processing.

[0096] Step 5.2, cross-scale optimization;

[0097] For the mask pairs formed in step 5.1 , there are corresponding classification label pairs and classification confidence pairs . ​​The pseudo labels are first optimized at the classification level, i.e. the classification category with lower confidence is revised to the classification category with higher predicted classification score in another scale. Specifically, for a pair of original scale and down-sampled 0.5 times scale pseudo labels, if the classification confidence in the pseudo label in one scale is less than that in the other scale, the classification label in the pseudo label in this scale should be revised to the classification label in the pseudo label in the other scale. For the revision of and , it can be expressed as:

[0098] (3)

[0099] Masked pair , corresponding also has IoU score pair . Next, the hybrid score is calculated by multiplying the classification confidence and the IoU score to evaluate the quality of the pseudo mask:

[0100] (4)

[0101] (5)

[0102] When the hybrid scores of the pseudo label masks in the two scales are both lower than the threshold , it represents that the masks in the two scales may have lower quality and exist noise, so the mask information in multiple scales is selected to be fused. Since the IoU score can effectively reflect the integrity of the mask, the IoU score guided weighted average is performed on the probability prediction maps corresponding to the multiple scale mask pairs to perform pixel-level fusion, and the optimized mask-level pseudo label is obtained:

[0103] (6)

[0104] Step 5.3, hybrid pseudo label selection;

[0105] In order to obtain the final optimized pseudo label, the hybrid set composed of multiple scale pseudo labels is selectively reserved. There are three cases: the first is for the pair , the mask obtained by pixel fusion in the second step , wherein is the single mask-level pseudo label in , the fused mask and the category corresponding to the highest classification confidence are reserved as the classification label, and the corresponding pseudo label is obtained; the second case is for the pair , the mask without pixel fusion in the second step, which represents that at least one mask in the pair has high quality, and its hybrid score is greater than the threshold , and that one is selected to be reserved Higher pseudo labels, get , The single pseudo label in is:

[0106] (7)

[0107] The last case is that no pair of pseudo labels is formed in the first step and , which represents that they are only generated in a single scale, all reserved, and supplemented in terms of the number of pseudo labels.

[0108] The final pseudo labels obtained after optimization by the MPC module can be recorded as:

[0109] (8)

[0110] Step 6, perform strong data augmentation on the unlabeled picture in the original scale, input the strongly augmented unlabeled picture into the student model to obtain the prediction result .

[0111] Step 7, calculate the loss function of the unlabeled picture using the prediction result of the student model for the strongly augmented picture in the original scale and the optimized pseudo label :

[0112] (9)

[0113] wherein, is the cross-entropy loss of classification calculation, is the Dice loss of mask calculation.

[0114] Step 8, perform strong data augmentation on the unlabeled picture in the down-sampled 0.5 times scale, input the strongly augmented unlabeled picture into the student model to obtain the prediction result .

[0115] Step 9, down-sample the optimized pseudo label to the corresponding scale to obtain , calculate the loss function of the unlabeled picture using the prediction result of the student model for the strongly augmented picture in the down-sampled 0.5 times scale and using the area adaptive additional scale learning (AAL) strategy .

[0116] In addition to the original scale, additional model inference and supervision are performed at the down-sampling 0.5 scale. Through the method of supervised training, the segmentation ability of the model at the corresponding scale can be effectively improved, and a more reliable reference is provided for the optimization of the original scale pseudo label. However, due to the poor segmentation performance of the model at the low resolution scale for small area instances, the problem of incompatible label area and training scale occurs, making the convergence of the model very challenging.

[0117] To solve this problem, the present application proposes an area adaptive strategy. By setting an area threshold All pseudo labels are divided into large area pseudo labels and small area pseudo labels. The small area pseudo labels are matched with the prediction results of the down-sampling 0.5 scale A bipartite graph matching is performed, and after obtaining the matching result, the IoU of the small area pseudo label and the corresponding matching prediction mask is calculated. If the IoU is less than a threshold , the small area pseudo label is discarded, and if it is greater than the threshold , it is retained.

[0118] Selective retention is to avoid the negative interference of the model forced to fit the pseudo label on the training process when the student model's prediction of a small area pseudo label at the down-sampling 0.5 scale is too bad, thereby increasing the stability of convergence.

[0119] The large area and the retained small area pseudo labels will supervise the student model at the additional scale, and the losses are and , respectively, and the formulas are as follows:

[0120] (10)

[0121] (11)

[0122] (12)

[0123] wherein, and are the number of large area and retained small area pseudo labels, respectively; is the prediction of the model at the down-sampling 0.5 scale, is the pseudo label optimized by MPC corresponding to the down-sampling, is the Dice loss calculated by the mask, is the cross-entropy loss calculated by the classification, is the total loss at the unsupervised down-sampling resolution.

[0124] Step 10, the loss values obtained in steps 2, 6 and 8 are weighted and summed to obtain the total loss , and the student model is trained by gradient backpropagation algorithm.

[0125] (13)

[0126] wherein, and are hyperparameters for balancing the loss terms.

[0127] Step 11, integrate the student model in multiple time steps by the EMA method, update the parameters of the teacher model, the formula is as follows:

[0128] (14)

[0129] wherein, denotes the parameters of the teacher model in the n-th iteration, denotes the parameters of the student model in the n-th iteration, denotes the parameters of the teacher model in the n-1-th iteration, is a hyperparameter for adjusting the amplitude of EMA, which is set to 0.01 in this paper.

[0130] Step 12, repeat steps 2 to 11, and constantly train the performance of the student model and the teacher model instance segmentation until the preset number of iterations is reached.

[0131] Step 13, after the training is completed, the teacher model or the student model is randomly selected to perform inference prediction on the input picture to obtain the instance segmentation result.

[0132] Based on the above method, the quality of the pseudo label generated by the model under the additional down-sampling 0.5 times scale can be effectively improved. As shown in Figure 3 , the initial pseudo label and the pseudo label optimized by the MPC module proposed in the application are compared by visualization. Compared with the pseudo label without optimization, the classification label of the pseudo label optimized by the MPC is more accurate, and the noise of the mask pixel can be effectively reduced. As shown in Figure 4 , the instance segmentation performance of the application and the currently optimal method PAIS is compared by visualization. Compared with PAIS, the accuracy and recall rate of the instance segmentation performance of the application MPIS are higher.

[0133] In order to better illustrate the effect achieved by the application, the following will be verified by experiment.

[0134] Instance segmentation is an important direction of development in the field of computer vision, such as autonomous driving and robot navigation. The task of instance segmentation is not to detect targets alone or assign class labels to pixels, but to unify these tasks together, so it has become one of the most challenging tasks in computer vision. The semi-supervised instance segmentation task is to alleviate the problem that it is not easy to obtain pixel-level labeled pictures, and to enhance the performance of the instance segmentation model by using labeled pictures and unlabeled pictures.

[0135] The present application is implemented on K-Net, and experiments are performed on COCO dataset, Cityscapes dataset and BDD100K dataset. The COCO dataset contains 123K images with 80 instance labels, and 1%, 2%, 5% and 10% of the COCO training set are used as labeled pictures, and the rest are used as unlabeled pictures; the Cityscapes dataset contains 5K images, and 5%, 10%, 20% and 30% of the Cityscapes training set are used as labeled pictures, and the rest are used as unlabeled pictures; the BDD100K dataset uses 7K images as labeled pictures and 63K images as unlabeled pictures. The experimental results of the COCO dataset are shown in Table 1. The experimental results of the Cityscapes dataset are shown in Table 2. The experimental results of the BDD100K dataset are shown in Table 3. The main comparison methods of the present application are DD, Noisy Boundaries, Polite Teacher and PAIS: DD generates pseudo labels by applying various transformations to the unlabeled data on the same teacher model; Noisy Boundaries introduces a boundary preservation map to learn detailed information in the boundary related region of the pseudo label; Polite Teacher introduces IoU score, and the classification score is used to jointly filter the pseudo label; PAIS uses IoU score and classification score to weight the learning of the mask and classification of the pseudo label.

[0136] Table 1. Experimental results on the COCO dataset.

[0137]

[0138] Table 2. Experimental results on the Cityscapes dataset.

[0139]

[0140] Table 3. Experimental results on the BDD100K dataset.

[0141]

[0142] In Table 1, to clearly show the performance improvement of semi-supervised learning on COCO dataset using the method of the present application, the following models are trained for fair comparison: 1) models trained using only 1%, 2%, 5% and 10% of the labeled images randomly sampled, i.e. K-Net (supervised); 2) models trained using 1%, 2%, 5% and 10% of the labeled images randomly sampled and using the same data augmentation applied in the semi-supervised setting, i.e. K-Net (supervised); 3) models trained using 1%, 2%, 5% and 10% of the labeled images randomly sampled and the rest of the unlabeled images by implementing the present application.

[0143] After adopting the method of the present application, the instance segmentation performance is significantly improved, reaching the optimal performance of the current semi-supervised instance segmentation in the partially labeled experimental setting of COCO. For example, when there is only 1% and 2% of the labeled data, the current optimal performance is achieved by PAIS implemented on Mask RCNN, reaching mAP of 21.2 and 25.0. The implementation of the present application on K-Net can reach mAP of 23.4 and 26.7, respectively, exceeding the mAP of SOTA by 2.2 and 1.7. When the labeled data is further increased, the current SOTA is PAIS implemented on K-Net, reaching mAP of 31.1. The implementation of MPIS on K-Net can reach mAP of 33.0, exceeding the mAP of PAIS by 1.9, reaching the optimal performance of the current semi-supervised instance segmentation in the partially labeled experimental setting of COCO.

[0144] In Table 2, to clearly show the performance improvement of semi-supervised learning on Cityscapes dataset using the present application, the following models are trained for fair comparison: 1) models trained using only 5%, 10%, 20% and 30% of the labeled images randomly sampled, i.e. K-Net (supervised); 2) models trained using 5%, 10%, 20% and 30% of the labeled images randomly sampled and using the same data augmentation applied in the semi-supervised setting, i.e. K-Net (supervised); 3) models trained using 5%, 10%, 20% and 30% of the labeled images randomly sampled and the rest of the unlabeled images by the present application.

[0145] After adopting the method of the present application, the instance segmentation performance is significantly improved, especially when the number of labeled images is limited. For example, when there is only 5% of the labeled data, the present application improves the mAP by 8.1 and 2.9, respectively, compared with the partially labeled fully supervised K-Net and PAIS in the same setting, reaching mAP performance of 13.1. When the labeled data is increased to 30%, the present application improves the mAP by 5.1 and 1.6, respectively, compared with the partially labeled fully supervised K-Net and PAIS in the same setting, reaching mAP performance of 25.5.

[0146] In Table 3, to clearly show the performance improvement of semi-supervised learning on BDD100K dataset using the present application, the following models are trained for fair comparison: 1) a model trained using only 7K labeled images sampled randomly, i.e., K-Net (supervised); 2) a model trained using 7K labeled images sampled randomly and applying the same data augmentation in the semi-supervised setting, i.e., K-Net (supervised); 3) a model trained using 7K labeled images sampled randomly and 63K unlabeled images by the present application.

[0147] After adopting the method of the present application, the instance segmentation performance is significantly improved, reaching the optimal performance of current semi-supervised instance segmentation in the BDD100K partial labeled experimental setting. Specifically, compared with the hybrid supervised method ShapeProp (using 7K labeled and 67K framed labeled pictures) and the semi-supervised method Noisy Boundaries (using 7K labeled and 67K unlabeled pictures) and PAIS (using 7K labeled and 63K unlabeled pictures), MPIS can reach the optimal performance of 27.9 in the training setting using 7K labeled and 63K unlabeled pictures.

[0148] The present application also provides a semi-supervised instance segmentation system based on multi-scale learning, characterized in comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to realize the steps of a semi-supervised instance segmentation method based on multi-scale learning.

[0149] The present application also provides a computer readable medium, which can be contained in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device.

[0150] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to realize the method described in the above embodiments.

[0151] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units.

[0152] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0153] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such modifications and changes as come within the scope of the present disclosure and the general principles thereof.

[0154] It should be understood that the present disclosure is not limited to the precise structures described above and illustrated in the drawings and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A semi-supervised instance segmentation method based on multi-scale learning, characterized in that: The method adds a multi-scale pseudo-label correction module and an additional scale learning strategy for area adaptation; the method specifically includes the following steps: Step 1: Randomly initialize the student model and teacher model; Step 2: Input the original scale labeled images into the student model to obtain the prediction results. ; Step 3: Using tags on labeled images and prediction results Calculate the loss function ; Step 4: Perform weak data augmentation on the unlabeled images at the original scale and the downsampled scale by 0.5 times. Obtain the prediction results of the weakly data-augmented unlabeled images from the teacher model, and finally obtain the pseudo-labels at the original scale. And pseudo-labels at a downsampling scale of 0.

5. ; Step 5: Remove pseudo-labels at the original scale. and pseudo-labels at a downsampling scale of 0.5 The input is fed into the multi-scale pseudo-label correction module to obtain the optimized pseudo-label. ; In step 5, the pseudo-labels at the original scale and pseudo-labels at a downsampling scale of 0.5 After the input is fed into the multi-scale pseudo-label correction module, the processing of the pseudo-label correction module includes three steps: multi-scale pseudo-label matching, cross-scale optimization, and hybrid pseudo-label selection. Step 6: Perform strong data augmentation on the unlabeled images at the original scale, and input the augmented unlabeled images into the student model to obtain the prediction results. ; Step 7: Calculate the student model's prediction results for the strongly data-augmented unlabeled images. And optimized pseudo tags Calculate the loss function for unlabeled images ; Step 8: Perform strong data augmentation on the unlabeled images downsampled at a scale of 0.5, and input the strongly augmented unlabeled images into the student model to obtain the prediction results. ; Step 9: Optimize the pseudo-tags Pseudo-labels are obtained by downsampling by 0.5 times. The prediction results of the student model for strongly data-enhanced images downsampled by 0.5 times are presented. and pseudo-tags Calculate the loss function for unlabeled images using an area-adaptive extra-scale learning strategy. ; In step 9, the loss function for the unlabeled image is calculated using an area-adaptive additional scale learning strategy as follows: By area threshold All pseudo-labels are divided into large-area pseudo-labels and small-area pseudo-labels. The small-area pseudo-labels are compared with the prediction results at a downsampling scale of 0.

5. After performing bipartite graph matching and obtaining the matching results, calculate the IoU between the small-area pseudo-label and its corresponding matching prediction mask. If the IoU is less than a threshold... Discard the small pseudo-label, and keep the larger ones; Large-area and preserved small-area pseudo-labels will supervise the student model at additional scales, with losses respectively. and The formula is as follows: (10) (11) (12) in, and These represent the number of pseudo-tags in the large area and the number of pseudo-tags retained in the small area, respectively. This is the model's prediction at a downsampling scale of 0.

5. These are the pseudo-labels optimized by MPC after downsampling. It is the Dice loss calculated using the mask. It is the cross-entropy loss calculated for classification. It is the total loss at the unsupervised downsampling resolution; Step 10: Calculate the total loss by weighted summation of the loss values ​​obtained in steps 3, 7, and 9. The student model was trained using the gradient backpropagation algorithm. Step 11: Integrate the student models across multiple time steps using the EMA method with exponential moving average, and update the parameters of the teacher model; Step 12: Repeat steps 2 to 11 to continuously train the instance segmentation performance of the student model and the teacher model until the preset number of iterations is achieved; Step 13: After training is complete, either the teacher model or the student model can be randomly selected to perform inference and prediction on the input image to obtain the instance segmentation result.

2. The semi-supervised instance segmentation method based on multi-scale learning according to claim 1, characterized in that: In step 3, the loss function The calculation is as follows: （1） in, It is the cross-entropy loss calculated for classification. It is the Dice loss calculated using masking.

3. The semi-supervised instance segmentation method based on multi-scale learning according to claim 1, characterized in that: The pseudo-label and Each includes a mask and Category tags and Classification confidence and IoU score and ; Step 5.1: Multi-scale pseudo-label matching; First, given the pseudo-labels at the original scale. The i-th mask in Pseudo-labels at a downsampling scale of 0.5 All masks Compare to find the pseudo-label at the j-th downsampling scale with the maximum IoU. This allows for the discovery of matching mask pairs across two scales. If the maximum intersection-union ratio is greater than the threshold This is considered as the prediction masks at two different scales pointing to the same instance, forming Pairing; Pseudo-labels that do not form pairs are retained; pseudo-labels that exist at the original scale but not at the downsampling scale are denoted as... ; Pseudo-labels that exist at the downsampling scale but not at the original scale are denoted as ; Step 5.2, cross-scale optimization; For the mask pairing formed in step 5.1 In pseudo-tags and There are corresponding category tags in the middle. And classification confidence level First, the pseudo-labels are optimized at the classification level, that is, the classification categories with low confidence are corrected to the classification categories with higher predicted classification scores at another scale. Mask pairing Correspondingly, there are also IoU scores. Next, the mixed score is calculated by multiplying the classification confidence score and the IoU score. To evaluate the quality of the pseudomask; when the blending score of the pseudo-label masks at both scales is below a threshold. At the same time, the probability prediction maps corresponding to multi-scale masks are subjected to IoU score-guided weighted averaging and pixel-level fusion to obtain optimized mask-level pseudo-labels. ; Step 5.3: Selecting mixed pseudo-tags; Selective retention is performed on the mixed set composed of multi-scale pseudo-labels; there are three cases: The first type is for pairing. The optimized mask-level pseudo-label obtained by pixel fusion in step 5.2 Preserve the optimized mask-level pseudo-labels And the category corresponding to the highest classification confidence is used as the classification label to obtain the corresponding pseudo-label. ; The second case is for pairing. No pixel fusion mask was used in step 5.2, which means that at least one of the paired masks is of high quality and has a high blending score. Greater than the threshold Choose which mixed score to keep. Higher pseudo-tags yield corresponding pseudo-tags. ; The last scenario is when no paired pseudo-tags are formed in step 5.

1. and All are retained, and supplementation is made from the perspective of the number of pseudo-tags; Finally, the optimized pseudo-tags were obtained. Recorded as: (8)。 4. The semi-supervised instance segmentation method based on multi-scale learning according to claim 3, characterized in that: In step 5.1, the pseudo-label at the j-th downsampling scale with the maximum IoU is found. The expression is as follows: (2) in, (.) is a function that binarizes the predicted mask probability; In step 5.2, the classification label pairs are... and The revised statement is as follows: (3) In step 5.2, the combined score of the two scales of pseudo-label masks is calculated as follows: (4) (5) In step 5.2, the optimized mask-level pseudo-label for: (6) In step 5.3, the pseudo-tag A single pseudo-tag in for: (7)。 5. The semi-supervised instance segmentation method based on multi-scale learning according to claim 1, characterized in that: In step 7, the loss function The calculation is as follows: （9） in, It is the cross-entropy loss calculated for classification. It is the Dice loss calculated using masking.

6. The semi-supervised instance segmentation method based on multi-scale learning according to claim 1, characterized in that: In step 10, the total loss for: (13) in, and It is a hyperparameter used to balance the loss term.

7. The semi-supervised instance segmentation method based on multi-scale learning according to claim 1, characterized in that: The parameters of the teacher model are updated as follows: （14） in, This represents the parameters of the teacher model in the nth iteration. This represents the parameters of the student model in the nth iteration. This represents the parameters of the teacher model in the (n-1)th iteration. It is a hyperparameter used to adjust the magnitude of the exponential moving average.

8. A semi-supervised instance segmentation system based on multi-scale learning, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a semi-supervised instance segmentation method based on multi-scale learning as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of a semi-supervised instance segmentation method based on multi-scale learning as described in any one of claims 1-7.

Images