A weakly supervised video instance segmentation method based on pixel erasing technology

Abstract

The present invention provides a weakly supervised video instance segmentation method based on pixel erasing technology. The method comprises: labeling the input original video data and image data; inputting the labeled video data and image data into the video instance segmentation VIS network, the VIS network jointly training the video data and image data, generating instance mask labels of the video data, and outputting updated video data; performing pixel erasing on the updated video data, inputting the pixel-erased video data and image data into the trained VIS network, the VIS network performing self-supervised joint training on the video data and image data, and outputting the category, instance ID, and mask prediction results of the original video data. The present invention can achieve results comparable to or even better than the most advanced fully supervised VIS work without requiring any manually labeled video instance masks. The pixel erasing method of the present invention can be applied to other weakly supervised segmentation tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of video content understanding, and in particular to a weakly supervised video instance segmentation method based on pixel erasing technology. Background Art

[0002] Video Instance Segmentation (VIS) aims to simultaneously detect, classify, segment, and track objects in videos and has been widely studied in recent years. Leveraging deep neural networks, existing methods such as CrossVIS, MaskProp, and STMask have achieved remarkable segmentation results in challenging scenarios such as severe crowding, occlusion, and motion blur. However, the success of these works often relies on dense, pixel-by-pixel annotations for each frame, which are very expensive to obtain. Furthermore, given the same annotation budget, VIS datasets typically have smaller scale and object diversity than traditional image segmentation datasets due to the temporal redundancy of videos (YouTube-VIS has 131k mask annotations and fewer than 5k unique instances, while the COCO dataset has over 886k mask annotations). Therefore, the labor-intensive and inefficient annotation requirements have limited the development of VIS.

[0003] In order to reduce the need for annotations in VIS, some recent methods have attempted to learn VIS in a weakly supervised manner without any video mask annotations. For example, FlowIRN uses only video category labels to learn VIS. However, it introduces too many cumbersome designs and has a significant performance gap with the fully supervised method. On the other hand, SOLO-Track chooses to learn VIS using an image dataset containing mask annotations and video data without any annotations. However, it requires that there is overlap in categories between image and video data, which limits its flexibility. At the same time, there is still a significant performance gap compared with the fully supervised method (the AP indicator gap on the YouTube-VIS 2019 dataset is more than 10%). The above problems limit the application of these methods in practical scenarios. Summary of the Invention

[0004] An embodiment of the present invention provides a weakly supervised video instance segmentation method based on pixel erasing technology to achieve effective instance segmentation of a video.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] A weakly supervised video instance segmentation method based on pixel erasing technology, comprising:

[0007] Label the input raw video data and image data;

[0008] The labeled video data and image data are input into the video instance segmentation VIS network. The VIS network jointly trains the video data and image data. The trained VIS network generates instance mask annotations for the video data and outputs the updated video data.

[0009] Perform pixel erasing on the updated video data and output the video data after pixel erasing;

[0010] The pixel-erased video data and image data are input into the trained VIS network, and the VIS network performs self-supervised joint training on the video data and image data, and outputs the category, instance ID and mask prediction results of the original video data.

[0011] Preferably, the labeling of the input original video data and image data includes:

[0012] Obtain the original video data to be instance decomposed, collect image data through the network, manually annotate the original video data frame by frame, and the annotation content includes instance category, instance detection box, and the ID of the same instance between each frame in the video. Manually annotate the image data, and the annotation content includes instance category, instance detection box, and instance mask, and output the annotated video data and image data.

[0013] Preferably, the labeled video data and image data are input into a video instance segmentation (VIS) network, the VIS network is jointly trained on the video data and image data, the trained VIS network generates instance mask annotations for the video data, and outputs updated video data, including:

[0014] The VIS network is established using the VIS model structure, and video data and image data are combined in a 1:1 ratio to form batch data. The batch data is passed into the VIS network. The video data and image data share the VIS network parameters. The video data uses a weak supervision method based on the detection box to supervise the segmentation training. The image data uses the standard cross entropy loss to supervise the segmentation results of the network. When there is a category overlap between the video data and the image instances in each batch, a pixel-level cycle consistency constraint is imposed between the same category instances of the video data and the image data. After several rounds of training, the video data involved in the training is tested to generate mask predictions, and the video detection box annotation is used to select the optimal mask as the final pseudo-mask annotation of the data. The updated video data is output, which includes the instance category, instance detection box, the ID of the same instance between each frame in the video, and the pseudo-mask annotation generated by the VIS network.

[0015] Preferably, when there is category overlap between the video data and the image instances in each batch, applying pixel-level cycle consistency constraints between the video data and the image data instances of the same category includes:

[0016] In each training batch, when the category label spaces of video data and image data partially overlap, pixel-level cycle consistency constraints are imposed between the same category instances of video data and image data. Given an image instance O from the same category, Img,i and video instance O Vid,j , for O Img,i For each pixel p in the foreground mask, calculate:

[0017]

[0018]

[0019] Where sim(p,q) represents the cosine similarity between pixel features p and q. If pixel p* falls in O Img,i If the image foreground mask is , then the cycle consistency constraint between the instances of the same category in the video data and the image data is considered to be satisfied.

[0020] Preferably, erasing pixels of the updated video data and outputting the video data after pixel erasure comprises:

[0021] The updated video data V t All instances in the video data V are randomly sampled. For each sampled instance object image, a rectangular erasure area is randomly selected. The rectangular erasure area is limited to the corresponding detection box surrounding the instance object. t The erased areas in the t , replace R with a random RGB value C t RGB pixels, hold video data V t The pseudo mask M t constant;

[0022] The training loss function for video segmentation is:

[0023]

[0024] where f θ (·) represents the segmentation network with θ as the parameter, ⊙ represents the multiplication of the corresponding pixels, Represents the true label Y and the predicted result Cross entropy loss between ;

[0025] Outputs the video data after pixel erasing.

[0026] Preferably, the pixel-erased video data and image data are input into a trained VIS network, and the VIS network performs self-supervised joint training on the video data and image data, and outputs the category, instance ID, and mask prediction results of the original video data, including:

[0027] The pixel-erased video data and image data are combined in a 1:1 ratio to form batch data, which is then input into the trained VIS network. The video data and image data share the VIS network parameters. When calculating the loss function, both the video and image data use masks to supervise their respective segmentation training. After the training is completed, the video test data is predicted, and the category, instance ID, and mask prediction results of the original video data are output.

[0028] As can be seen from the technical solutions provided by the embodiments of the present invention described above, this invention proposes a novel weakly supervised VIS training method that significantly reduces reliance on annotation while maintaining excellent VIS performance. Our MaskFreeVIS, without requiring any manually labeled video instance masks, can achieve comparable or even superior results to state-of-the-art fully supervised VIS work. Our pixel erasing method can be applied to other weakly supervised segmentation tasks.

[0029] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A processing flow chart of a weakly supervised video instance segmentation method based on pixel erasing technology provided by an embodiment of the present invention.

[0032] Figure 2 This is a test visualization of the present invention on the mainstream video instance segmentation dataset YouTube-VIS 2019. The three consecutive columns in each row are the same video sequence, and the foreground objects are predicted using a solid color mask.

[0033] Figure 3 Schematic diagram showing the effect of the pixel erasing technology according to an embodiment of the present invention.

[0034] Figure 4This is a comparison chart of the beneficial effects of the method of the present invention and the results of other video instance segmentation work. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0036] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0038] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0039] This paper proposes a weakly supervised video instance segmentation method based on pixel erasing technology, and a novel weakly supervised VIS training method to narrow the gap mentioned above, which strikes a balance between annotation cost and model performance. We do not need any video mask annotation, because mask annotation accounts for most of the annotation cost (each instance mask per frame takes about 79.2 seconds). Instead, we only use detection box annotations that are much more convenient and easy to obtain (each instance detection box annotation takes about 7 seconds) as well as category annotations and instance ID annotations, and take advantage of a large number of image datasets, such as COCO, which have a large number of ready-made and easy-to-use instance annotations, which can be used as auxiliary supervision signals to guide segmentation learning in VIS tasks. Please note that unlike existing weakly supervised VIS work, we do not require the category sets of images and videos to completely overlap, which makes our method more flexible and practical. Furthermore, by only annotating the detection boxes in the key frames and then propagating them to the remaining frames, the labeling cost can be further reduced.

[0040] Figure 1 The processing flow chart of a weakly supervised video instance segmentation method based on pixel erasing technology provided by an embodiment of the present invention is as follows: Figure 1 As shown, the processing steps include the following:

[0041] Step S10: label the input video data and image data.

[0042] Step S20: Input the labeled video data and image data into the VIS network. The VIS network jointly trains the video data and image data. The trained VIS network generates instance mask annotations for the video and outputs updated video data, including instance categories, instance detection boxes, IDs of the same instances between each frame in the video, and pseudo-mask annotations.

[0043] Step S30: Erasing pixels of the updated video data and outputting the erased video data.

[0044] Step S40: Input the erased video data and image data into the trained VIS network. The VIS network performs self-supervised joint training on the video data and image data. Both the video and image data use masks to supervise their respective segmentation training. After the training is completed, the VIS network outputs the category, instance ID and mask prediction results of the video data.

[0045] Specifically, step S10 includes: obtaining the original video data to be instance-decomposed, and collecting image data through the network. This image data set does not need to be a coherent sequence derived from the video, nor does it need to overlap with the video data. The original video data is manually annotated frame by frame. The annotation content includes instance category, instance detection box, and ID of the same instance between each frame in the video, but does not include instance mask annotation. The image data is also manually annotated. The annotation content includes instance category, instance detection box, and instance mask. The annotated video data and image data are output.

[0046] Specifically, the above step S20 includes: taking the above-mentioned annotated video and image data as input, and establishing a VIS network using the existing VIS model structure. The video data and image data are combined in a 1:1 ratio to form batch data, and the batch data is passed into the VIS network. The video data and image data share the VIS network parameters, but when calculating the loss function, the image data uses manually annotated masks to supervise the segmentation training, and the video data uses a weak supervision method based on detection boxes to supervise the segmentation training. When there is a category overlap between the video data and image instances in each batch, pixel-level cycle consistency constraints can be imposed between the same category instances of the video and image to further enhance the auxiliary supervision role of image mask annotation for video segmentation training. After several rounds of training, the video data participating in the training is tested to generate mask predictions, and the video detection box annotation is used to select the optimal mask as the final pseudo-mask annotation. The updated video data is output, including the instance category, instance detection box, the ID of the same instance between each frame in the video, and the pseudo-mask annotation generated by the VIS network.

[0047] First, the video and image data are jointly trained. Note that the video data does not contain mask annotations, while the additional image dataset contains rich mask annotations, but the category label space may not overlap with the category label space in the video. During training, each training batch contains both image and video data. For image data, the standard cross entropy loss is used to supervise the segmentation results of the network. For video data, since there are no mask annotations, we use the mapping loss in BoxInst to supervise the segmentation results of the network. Since we do not assume that these datasets share the same category label space, the network classification layer is defined separately for each dataset. After several rounds of training, the model can generate high-quality pseudo masks on the unlabeled video training set for subsequent self-training.

[0048] If the class label spaces of video and image data partially overlap, we can additionally use pixel-level cycle consistency constraints to strengthen the guidance of image data on video segmentation learning. Specifically, in each training batch, given image instances O from the same category Img,iand video instance O Vid,j , for O Img,i For each pixel p in the foreground mask, we can compute And further Where sim(p,q) represents the cosine similarity between pixel features p and q. If pixel p* falls within O Ing,i If the image foreground mask is based on the image, the cycle consistency constraint is considered to be satisfied, and we apply the InfoNCE loss to enforce feature similarity between matching pixel pairs. We find that this cycle-consistency-based image-to-video guidance not only produces better pseudo masks but also reduces training instability (smaller variance across multiple runs).

[0049] Specifically, the above step S30 includes: taking the video data updated in S20 as input, and performing a pixel erasing operation. All instances in the video data are randomly sampled, and each instance represents a unique individual in the video frame. For example, each person or car in the picture is an instance. For each sampled instance object image, we randomly select a rectangular erasing area, which is confined to the corresponding detection box surrounding the instance object. The image RGB value of the erased area is replaced with a random value, and the corresponding pseudo-mask annotation in the area is not changed. We also generate random areas within the entire image, that is, within the range including all instance areas and the background area other than that. Output the erased video data.

[0050] Randomly select regions in the video data and the detection box of each instance and erase the corresponding RGB values, replacing them with random values, while keeping the corresponding pseudo-mask unchanged. Random erasing can also be used to reduce the noise of the pseudo-mask to provide a stable training signal. After randomly erasing the RGB pixel values, the model becomes less sensitive to noise in the pseudo-mask due to the lack of corresponding RGB pixels. Therefore, pixel erasing improves the robustness of the model when learning from noisy pseudo-masks. In addition, we also experimentally demonstrate that the pixel erasing technique can be applied to other weakly supervised tasks that involve pseudo-mask training, such as weakly supervised image semantic segmentation and instance segmentation.

[0051] Using the above methods, our proposed weakly supervised video instance segmentation method (MaskFreeVIS) based on pixel erasing technology can achieve comparable or even better results than the most advanced fully supervised VIS work on the YouTube-VIS 2019 and YouTube-VIS 2021 datasets. At the same time, our MaskFreeVIS also exceeds the previous state-of-the-art weakly supervised VIS work by 9.2% in mAP.

[0052] Specifically, step S40 includes the following steps: using the erased video data and image data as input, and continuing to use the network trained in S20 as the initial network. The video data and image data are still combined in a 1:1 ratio to form small batches, which are then passed into the VIS network. The video data and image data share network parameters, and when calculating the loss function, both video and image data use masks to supervise their respective segmentation training. After training, predictions are made on the video test data. We do not modify the testing process of the VIS network used. The network will output the category, instance ID, and mask prediction of the video instance.

[0053] We first introduce how to leverage rich image data to guide weakly supervised VIS training. Next, we introduce the two training phases involved in MaskFreeVIS: pseudo-mask generation and pixel-erasing self-training.

[0054] This paper studies how to learn VIS networks in a weakly supervised manner, i.e., the expensive video mask annotations that constitute the vast majority of the VIS annotation cost are removed. Instead, we only retain the detection box annotations, which have much lower annotation costs, as well as the instance category and ID annotations. The annotation budget can be further reduced by only annotating the boxes in the keyframes and then propagating to the rest of the frames. While there have been some works exploring weakly supervised VIS learning, they still suffer from a significant performance gap compared to the state-of-the-art fully supervised methods. In this work, we attempt to narrow this gap by proposing a novel weakly supervised method for VIS networks that strikes a balance between annotation cost and model performance. Specifically, we leverage a large number of image datasets with rich mask annotations, such as COCO, which are easy to use and cheap to annotate, and can be used as auxiliary supervision information to guide segmentation learning in VIS. Unlike existing weakly supervised methods, we do not require the category label sets of image and video data to overlap with each other to keep the method flexible and practical. Formally, we have a video dataset D for learning VIS Vid , which contains instance categories, instance IDs, and detection box annotations, but does not contain any mask annotations. At the same time, we have another image dataset D Img , with instance-level annotations, including category, detection box, and mask annotations.

[0055] The proposed method does not require any video mask annotation and can be easily applied to practical application scenarios with abundant image mask annotations. During the pseudo-mask generation phase, the method uses abundant image mask annotations as auxiliary supervision information in the VIS learning process to help improve segmentation performance. The pseudo-mask is then predicted on video data without mask annotations.

[0056] The method of the present invention uses a pixel erasing method for self-training to mitigate the negative impact of noise in the pseudo-mask. Given a generated pseudo-mask, it can be directly used for subsequent training. However, directly using these pseudo-masks for model fine-tuning will inevitably face the potential risk of overfitting to noise due to imperfect pseudo-masks. To overcome this problem, the present invention proposes a simple and effective pixel erasing technique, which aims to mitigate the negative impact of noise when learning with noisy pseudo-masks.

[0057] We first perform a t We randomly sample object instances in the video data V. For each sampled object instance, we randomly select a rectangular erasing region that is confined to the corresponding detection box surrounding the object instance. t The erased areas in the t Next, we replace R with a random RGB value C t RGB pixels, while maintaining the pseudo mask M t unchanged. Therefore, the training loss function for video segmentation is:

[0058]

[0059] where f θ (·) represents the segmentation network with θ as the parameter, ⊙ represents the multiplication of the corresponding pixels, Represents the true label Y and the predicted result In fact, we not only generate random regions in the detection box of the instance, but also generate random regions in the entire image.

[0060] Subsequent experiments show that applying random erasing to manually annotated masks does not lead to any performance improvement, demonstrating that pixel erasing serves more than just a data augmentation tool. Rather, it helps mitigate the negative impact of learning with noisy pseudo-masks. When the pseudo-mask is incorrect, erasing the RGB pixel values ​​is equivalent to deleting the corresponding erroneous regions in image space, and training on such pseudo-masks does not lead to overfitting to the noise. On the other hand, when the pseudo-mask is correct, the model is forced to predict spatially correlated masks despite only being given partial RGB content of the instance, which improves the model's robustness when learning from noisy supervision.

[0061] Our pixel erasing technique is simple and does not require any architectural modifications. Therefore, it can also be easily applied to other weakly supervised tasks that involve learning with pseudo masks, such as weakly supervised image semantic segmentation or weakly supervised instance image segmentation.

[0062] Experimental results

[0063] (1) Training dataset and evaluation indicators

[0064] We evaluate our method on the YouTube-VIS 2019, YouTube-VIS 2021, and OVIS datasets. YouTube-VIS 2019 is the first large-scale VIS dataset, consisting of 2,238 training videos, 302 validation videos, and 343 test videos. Each video has an average duration of 4.61 seconds and covers 40 object categories. YouTube-VIS 2021 is an extended version of YouTube-VIS 2019. It also contains 40 categories, but with some changes to the category set. OVIS is a relatively new and more challenging dataset that focuses on video scenes with severe occlusions. It contains 25 categories, 607 training videos, 140 validation videos, and 154 test videos. We use the standard average precision (AP) for evaluation.

[0065] (2) Network deployment details

[0066] Our method builds on the state-of-the-art VIS method, Mask2Former-VIS. To handle long video sequences, we adapt Mask2Former-VIS's offline inference form, which processes the entire video simultaneously, to an online approach, achieving serial, frame-by-frame inference. During training, we optimize each frame independently. During inference, we compute feature similarity between instance predictions across frames to perform tracking. We conduct experiments using both offline and online Mask2Former-VIS to demonstrate the generalizability of our method.

[0067] We utilize 80k training images from the COCO dataset as auxiliary supervision for learning VIS. We set the image:video sample ratio in each batch to 1:1. We use a simple test-time augmentation method to generate pseudo masks (resize the image to 360, 480, 600, and 720 pixels on the shorter side, flip it horizontally, and average the predictions of these copies as the output). We only retain those pseudo masks that have the highest overlap with the detection bounding box annotations; mask predictions that fall outside the detection bounds are set to background. Unless otherwise specified, we perform random erasure according to the original random erasure data augmentation method, with the same hyperparameters as Mask2Former-VIS. Our method is implemented on the detectron2 framework, and the entire network is trained using 8 NVIDIA V100 GPUs, with each GPU processing 2 video data and 2 images at each iteration.

[0068] (3) Comparison of experimental results

[0069] Since this work aims to avoid pixel-by-pixel video mask annotations while learning a weakly supervised VIS network with comparable performance to fully supervised VIS, it is necessary to establish a strong fully supervised baseline to better evaluate the performance of MaskFreeVIS. We use the original offline Mask2Former-VIS and our modified online Mask2Former-VIS as representatives of fully supervised methods. For both models, we pre-load the weights of Mask2Former trained on the COCO instance segmentation dataset and then fine-tune them on fully annotated video data. Following standard practice in the field, we study the results using ResNet-50 and Swin-L as backbone networks for fair comparison.

[0070] Figure 4 The figure compares the beneficial effects of the proposed method with the effects of other video instance segmentation works. We first compare MaskFreeVIS with the state-of-the-art weakly supervised VIS methods. Note that SOLO-Track requires complete category overlap between the video and image datasets, so it is only evaluated on the 20-category subset of YouTube-VIS 2019 whose categories overlap with COCO. On the other hand, Flow-IRN is only trained on video data with classification labels. It does not use weights pre-trained on COCO and fine-tuned on the video dataset like most fully supervised VIS methods. As shown in Table 1, both SOLO-Track and Flow-IRN lag significantly behind the existing state-of-the-art fully supervised VIS methods. In contrast, our MaskFreeVIS significantly outperforms these two weakly supervised methods.

[0071] Compared with the fully supervised VIS method, our MaskFreeVIS does not have expensive video mask annotation, but can still be comparable to the fully supervised methods. More interestingly, MaskFreeVIS using ResNet-50 as the backbone network even surpasses all fully supervised methods on YouTube-VIS 2019. On the more challenging YouTube-VIS 2021 and OVIS datasets, despite the lack of video mask annotation, our MaskFreeVIS still achieved relatively good performance (using ResNet-50 as the backbone network on YouTube-VIS 2021 and OVIS, the gap with the fully supervised baseline is <1.8% AP and <2.4% AP respectively). In addition to using the CNN structure as the backbone network, we also show the results of using the transformer structure as the backbone network. Consistent with previous experimental results, MaskFreeVIS is comparable to the state-of-the-art fully supervised methods on all three datasets. We believe this is an important step towards maintaining the performance of VIS while reducing its expensive annotation cost.

[0072] Figure 2 This is a test visualization of the present invention on the mainstream video instance segmentation dataset YouTube-VIS 2019, as shown in Figure 2 As shown in Figure 2, the three consecutive columns in each row are the same video sequence, and the foreground objects are predicted using solid color masks. Despite being trained without video mask annotations, our MaskFreeVIS can still generate accurate segmentation masks.

[0073] (4) Ablation experiment

[0074] The main idea of ​​this work is to leverage the rich image dataset and mask annotations while learning VIS using only the bounding box level annotations on the video dataset. To verify the rationality of this, we construct a baseline that only uses video bounding box annotation supervision. Specifically, we use the BoxInst

[0075] Table 1 Experimental results on YouTube-VIS 2019 / 2021 and OVIS datasets

[0076]

[0077]

[0078] Mask2Former-VIS is supervised by the projection loss and pairwise loss in

[15] , which we call Mask2Former-VIS-bbox. As shown in Table 1, without additional image mask supervision, the model can only utilize low-level prior information, such as the detection box density prior or the color similarity prior between adjacent pixels. This introduces incorrect supervision information in videos with a large amount of object deformation, motion blur, and occlusion, resulting in poor model performance.

[0079] Next, we study the impact of pixel erasing when training with pseudo masks containing noise. As shown in Table 2, pixel erasing consistently improves VIS performance on all three datasets when using both backbone networks, demonstrating the universality of its functionality.

[0080] Figure 3 This figure shows a visualization of the effects of the pixel erasing technique used in an embodiment of the present invention. Directly using a pseudo-mask containing noise can lead to suboptimal training results. Our pixel erasing method can effectively alleviate network overfitting to noise. Figure 3 The first column is the input video data; the second column is the manually annotated real mask; the third column is the pseudo mask containing noise; the fourth column is the result of directly using the pseudo mask for training without pixel erasing, and then predicting the training set images. The network's prediction of the noisy area contains too many errors; the fifth column uses pixel erasing for training, which greatly alleviates the above phenomenon.

[0081] As expected, noisy pseudo masks produce false supervision and subsequently lead to suboptimal performance. On the other hand, pixel erasing techniques help mitigate the negative effects of learning with noisy pseudo masks.

[0082] Table 2 Comparison of pixel erasing effects

[0083]

[0084] We further sought to gain deeper insights into why pixel erasing works. As mentioned earlier, pixel erasing is technically equivalent to random erasing augmentation, a standard data augmentation tool for deep neural network training. Therefore, one could argue that pixel erasing plays the same role as data augmentation. However, as shown in the last row of Table 3, applying pixel erasing to real human-annotated masks does not lead to any performance improvement, suggesting that pixel erasing acts as more than just data augmentation. We also investigate different erasing targets in Table 3. We find that simultaneously removing color information and the corresponding pseudo-mask leads to a decrease in performance (45.7% → 44.5% AP). We believe that erasing the pseudo-mask results in a loss of effective supervision, especially when the pseudo-mask is correct. In contrast, erasing only color information while leaving the pseudo-mask unchanged forces the model to predict spatially correlated masks, which improves its robustness (45.7% → 47.5% AP).

[0085] In previous weakly supervised segmentation work, a common strategy for dealing with noisy pseudo-masks is to ignore the loss value in regions where the mask prediction is uncertain to prevent erroneous supervision information. Similarly, we also tried ignoring the loss value of randomly sampled regions instead of erasing the color values. However, we found that this strategy did not work.

[0086] Table 3 Analysis of the effect of pixel erasing

[0087]

[0088] (5) Other applications of pixel erasing methods

[0089] In the previous section, we demonstrated the effectiveness of pixel erasing in handling noisy pseudo-masks when training VIS. Next, we experimentally demonstrate that our pixel erasing technique is generalizable across different tasks and can be applied to other weakly supervised segmentation tasks that involve the use of pseudo-masks, such as weakly supervised instance segmentation and weakly supervised semantic image segmentation.

[0090] Weakly supervised instance segmentation aims to learn instance segmentation using only weak annotations. For this purpose, we assume that the model only has detection boxes and category labels available. We first use BoxInst, an excellent instance segmentation method based on detection box supervision, to generate pseudo masks on COCO. Next, we use CondInst to fine-tune on these pseudo masks, and the results are summarized in Table 4. We first notice that with a training epoch of 3, simple fine-tuning with these noisy pseudo masks leads to worse results, indicating that the network is overfitting to the wrong noise. In contrast, adding pixel erasing techniques during fine-tuning steadily improves network performance.

[0091] We also conducted experiments on weakly supervised semantic segmentation on the PASCAL VOC 2012 dataset, using two recent weakly supervised semantic segmentation works: L2G and RCA. We fine-tuned DeepLabv2 on the pseudo masks generated by L2G and RCA, followed by CRF post-processing. As shown in Table 5, pixel erasing techniques are also very effective in weakly supervised semantic segmentation.

[0092] Table 4 The role of pixel erasing in weakly supervised instance segmentation

[0093]

[0094] Table 5 The role of pixel erasing in weakly supervised semantic segmentation

[0095]

[0096] In summary, the present invention proposes a weakly supervised video instance segmentation method based on pixel erasing technology: the MaskFreeVIS method. This method can achieve comparable or even superior results to the fully supervised VIS method without manual pixel-level annotation of videos. This method leverages image datasets, which are more abundant and readily available than video data, and only requires bounding box-level annotation on the video dataset.

[0097] The MaskFreeVIS method of the present invention achieves a stable improvement (0.6-2.6% AP) in weakly supervised training of both online and offline VIS networks without changing the network structure. Furthermore, the core method of MaskFreeVIS, pixel erasing, also consistently improves weakly supervised image semantic segmentation and instance segmentation.

[0098] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0099] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0100] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0101] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A weakly supervised video instance segmentation method based on pixel erasing technology, characterized in that: include: Label the input raw video data and image data; The labeled video data and image data are input into the video instance segmentation VIS network. The VIS network jointly trains the video data and image data. The trained VIS network generates instance mask annotations for the video data and outputs the updated video data. Perform pixel erasing on the updated video data and output the video data after pixel erasing; Input the pixel-erased video data and image data into the trained VIS network, which performs self-supervised joint training on the video data and image data and outputs the category, instance ID, and mask prediction results of the original video data; The labeled video data and image data are input into the video instance segmentation VIS network, the VIS network performs joint training on the video data and image data, the trained VIS network generates instance mask annotations for the video data, and outputs updated video data, including: A VIS network is established using the VIS model structure, and video data and image data are combined in a 1:1 ratio to form batch data. The batch data is passed into the VIS network. The video data and image data share the VIS network parameters. The video data uses a weak supervision method based on the detection box to supervise the segmentation training. The image data uses a standard cross-entropy loss to supervise the segmentation results of the network. When there is a category overlap between the video data and the image instances in each batch, a pixel-level cycle consistency constraint is imposed between the same category instances of the video data and the image data. After several rounds of training, the video data involved in the training is tested to generate mask predictions, and the video detection box annotation is used to select the optimal mask as the final pseudo-mask annotation of the data. The updated video data is output, which includes the instance category, instance detection box, the ID of the same instance between each frame in the video, and the pseudo-mask annotation generated by the VIS network. The pixel-erased video data and image data are input into the trained VIS network. The VIS network performs self-supervised joint training on the video data and image data, and outputs the category, instance ID, and mask prediction results of the original video data, including: The pixel-erased video data and image data are combined in a 1:1 ratio to form batch data, which is then input into the trained VIS network. The video data and image data share the VIS network parameters. When calculating the loss function, both the video and image data use masks to supervise their respective segmentation training. After the training is completed, the video test data is predicted, and the category, instance ID, and mask prediction results of the original video data are output.

2. The method according to claim 1, characterized in that The labeling of the input original video data and image data includes: Obtain the original video data to be instance decomposed, collect image data through the network, manually annotate the original video data frame by frame, and the annotation content includes instance category, instance detection box, and the ID of the same instance between each frame in the video. Manually annotate the image data, and the annotation content includes instance category, instance detection box, and instance mask, and output the annotated video data and image data.

3. The method according to claim 1, characterized in that When there is category overlap between the video data and the image instances in each batch, pixel-level cycle consistency constraints are imposed between the video data and the image data instances of the same category, including: In each training batch, when the category label spaces of video data and image data partially overlap, pixel-level cycle consistency constraints are imposed between the same category instances of video data and image data. Given an image instance O from the same category, Img,i and video instance O Vid,j , for O Img,i For each pixel p in the foreground mask, calculate: Where sim(p,g) represents the cosine similarity between pixel features p and q. If pixel p * Falling on O Img,i If the image foreground mask is , then the cycle consistency constraint between the instances of the same category in the video data and the image data is considered to be satisfied.

4. The method according to claim 3, characterized in that Erasing pixels of the updated video data and outputting the video data after pixel erasure includes: The updated video data V t All instances in the video data V are randomly sampled. For each sampled instance object image, a rectangular erasure area is randomly selected. The rectangular erasure area is limited to the corresponding detection box surrounding the instance object. t The erased areas in the t , replace R with a random RGB value C t RGB pixels, hold video data V t The pseudo mask M t constant; The training loss function for video segmentation is: where f θ (·) represents the segmentation network with θ as the parameter, ⊙ represents the multiplication of the corresponding pixels, Represents the true label Y and the predicted result Cross entropy loss between ; Outputs the video data after pixel erasing.

Images