Video object segmentation method based on weakly supervised learning

Abstract

The present invention provides a video target segmentation method and device based on weakly supervised learning, the method comprising the steps of: S1, inputting the original video and positioning frame: inputting the original video, extracting frames to obtain two-dimensional images of each frame, and obtaining the target positioning frame of each frame, and performing positioning at the positioning frame level; cutting on the original image according to the positioning frame to obtain the final cut image; S2, inputting the cut image and the target positioning frame into the pipeline segmentation model to obtain the segmentation mask of the current frame, i.e., the pseudo label; S3, training the video target segmentation model with the pseudo label. The present invention can achieve higher segmentation accuracy, improve the quality of pseudo labels, improve the problems of poor continuity, object occlusion and missed detection caused by the original independent segmentation of each video frame from the detection frame to the segmentation mask, and the model is more robust.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video target segmentation method based on weakly supervised learning. Background Art

[0002] For the task of video object segmentation, the mainstream method is now based on fully supervised video object segmentation, including space-time memory models (Space-Time Memory Networks, STM), collaborative video object segmentation by foreground-background integration (Collaborative video object segmentation by foreground-background integration, CFBI), and learning what to learn for video objects segmentation (LWL). This type of method requires a large number of precisely annotated video datasets during the training phase, that is, these videos need to be annotated frame by frame and pixel by pixel. In the prediction phase, the segmentation mask of the first frame of each video needs to be provided as a template to tell the model which objects need to be tracked. Although fully supervised learning methods can achieve higher segmentation accuracy, this type of method overly relies on video training sets that are precisely annotated frame by frame and pixel by pixel, and the cost of manual annotation is too high. In addition, frames are often similar, resulting in a lot of duplication of work in annotation.

[0003] Another similar work is unsupervised video object segmentation that only uses unlabeled video learning. The training process of this type of method is unsupervised, and it mainly uses the natural attributes of the video (such as temporal consistency, color, etc.) as supervision signals to learn high-level semantic information, which is then used for evaluation on video object segmentation tasks. Or it is to transfer the feature extraction method learned on static images to videos and use video semantic learning. This type of method no longer relies on large-scale accurately labeled datasets, but because there is no guidance from the first frame segmentation mask, their final effect is still far behind the fully supervised video object segmentation method.

[0004] At present, some researchers have proposed weakly supervised video object segmentation, that is, using labels with coarser granularity than segmentation masks (such as positioning boxes, polygons, points) for training, so that the model can learn the segmentation function. For example, researcher Paul Voigtlaender proposed a semi-automatic video annotation method, which uses a bounding box to segmentation mask (box2seg for short) model, first using the positioning box to generate a segmentation mask as a pseudo-label, and then manually refining the pseudo-label, and finally using it to train the video object segmentation network. However, the disadvantage is that the detection box to segmentation mask model they used is trained on static images, and the segmentation effect is unstable; on the other hand, they introduced manual interactive correction, and the cost is still high.

[0005] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the content included in the background section is not an admission by the applicant that the prior art is available. Summary of the invention

[0006] The purpose of the present invention is to provide a video target segmentation method based on weakly supervised learning, so as to overcome the problems that the traditional detection frame to segmentation mask model cannot utilize temporal information, resulting in unstable segmentation effect and introducing manual interactive correction, resulting in high cost.

[0007] In order to achieve the above-mentioned invention object, the present invention adopts the following technical solutions:

[0008] The video object segmentation method based on weakly supervised learning includes the following steps:

[0009] S1. Input original video and positioning frame: Input original video, extract frames to obtain two-dimensional images of each frame, and also obtain target positioning frames of each frame, and perform positioning at the positioning frame level; cut the original image according to the positioning frame to obtain the final cut image;

[0010] S2, inputting the cut image and the target positioning frame into the pipeline segmentation model to obtain a segmentation mask of the current frame, i.e., a pseudo label;

[0011] S3. Training a video object segmentation model using the pseudo labels.

[0012] Further:

[0013] In step S1, the method for obtaining the final cut image is: expanding 20% ​​of the pixels outward according to the positioning frame, cutting on the original image, and obtaining the final cut image.

[0014] Further:

[0015] The pipeline segmentation model is a fully convolutional neural network architecture that fully utilizes the temporal information of the input sequence through a non-local module to capture long-term dependencies in space and time; the non-local module consists of a two-part architecture of a non-local block and a split encoder, wherein the split encoder includes a query encoder and a reference encoder, and the architecture can apply the image semantic information and prediction results of the previous frame to the prediction of the current frame.

[0016] Further:

[0017] In step S2, obtaining the segmentation mask of the current frame includes the following steps:

[0018] Input the current frame + positioning frame, the true segmentation mask and the previous frame segmentation mask, where the current frame + positioning frame is encoded by the query encoder, the true segmentation mask and the previous frame segmentation mask are encoded by the reference encoder, and then the feature maps obtained by the query encoder and the reference encoder are processed by the non-local block and then decoded by the decoder to obtain the segmentation mask of the current frame.

[0019] Further:

[0020] The specific pipeline segmentation model structure is as follows:

[0021] The input includes the current frame plus the corresponding positioning frame, the previous frame plus the segmentation mask, the positioning frame is cascaded after the current frame as the fourth channel, and the segmentation mask is cascaded after the previous frame as the fourth channel.

[0022] Further:

[0023] The segmentation mask includes the ground-truth mask of the first frame and the masks of the predicted results of the subsequent frames.

[0024] Further:

[0025] The current frame and the previous frame are cropped and resized respectively, and encoded with different encoders respectively. The query encoder is used for the current frame input, and the reference encoder is used for the previous frame input.

[0026] Further:

[0027] After encoding by the two encoders respectively, the feature maps of all input frames are cascaded and passed through the spatiotemporal non-local block to obtain the feature map after softmax normalization; then, the obtained feature map is upsampled and decoded to finally obtain the prediction result of the current frame t.

[0028] Further:

[0029] In step S3, during the training phase, the cross entropy cost function adopts a partially truncated cross entropy loss, that is, a partially truncated cross entropy cost function.

[0030] The video object segmentation device based on weakly supervised learning uses the above method to perform video object segmentation.

[0031] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above method.

[0032] The present invention has the following beneficial effects:

[0033] The video target segmentation method based on weakly supervised learning proposed in the present invention can greatly reduce the dependence on the precisely labeled video dataset, and can also achieve high segmentation accuracy by only using the positioning box as supervision information for training.

[0034] In some embodiments, there are also the following advantages:

[0035] Improvements have been made to the detection frame to segmentation mask model, integrating the temporal information of the video. This allows for the generation of higher-quality pseudo-labels, and improves issues such as poor continuity and missed detection of objects caused by the original independent segmentation of each video frame from the detection frame to the segmentation mask.

[0036] The generated pseudo labels were screened and refined to remove bad samples and fine-tune them with manually annotated accurate segmentation masks to improve the quality of the pseudo labels.

[0037] The pixel-by-pixel cross entropy cost function is improved by using a truncated cross entropy loss function, so the model is more robust and individual misclassified pixels will not dominate the entire loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a video target segmentation method based on weakly supervised learning in an embodiment of the present invention;

[0039] Figure 2a-2d is a schematic diagram of a positioning frame cut on an original image in an embodiment of the present invention;

[0040] Figure 3a-3h is a schematic diagram of predicting a segmentation mask using an image and a positioning box in an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of a detection box to segmentation mask model in an embodiment of the present invention;

[0042] Figure 5is a schematic diagram of a pipeline segmentation model in an embodiment of the present invention;

[0043] Figure 6 is a schematic diagram of the positioning frame of the original image and the image after being converted into a matrix in an embodiment of the present invention;

[0044] Figure 7 It is a schematic diagram of predicting a segmentation mask through a pipeline segmentation model in the case of two reference frames according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below by specific embodiments in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention.

[0046] For the task of video target segmentation, the present invention proposes a method that can be trained using only positioning box annotations, which can greatly reduce the reliance on video training sets with precise frame-by-frame and pixel-by-pixel annotations. The inventive concept of the method of the present invention is: first, a tube segmentation (Tube to segmentation, referred to as Tube2Seg) model that integrates video timing information is used to obtain pseudo labels, and a more robust loss function, pseudo label screening and other methods are used to optimize the pseudo labels, and finally the generated pseudo labels are used to train the video target segmentation model.

[0047] The method proposed in the present invention for training using only positioning frames only requires annotation at the positioning frame level in the training phase, and only provides the real positioning frame of the first frame in the prediction phase. It does not require precise annotation of each frame and pixel of the video, which can greatly reduce the annotation workload.

[0048] The embodiment of the present invention provides a video object segmentation method based on weakly supervised learning. The specific process is shown in FIG1 . The method mainly includes the following four steps:

[0049] S1. Input the original video and the corresponding positioning frame

[0050] like Figure 2a-2d As shown in the figure, the original video is extracted to obtain the two-dimensional image of each frame, and the target positioning frame of each frame is also obtained (i.e. Figure 2b-2d According to the positioning frame, 20% of the pixels are expanded outward (i.e., the dotted frame in Figures 2a-2d), and the original image is cut to obtain the final cut image (Figure 2d).

[0051] S2. Pipeline segmentation model obtains segmentation pseudo labels

[0052] like Figure 3a-3hAs shown in Figure 1, the purpose of the detection box to segmentation mask model is to use the input image and positioning box to predict the segmentation mask (i.e., pseudo label) of the target object in the corresponding positioning box. Figure 3a , Figure 3c , Figure 3e and Figure 3g are different input images and target positioning boxes ( Figure 3a , Figure 3c , Figure 3e and the solid line box in Figure 3g), Figure 3b , Figure 3d , Figure 3f and Figure 3h They are Figure 3a , Figure 3c , Figure 3e and Figure 3g The predicted segmentation mask corresponds to the target object in the positioning box.

[0053] The traditional detection box to segmentation mask model is mainly an encoder-decoder structure, such as Figure 4 As shown in the figure, the original image and the real detection box (positioning box) are input, and the encoder learns higher-level semantic information, and the decoder obtains a high-resolution pixel-by-pixel segmentation mask. However, the traditional detection box to segmentation mask model is based on two-dimensional images and does not integrate the temporal information of the video. Therefore, directly using the detection box to segmentation mask segmentation frame by frame on the video will result in unstable segmentation results due to occlusion, deformation, and appearance changes.

[0054] In order to overcome the problem that the traditional detection box to segmentation mask model cannot utilize temporal information, the present invention improves the detection box to segmentation mask model, and the improved new model can be called a tube segmentation (Tube tosegmentation, referred to as Tube2Seg) model. The tube segmentation model is still a fully convolutional neural network architecture, which makes full use of the temporal information of the input sequence through a non-local module to capture long-term dependencies in space and time. The non-local module consists of a two-part architecture of a non-local block and a split encoder, wherein the split encoder includes a query encoder and a reference encoder. This architecture can apply the image semantic information and prediction results of the previous frame to the prediction of the current frame. Figure 5As shown, the current frame + positioning frame, the true segmentation mask (i.e., the first frame segmentation mask) and the previous frame segmentation mask are input, wherein the current frame + positioning frame is encoded by the query encoder, the true segmentation mask and the previous frame segmentation mask are encoded by the reference encoder, and then the feature maps obtained by the query encoder and the reference encoder are processed by the non-local block and then decoded by the decoder to obtain the segmentation mask (i.e., pseudo label) of the current frame.

[0055] The specific pipeline segmentation model structure is as follows:

[0056] The input includes the current frame + corresponding positioning frame, the previous frame + segmentation mask (including the real mask of the first frame and the mask of the prediction result of the subsequent frame), and the positioning frame is cascaded after the current frame as the fourth channel, and the segmentation mask is cascaded after the previous frame as the fourth channel. Figure 6 As shown, input the original image + positioning frame, where the 0-1 matrix is ​​actually the positioning frame converted to the original Figure 1 The matrix is ​​as follows: 0 represents pixels outside the positioning box, and 1 represents pixels inside the positioning box.

[0057] After cropping and resizing, the two parts are encoded with different encoders, the query encoder for the current frame input and the reference encoder for the previous frame input.

[0058] Figure 7 What is shown in the figure is the situation where there are only two reference frames, namely the previous frame and the first frame. These are also the two most important frames. The first frame has the most accurate true mask, and the object shapes of the previous frame and the current frame are most similar. In actual operation, other intermediate frames can be added as needed.

[0059] In the picture represents matrix multiplication, represents pixel-by-pixel addition. After encoding by the two encoders, the feature maps of all input frames are concatenated (represented as ), and obtain the feature map after softmax normalization through the spatiotemporal non-local block. Then, the obtained feature map is upsampled and decoded to finally obtain the prediction result of the current frame t.

[0060] It can be expressed as:

[0061]

[0062]

[0063] Where softmax(x) represents the normalized exponential function, x is the feature vector of the input frame, T represents the transposition operation, and 1*1 convolution operation is used to perform linear mapping of g(x), θ(x) and φ(x) respectively. W represents the weight of the mapping, and W θ is the weight matrix in the θ(x) mapping, so the mapping can be expressed as (xW θ ) and (xW φ ), the results of the two mappings are then matrix multiplied together (the first item needs to be transposed to match), and we get (xW θ ) T (xW φ )=x T W θ T W φ x. This can strengthen the connection between pixels at different positions in the same frame, as well as the connection between frames, and improve long-distance dependencies.

[0064] S3. Screening and Refining of Pseudo Labels There are still some bad samples when using the improved pipeline segmentation model to generate segmentation pseudo labels. In order to improve the quality of pseudo labels, the present invention adopts three measures:

[0065] The first is bad sample filtering. We filter based on the confidence of the generated mask and delete the segmentation masks whose overall confidence is less than a certain threshold (such as 70%) as bad samples. During the training phase, the labels of these frames are set to empty and no gradient is returned. Another method is manual intervention to filter bad samples, which is more accurate but requires additional manpower.

[0066] The second is refinement. The pipeline segmentation model is refined by annotating the real mask of one frame in each video, which can greatly improve the quality of pseudo-labels.

[0067] The third is to use a more robust loss function during the training phase. The traditional segmentation network uses a pixel-by-pixel cross entropy cost function l θ (x, y) = -log p θ (x, y), when the confidence p(x, y) that a certain pixel x is classified as y is very small, the value of the negative logarithm will approach infinity, so individual misclassified pixels will cause excessive loss and easily cause overfitting. In order to enhance the robustness of the loss function, we improve the cross entropy cost function to a partially truncated cross entropy loss (Partially Huberised cross-entropy loss) Partially truncated cross entropy cost function. The function is defined as follows: when the confidence p(x, y) is less than a certain threshold (τ=2,3,4,5, the hyperparameter can be adjusted according to the actual situation), the loss function is truncated to linear, thereby limiting the misclassification and enhancing the overall robustness.

[0068]

[0069] Preferably, other human intervention methods can be used to interactively improve the quality of pseudo-labels.

[0070] S4. Training video object segmentation model with pseudo labels

[0071] We use the original video and the obtained segmentation pseudo-labels to train the video object segmentation network, including but not limited to the currently effective space-time memory model (Space-Time Memory Networks, referred to as STM), the collaborative video object segmentation by foreground-background integration model (Collaborative video object segmentation by foreground-background integration, referred to as CFBI), and the small sample learner model (Learning what to learn for video object segmentation, referred to as LWL). This part can be directly trained using the current best video object segmentation model. Preferably, the generated pseudo-labels can also be used to train the video instance segmentation model, which can be applied to video instance segmentation.

[0072] Effect Analysis of Video Target Segmentation Methods

[0073] We use the video object segmentation model trained with pseudo labels for prediction. The following table shows the evaluation results of STM-VOS (Video Object Segmentation Spatiotemporal Memory Model) trained under different settings on the DAVIS 2017 and YouTube-VOS2018 datasets. The first three rows are the experimental results cited from the STM article [Oh SW, Lee JY, Xu N, et al. Video Object Segmentation using Space-Time Memory Networks [J]. 2019.]. The next seven rows are the results of our experimental reproduction. From the fully supervised mode to the weakly supervised mode, we verified a variety of different strategies, such as filtering noisy samples, using a real mask for fine-tuning, and using robust loss functions. Note that in the value of the second column "Number of Video Annotation Masks", the symbol {*} indicates that the quality of the pseudo-label needs to be manually evaluated.

[0074]

[0075] When using detection box to segmentation mask + refinement + robustness function, the final gap between the training and the fully supervised mode is only 1.4% (YouTube-VOS dataset).

[0076] The comparison with other weakly supervised video target segmentation methods is as follows: We list the supervision information used by each method, and it can be seen that our method is particularly effective on the YouTube-VOS dataset, and can achieve a good compromise between the amount of labeled data and the final effect. The following table shows the comparison of spatiotemporal memory video target segmentation methods (including two settings of STM-VOS: Box2seg model + robust loss function, Box2seg model + label refinement), two weakly supervised video target segmentation methods (including PReMVOS, Siam R-CNN) and four unsupervised video target segmentation methods (including MAST, MuG, UVC, Corrflow). The second column "Number of labeled image masks" shows whether these methods use segmentation masks of static images. The third column "Number of labeled video detection boxes" shows whether these methods use detection boxes of video sequences. The fourth column "Number of labeled video masks" shows the number of video segmentation masks used by these methods. It can be seen that we have largely surpassed all previous methods.

[0077]

[0078] Practical application scenarios and applications

[0079] Video object segmentation is a very important direction in the field of computer vision. It has related needs in video editing, scene understanding, video surveillance, and event retrieval. For example, to obtain the position and area occupied by the human body in the input video, or to further emphasize, outline, or eliminate the human body, video object segmentation technology is required.

[0080] In industrial applications, the background masking / replacement application of video conferencing is a very typical video target segmentation technology. First, the human body in the video is segmented out, and the human body is retained as the foreground, and the other parts are used as the background area. The background is further replaced with different background patterns such as study, office, and seaside according to user preferences. For example, the anti-blocking function of the barrage in various video playback software is also implemented based on the video target segmentation technology. For videos with a large number of barrages sent by users in Tencent Video and Bilibili, the barrage completely blocks the main characters. Other users want to be able to see the characters in the video clearly, but do not want to completely close the interesting barrage. At this time, the main character is segmented out as the foreground through the video target segmentation technology. The barrage will not slide in front of the main character after encountering the mask of the main character, so that the anti-blocking function of the main character can be realized.

[0081] In actual applications, it is very time-consuming and expensive to annotate video targets frame by frame, and there is a lot of redundant work due to the high similarity between video frames. If it is possible to learn the video segmentation model using only cheap and easily available detection box annotations, the application cost will be greatly reduced. In an embodiment of the present invention, the detection box only needs to determine the four key corner points of the target, while the segmentation annotation requires pixel-by-pixel annotation. The method in the embodiment of the present invention can obtain a model effect close to that obtained by learning the complete pixel-by-pixel annotation method using only detection box information or fewer segmentation annotations, which can greatly improve efficiency and reduce costs.

[0082] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A video target segmentation method based on weakly supervised learning, characterized in that: The steps include: S1. Input original video and positioning frame: Input original video, extract frames to obtain two-dimensional images of each frame, and also obtain target positioning frames of each frame, and perform positioning at the positioning frame level; cut the original image according to the positioning frame to obtain the final cut image; S2, inputting the cut image and the target positioning frame into the pipeline segmentation model to obtain a segmentation mask of the current frame, i.e., a pseudo label; S3, training a video object segmentation model using the pseudo labels; Among them, the pseudo labels are screened and refined, including: Bad sample filtering: Screening is performed based on the confidence of the generated mask. Segmentation masks with an overall confidence less than a specific threshold are considered bad samples and deleted. During the training phase, the labels of these frames are set to empty and no gradients are returned. Alternatively, bad samples can be screened through manual intervention. Refinement: The pipeline segmentation model is refined with the ground-truth mask of one frame of each video to improve the quality of pseudo-labels. During the training phase, the cross entropy cost function is improved to a partially truncated cross entropy cost function, which is defined as follows: ； When confidence Less than or equal to threshold , the loss function is truncated to linearity to limit misclassification, where is a hyperparameter.

2. The video target segmentation method based on weakly supervised learning according to claim 1 is characterized in that: In step S1, the method for obtaining the final cut image is: expanding 20% ​​of the pixels outward according to the positioning frame, cutting on the original image, and obtaining the final cut image.

3. The video target segmentation method based on weakly supervised learning according to claim 1 is characterized in that: The pipeline segmentation model is a fully convolutional neural network architecture that fully utilizes the temporal information of the input sequence through a non-local module to capture long-term dependencies in space and time; the non-local module consists of a two-part architecture of a non-local block and a split encoder, wherein the split encoder includes a query encoder and a reference encoder, and the architecture can apply the image semantic information and prediction results of the previous frame to the prediction of the current frame.

4. The video target segmentation method based on weakly supervised learning according to claim 3 is characterized in that: In step S2, obtaining the segmentation mask of the current frame includes the following steps: Input the current frame + positioning frame, the true segmentation mask and the previous frame segmentation mask, where the current frame + positioning frame is encoded by the query encoder, the true segmentation mask and the previous frame segmentation mask are encoded by the reference encoder, and then the feature maps obtained by the query encoder and the reference encoder are processed by the non-local block and then decoded by the decoder to obtain the segmentation mask of the current frame.

5. The video object segmentation method based on weakly supervised learning according to claim 4 is characterized in that: The specific pipeline segmentation model structure is as follows: The input includes the current frame plus the corresponding positioning frame, the previous frame plus the segmentation mask, the positioning frame is cascaded after the current frame as the fourth channel, and the segmentation mask is cascaded after the previous frame as the fourth channel.

6. The video object segmentation method based on weakly supervised learning according to claim 5 is characterized in that: The segmentation mask includes the ground-truth mask of the first frame and the masks of the predicted results of the subsequent frames.

7. The video object segmentation method based on weakly supervised learning according to claim 3 is characterized in that: The current frame and the previous frame are cropped and resized respectively, and encoded with different encoders respectively. The query encoder is used for the current frame input, and the reference encoder is used for the previous frame input.

8. The video object segmentation method based on weakly supervised learning according to claim 7 is characterized in that: After encoding by the two encoders respectively, the feature maps of all input frames are cascaded and passed through the spatiotemporal non-local block to obtain the feature map after softmax normalization; then, the obtained feature map is upsampled and decoded to finally obtain the prediction result of the current frame t.

9. The video object segmentation method based on weakly supervised learning according to claim 1, characterized in that: In step S3, during the training phase, the cross entropy cost function adopts a partially truncated cross entropy loss, that is, a partially truncated cross entropy cost function.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 9.

Images