Online video instance segmentation method and system based on mask propagation

Abstract

The invention provides an online video instance segmentation method and system based on mask propagation, and the method comprises the following steps: S1, achieving the cross-frame propagation of a target mask through a mask propagation model, obtaining a prediction mask of a current frame, and achieving the correlation of a target through the calculation of the intersection-union ratio of the prediction mask; and S2, performing back propagation on the target by using the mask propagation model so as to complement the missed target mask and enhance the continuity of the track. Objects between different frames are associated on the basis of a mask propagation mechanism and in combination with the intersection-to-union ratio of masks, and meanwhile, in order to relieve the scene that a segmentation model fails to segment shielded objects, non-significant objects and the like, a reverse mask propagation technology is provided to complement the failed frames. The method aims at performing fine segmentation and continuous tracking on various scenes such as intelligent monitoring, automatic driving, augmented reality and video editing on the target in the video, and has wide research and application prospects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, and particularly relates to an online video instance segmentation method and system based on mask propagation. BACKGROUND

[0002] Video instance segmentation [1] is an important task in the field of computer vision, aiming to simultaneously realize the classification, segmentation and tracking of targets in a video. With the development of deep learning technology, the method of video instance segmentation has made rapid progress, among which the mainstream methods are divided into two types, namely online video instance segmentation method [2][3] and offline video instance segmentation method [4][5] , wherein the online method usually performs segmentation on each frame of the video, and then associates the objects between different frames, and the offline method simultaneously performs segmentation and tracking on the targets in the entire video segment.

[0003] The online method usually decouples segmentation and tracking and optimizes the two parts respectively. For segmentation, previous methods usually adopt picture instance segmentation methods such as Mask2Former [6] or Mask R-CNN [7] ; for the tracking module, the previous mainstream method usually matches based on the similarity of target features, and the above method has the following defects:

[0004] 1. Low single-frame segmentation accuracy, frame-by-frame segmentation of targets lacks temporal context information, and is prone to miss or misjudge some scenes such as occlusion, low light, blur, non-significant targets, etc., thereby causing tracking failure and subsequent error accumulation.

[0005] 2. Non-robustness based on feature matching, the current mainstream method (such as CTVIS [2] ) usually uses a matching strategy based on the similarity of target features to associate targets between different frames, but is limited by the representation ability and generalization of the segmentation model, and the model is prone to ID drift, missed detection or false matching in scenes with fast motion, severe occlusion or similar object interference.

[0006] At present, with the progress of instance segmentation and multi-target tracking technology, video instance segmentation has become a research hotspot. However, existing methods mostly rely on target feature similarity for inter-frame target association, and in complex video scenes such as occlusion, deformation, light change or target disappearance and reappearance, the matching process is prone to errors, leading to target ID drift or tracking loss. At the same time, the segmentation model often has missed detection or misjudgment in some frames, further affecting the completeness of the target trajectory and the segmentation accuracy.

[0007] REFERENCES

[0008] [1] Yang L, Fan Y, Xu N. Video instance segmentation [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019:5188-5197.

[0009] [2] Ying K, Zhong Q, Mao W, et al. Ctvis: Consistent training for online video instance segmentation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023:899-908.

[0010] [3] Huang D A, Yu Z, Anandkumar A. Minvis: A minimal video instance segmentation framework without video-based training [J]. Advances in Neural Information Processing Systems, 2022, 35:31265-31277.

[0011] [4] Heo M, Hwang S, Oh S W, et al. Vita: Video instance segmentation via object token association [J]. Advances in neural information processing systems, 2022, 35:23109-23120.

[0012] [5] Wang Y, Xu Z, Wang X, et al. End-to-end video instance segmentation with transformers [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021:8741-8750.

[0013] [6]Cheng B, Misra I, Schwing A G, et al. Masked-attention mask transformer for universal image segmentation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022:1290-1299.

[0014] [7]He K, Gkioxari G, Dollar P, et al. Mask r-cnn[C] / / Proceedings of the IEEE international conference on computer vision. 2017:2961-2969.

[0015] [8]Huang D A, Yu Z, Anandkumar A. Minvis: A minimal video instance segmentation framework without video-based training[J]. Advances in Neural Information Processing Systems, 2022, 35:31265-31277.

[0016] [9]Wu J, Liu Q, Jiang Y, et al. In defense of online models for video instance segmentation[C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022:588-605.

[0017]

[10] Zhang T, Tian X, Wu Y, et al. Dvis: Decoupled video instance segmentation framework[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023:1282-1291.

[0018]

[11] Zhou Y, Zhang T, Ji S, et al. Improving video segmentation via dynamic anchor queries [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 446-463. Summary of the Invention

[0019] This invention addresses the aforementioned issues and aims to provide an online video instance segmentation method and system based on mask propagation. This method replaces traditional feature matching methods with a mask propagation strategy and introduces a reverse mask propagation mechanism to supplement missed frames, thereby achieving more stable, accurate, and robust video instance segmentation. This method has significant practical application implications.

[0020] The present invention provides an online video instance segmentation method based on mask propagation, which has the following characteristics and includes the following steps: S1, using a mask propagation model to realize cross-frame propagation of a target mask to obtain a predicted mask in the current frame, and then realizing the association of the target by calculating the intersection-over-union ratio of the predicted masks; S2, using the mask propagation model to backpropagate the target, thereby completing the missed target mask and enhancing the continuity of the trajectory.

[0021] The online video instance segmentation method based on mask propagation provided by the present invention may also have the following features: wherein, in step S1, for each frame, the image instance segmentation model is first used to extract the candidate target mask set of the current frame, and then the known target mask in the previous frame is propagated to the current frame through the mask propagation model to obtain a cross-frame prediction mask, and the target matching matrix is ​​constructed by calculating the intersection-over-union ratio between the predicted mask and the current frame detection mask, and the target matching is completed through a greedy strategy, thereby achieving robust target association and trajectory update.

[0022] The online video instance segmentation method based on mask propagation provided by the present invention may also have the following features: wherein step S1 specifically includes the following sub-steps:

[0023] S1-1, given a video sequence Among them I t represents the t-th frame image, T represents the total number of video frames;

[0024] S1-2, first through the image segmentation model Perform target instance segmentation on each frame of image to obtain a candidate segmentation set for each frame in represents the jth candidate mask, Nt the number of detected candidate target instances for the current frame;

[0025] S1-3, maintain a memory bank for storing the identity and mask information of tracked targets in history frames;

[0026] S1-4, for the current frame t, from all known target masks in the last frame t-1 use mask propagation model to propagate the mask and get the predicted mask in the current frame:

[0027]

[0028] where K t-1 represents the number of instances stored at time t-1 , and the above process represents propagating the i-th target mask in the t-1 frame to the t frame to estimate its corresponding mask in the current frame

[0029] S1-5, calculate the mask intersection over union (IoU) between all predicted masks and the current frame segmentation result , and construct a matching matrix where the (i,j) element is:

[0030]

[0031] S1-6, set the matching threshold θ (e.g. 0.5) and use the greedy strategy to match the matrix M:

[0032] if M i,j ≥ θ, then and are the same instance, then is regarded as the mask of instance i at time t,

[0033] if for all j, M i,j < θ, then is regarded as the mask of instance i at time t;

[0034] S1-7, regard the unmatched as a new target, assign a new ID (K t-1 +1) and add it to the memory bank

[0035] ​In the online video instance segmentation method based on mask propagation provided by the application, the reverse mask propagation strategy can be further used to solve the problem that part of the target is not detected in some frames, the mask of the target is propagated to earlier frames from the frame where the target first appears, the existing area of the target in the historical frames is predicted and the mask is completed, the complete track is further recovered, and the missing detection problem caused by occlusion, weak saliency or detection failure is alleviated.

[0036] In the online video instance segmentation method based on mask propagation provided by the application, the reverse mask propagation strategy can be further used to solve the problem that part of the target is not detected in some frames, the mask of the target is propagated to earlier frames from the frame where the target first appears, the existing area of the target in the historical frames is predicted and the mask is completed, the complete track is further recovered, and the missing detection problem caused by occlusion, weak saliency or detection failure is alleviated.

[0037] S2-1, for each instance i, record the frame number where the target first appears and the corresponding mask Then the following reverse propagation process is performed:

[0038]

[0039] S2-2, wherein represents the prediction result of propagating the mask of the instance i from frame t+1 to frame t;

[0040] S2-3, for each frame t, add to the memory library, and complete the mask of the target in frame t;

[0041] S2-4, after the entire video sequence is processed, the Tracklet NMS strategy is applied to all target tracks to remove redundant tracks that are highly overlapped in space and time, and only the track with the highest confidence is retained, so as to improve the segmentation quality and track stability.

[0042] The application further provides an online video instance segmentation system based on mask propagation, which has the following characteristics: a forward mask propagation and matching module is used to realize cross-frame propagation of the target mask by using a mask propagation model, to obtain a predicted mask in the current frame, and to realize association of the target by calculating the intersection over union of the predicted mask; a reverse mask propagation and completion module is used to propagate the target in reverse by using the mask propagation model, so as to complete the missed target mask and enhance the continuity of the track.

[0043] Effects of the application

[0044] According to the online video instance segmentation method and system based on mask propagation, it is suitable for various computer vision practical application scenarios, including but not limited to: intelligent monitoring and behavior recognition, dynamic object understanding in automatic driving, video editing and object removal, robot vision and human-computer interaction system, target tracking and alarm in security system, especially suitable for scenes with high precision requirements, complex inter-frame association or serious target occlusion, while maintaining high-precision segmentation, ensuring tracking stability and real-time performance, which has great practical application significance. Specifically, the method has the following beneficial effects:

[0045] (1) The target matching is more robust: compared with the traditional matching strategy based on feature vectors, the present application uses the mask information after mask propagation for IoU matching, which significantly improves the accuracy and stability of cross-frame target matching, especially in complex scenes with rapid motion, severe deformation or occlusion.

[0046] (2) Strong trajectory completion ability: the present application first introduces a reverse mask propagation mechanism in video instance segmentation. The mechanism can complete the target trajectory through the mask reverse propagation technology in the early frames where the segmentation model fails to detect the target, effectively improving the temporal integrity and segmentation continuity of the trajectory, and alleviating the problem of missed detection.

[0047] (3) Modular plug and play: the present application can modularly call existing picture instance segmentation models and mask propagation models, which has good universality, scalability and deployment convenience, and can quickly adapt to different backbone networks and video scenes.

[0048] (4) The performance index is significantly superior: the evaluation on the YouTube-VIS2021 verification set shows that the present application (SAM2VIS) achieves excellent performance when the backbone network is ResNet-50, Swin-L and ViT-L, with the highest MaskAP reaching 62.2%, which is 2 to 5 percentage points higher than the current SOTA method (such as CTVIS, DVIS-DAQ) in multiple evaluation indexes. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of the online video instance segmentation method based on mask propagation in the embodiments of the present application;

[0050] Figure 2 is a use schematic diagram of the online video instance segmentation method based on mask propagation in the embodiments of the present application;

[0051] Figure 3 is a visual comparison diagram of the online video instance segmentation method based on mask propagation in the embodiments of the present application and the existing method CTVIS when processing occlusion and position change.

[0052] Figure 4 is the visualization comparison chart of the online video instance segmentation method based on mask propagation in the embodiment of the application and the existing method CTVIS when a new object appears. DETAILED DESCRIPTION

[0053] In the description of the present application, it should be explained that, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection or can communicate with each other; it can be directly connected, or indirectly connected through intermediate medium, it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0054] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the following embodiments combine with the drawings to specifically describe the online video instance segmentation method and system based on mask propagation of the present application.

[0055] Figure 1 is the flow chart of the online video instance segmentation method based on mask propagation in the embodiment of the present application. Figure 2 is the use schematic diagram of the online video instance segmentation method based on mask propagation in the embodiment of the present application.

[0056] As Figures 1-2 shown, the online video instance segmentation method based on mask propagation in the embodiment includes the following steps:

[0057] S1, the cross-frame propagation of target mask is realized by using mask propagation model, the predicted mask in the current frame is obtained, and the association of the target is realized by calculating the intersection over union of the predicted mask.

[0058] Specifically, for each frame, first, the candidate target mask set of the current frame is extracted by using the image instance segmentation model, then the known target mask in the previous frame is propagated to the current frame by the mask propagation model, and the cross-frame predicted mask is obtained. By calculating the intersection over union between the predicted mask and the current frame detection mask, a target matching matrix is constructed, and the target matching is completed by the greedy strategy, so as to realize the stable target association and trajectory update.

[0059] Step S1 specifically includes the following sub-steps:

[0060] S1-1, a video sequence is given Where I t represents the t-th frame image, and T represents the total number of video frames.

[0061] S1-2, first pass the image segmentation model Target instance segmentation is performed on each frame of image to obtain the candidate segmentation set of each frame Wherein represents the jth candidate mask, N t is the number of candidate target instances detected in the current frame.

[0062] S1-3, in order to realize the association of cross-frame target, maintain a memory library For storing the identity and mask information of the tracked target in the historical frame.

[0063] S1-4, for the current frame t, from all known target masks in the last frame t-1 Using the mask propagation model Mask propagation is carried out to obtain the predicted mask in the current frame:

[0064]

[0065] Wherein K t-1 represents the instance number stored at time t-1 The above process means that the ith target mask in the t-1th frame is propagated to the tth frame, and its corresponding mask in the current frame is estimated

[0066] S1-5, calculate the mask intersection and union ratio (IoU) between all predicted masks And the segmentation result of the current frame A matching matrix is constructed Wherein the (i,j) element is:

[0067]

[0068] S1-6, set the matching threshold θ (for example, 0.5), and use the greedy strategy to match the matrix M:

[0069] If M i,j ≥θ, it is considered that And It is the same instance, then It is regarded as the mask of instance i at time t.

[0070] If for all j, M i,j <θ, then As the mask of instance i at time t;

[0071] S1-7, the unmatched It is regarded as a new target, and a new ID (K t-1 +1) is assigned and added to the memory library​

[0072] The goal of this stage is to perform target segmentation and matching frame by frame according to the above process, and to update .

[0073] S2, the target is back-propagated using the mask propagation model, so as to complete the missed target mask and enhance the continuity of the track.

[0074] After all the frames are processed in the forward direction, it is found that many target instances are not detected from the first frame, which may be missed due to occlusion, low saliency or segmentation failure. Therefore, a backward mask propagation strategy is introduced to complete the early missing section of the target track.

[0075] Specifically, for the problem that some targets are not detected in some frames, a backward mask propagation strategy is proposed, which propagates the mask of the target from the frame where it first appears to earlier frames, predicts its existing area in the historical frames and completes the mask, further restores the complete track, and thus alleviates the missing detection problem caused by occlusion, weak saliency or detection failure.

[0076] Step S2 specifically includes the following sub-steps:

[0077] S2-1, for each instance i, record the frame number where the instance first appears and the corresponding mask Then perform the following back propagation process:

[0078]

[0079] S2-2, wherein represents the prediction result of back-propagating the mask of instance i from frame t+1 to frame t.

[0080] S2-3, for each frame t, add to the memory library and complete the mask of the target in frame t.

[0081] S2-4, after the entire video sequence is processed, apply the Tracklet NMS (Non-Maximum Suppression) strategy to all target tracks to remove redundant tracks that are highly overlapped in space and time, and only keep the one with the highest confidence, thereby improving the segmentation quality and track stability.

[0082] Figure 3 is the visualization comparison chart of the online video instance segmentation method based on mask propagation in the embodiment of the application and the existing method CTVIS when processing occlusion and position change

[0083] As Figure 3As shown, the SAM2VIS method of the present invention is effective in processing occlusion and position changes and can successfully track the target.

[0084] Figure 4 This is a visual comparison diagram of the online video instance segmentation method based on mask propagation in an embodiment of the present invention and the existing method CTVIS when new objects appear.

[0085] like Figure 4 As shown, when a new object appears, the SAM2VIS method of the present invention can successfully assign the correct ID, while the baseline method suffers from ID drift and duplication.

[0086] As shown in Table 1, the evaluation on the YouTube-VIS2021 validation set shows that the present invention (SAM2VIS) achieves excellent performance in the configuration of ResNet-50, Swin-L and ViT-L backbone networks, with the highest MaskAP reaching 62.2%, exceeding the current SOTA methods (such as CTVIS, DVIS-DAQ) by 2 to 5 percentage points in multiple evaluation indicators.

[0087] Experimental results show that the method of the present invention significantly outperforms current SOTA methods on mainstream video instance segmentation datasets such as YouTube-VIS2021, achieving a maximum accuracy of 62.2% in the MaskAP metric, and exhibits greater stability and integrity in complex scenarios such as occlusion, rapid motion, and target reappearance. This method can be equipped with any segmentation model and tracking model, has plug-and-play capabilities, and is applicable to a variety of practical scenarios such as autonomous driving, security monitoring, video editing and understanding, and has broad practical application value and engineering promotion prospects.

[0088] Table 1

[0089]

[0090] The present invention also discloses an online video instance segmentation system based on mask propagation, which includes a forward mask propagation and matching module and a reverse mask propagation and completion module.

[0091] The forward mask propagation and matching module follows step S1 above, uses the mask propagation model to propagate the target mask across frames, obtains the predicted mask in the current frame, and then calculates the intersection-over-union ratio of the predicted masks to achieve target association.

[0092] The reverse mask propagation and completion module follows step S2 above and uses the mask propagation model to reversely propagate the target, thereby completing the missed target mask and enhancing the continuity of the trajectory.

[0093] Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An online video instance segmentation method based on mask propagation, characterized in that: The steps include: S1, using the mask propagation model to realize cross-frame propagation of the target mask, obtain the predicted mask in the current frame, and then realize the association of the target by calculating the intersection-over-union ratio of the predicted masks; S2, using the mask propagation model to back-propagate the target, thereby completing the missed target mask and enhancing the continuity of the trajectory.

2. The online video instance segmentation method based on mask propagation according to claim 1, characterized in that: in, In step S1, for each frame, the image instance segmentation model is first used to extract the candidate target mask set of the current frame. Then, the known target mask in the previous frame is propagated to the current frame through the mask propagation model to obtain the cross-frame prediction mask. By calculating the intersection-over-union ratio between the predicted mask and the current frame detection mask, the target matching matrix is ​​constructed. The target matching is completed through a greedy strategy, thereby achieving robust target association and trajectory update.

3. The online video instance segmentation method based on mask propagation according to claim 2, characterized in that: in, Step S1 specifically includes the following sub-steps: S1-1, given a video sequence Among them I t represents the t-th frame image, T represents the total number of video frames; S1-2, first through the image segmentation model Perform target instance segmentation on each frame of image to obtain a candidate segmentation set for each frame in represents the jth candidate mask, N t The number of candidate object instances detected for the current frame; S1-3, maintain a memory bank Used to store the identity and mask information of tracked targets in historical frames; S1-4, for the current frame t, all known target masks from the previous frame t-1 In the example above, we use the mask propagation model Perform mask propagation to obtain the predicted mask in the current frame: where K t-1 Indicates time t-1 The number of instances stored in the above process means that the i-th target mask in the t-1 frame Propagate to the tth frame and estimate its corresponding mask in the current frame S1-5, calculate all prediction masks Segmentation result with the current frame The mask intersection over union (IoU) between them is used to construct a matching matrix The (i,j)th element is: S1-6, set the matching threshold θ (for example, 0.5) and use the greedy strategy to match the matrix M: If M i,j ≥θ, then it is considered and is the same instance, then is regarded as the mask of instance i at time t, If for all j, M i,j <θ, then As the mask of instance i at time instant; S1-7, the unmatched Treated as a new target, assigned a new ID (K t-1 +1) and added to the memory bank 4. The online video instance segmentation method based on mask propagation according to claim 1, characterized in that: in, In step S2, to address the problem of some targets not being detected in certain frames, a reverse mask propagation strategy is proposed. Starting from the frame where the target first appears, its mask is propagated forward to earlier frames, its presence area in historical frames is predicted and the mask is completed, further restoring the complete trajectory, thereby alleviating the problem of missed detection due to occlusion, weak saliency or detection failure.

5. The online video instance segmentation method based on mask propagation according to claim 4, characterized in that: in, Step S2 specifically includes the following sub-steps: S2-1, for each instance i, record the frame number where it first appears and the corresponding mask Then perform the following back propagation process: S2-2, where represents the prediction result of backpropagating the mask of instance i from frame t+1 to frame t; S2-3, for each frame t, Add to the memory bank and use it as the mask completion for the target in frame t; S2-4, after the entire video sequence is processed, the Tracklet NMS strategy is applied to all target tracks to remove redundant tracks that are highly overlapped in space and time, and only retain the one with the highest confidence, thereby improving the segmentation quality and track stability.

6. An online video instance segmentation system based on mask propagation, characterized in that: include: The forward mask propagation and matching module uses the mask propagation model to propagate the target mask across frames, obtains the predicted mask in the current frame, and then calculates the intersection-over-union ratio of the predicted masks to achieve target association; The reverse mask propagation and completion module uses the mask propagation model to reversely propagate the target, thereby completing the missed target mask and enhancing the continuity of the trajectory.

Images