Video object segmentation method based on key point and target frame joint tracking
By using a joint tracking method of key points and target bounding boxes, the selection of candidate bounding boxes and the filtering of unreliable information are optimized, solving the problem of video target segmentation in zero-shot scenarios. This achieves accurate segmentation of unseen targets and improves the robustness and accuracy of video target segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2024-03-11
- Publication Date
- 2026-05-19
AI Technical Summary
Existing video object segmentation methods struggle to achieve accurate object segmentation in zero-shot scenarios, especially since sparse keypoints lack instance and semantic information about the overall object, leading to local segmentation and incorrect segmentation.
A joint tracking method based on keypoints and bounding boxes is adopted. By constructing keypoint tracking branches and bounding box tracking branches, combining memory mechanisms and Hungarian algorithms to optimize candidate box selection, and designing a joint point-bounding filter, the accuracy of video target segmentation is improved.
It achieves accurate segmentation of unseen targets and videos with different data distributions in zero-sample scenarios, reduces dependence on large-scale labeled data, and improves the robustness and accuracy of video target segmentation.
Smart Images
Figure CN118172536B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video target segmentation, and more specifically to a video target segmentation method based on joint tracking of key points and target boxes. Background Technology
[0002] In recent years, video object segmentation technology has developed rapidly and has been widely used in video editing, video surveillance, and autonomous driving. Compared with image segmentation technology, video object segmentation is more challenging. In addition to achieving accurate image object segmentation, it is also necessary to consider the temporal semantic consistency of the objects to be segmented.
[0003] Classical video object segmentation methods employ complex memory mechanisms to match features from the current frame with those from historical frames, achieving relatively good segmentation results. However, they heavily rely on pre-training on large-scale labeled video object segmentation datasets and lack exploration of unknown targets, especially in zero-shot scenarios. Video object segmentation methods based on the Segmentation All Model (SAM) extend SAM to video object segmentation tasks, enabling the model to accept flexible points and bounding boxes as inputs. Furthermore, combining SAM segmentation results with those of traditional methods achieves even better results. Leveraging SAM's powerful zero-shot image segmentation capabilities, this type of method achieves zero-shot segmentation of all targets in videos; however, it still requires pre-training on large-scale video segmentation data. Another type of SAM-based video object segmentation model extracts keypoints of the target and uses point tracking methods to trace their trajectories, using the trajectory coordinates as input to SAM to complete video object segmentation. While these methods eliminate the reliance on expensive, labeled video target segmentation data, sparse keypoints only represent local information about various parts of the target, lacking an instance, semantics, and perception of the target as a whole, which leads to local segmentation and missegmentation of the target. Summary of the Invention
[0004] To address the aforementioned technical shortcomings, the present invention aims to provide a video target segmentation method based on joint tracking of key points and bounding boxes. This method leverages the semantic consistency of the target's appearance within a video sequence and enhances the overall perception of the target by adding bounding box tracking, thereby improving the video target segmentation effect and enabling better segmentation of targets of interest.
[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0006] The video target segmentation method based on joint tracking of key points and target boxes of the present invention is characterized by the following steps:
[0007] Step 1: Obtain the video data to be segmented and the data annotations of the reference frames, and after preprocessing, obtain the text description of the target.
[0008] Step 1.1: Obtain the video data to be segmented, denoted as... Among them, I i Let i represent the i-th frame of the image, and T represent the number of frames in the video data; The first frame of the image is used as the reference frame, and the data of the reference frame is labeled and denoted as M;
[0009] Step 1.2: Process the reference frame using an image description model to generate a text description of the target, denoted as...
[0010] Step 2: Construct a keypoint tracking branch to extract and track the key points of the target, obtaining the motion trajectory coordinates of the target key points, denoted as... Where N represents the number of target key points;
[0011] Step 3: Construct the target bounding box tracking branch for... Process the text to generate a description. The candidate bounding boxes are generated; and the candidate bounding box closest to the target position in the reference frame is stored in the memory bank. By comparing the candidate bounding boxes in the current frame image with the target bounding boxes stored in the memory bank, the best target bounding box in the current frame image is selected as the tracking result of the current frame image, so as to achieve target bounding box tracking.
[0012] Step 4: Construct a joint filter for the bounding box to analyze the coordinates of the motion trajectory. The tracking results are filtered out to obtain the filtered motion trajectory coordinates and tracking results, which are then input into the SAM model to obtain the segmentation results of each frame of the image, thereby completing the target segmentation of the video data.
[0013] The video target segmentation method based on joint tracking of key points and target boxes described in this invention is also characterized in that step 1.2 includes:
[0014] The pixel position of the target in the first frame image I1 is obtained using the data annotation M of the reference frame, and the first frame image I1 and the annotation M are multiplied to obtain the cropped image of the target.
[0015] Crop the image The first frame image I1 is input into the image description model for processing, resulting in a background-free text description T. object and text descriptions with background information T bg ;
[0016] T object and Tbg After fusion, the target text description
[0017] Step 2 includes:
[0018] Step 2.1: Extract N key points from reference frame I1 to represent the target and background, denoted as...
[0019] The pixel coordinate sequence P of the target in the reference frame I1 is obtained using data annotation M. seq ;
[0020] P is selected using the K-centroid clustering algorithm. seq K in p The center points are used as positive sample points of the target, representing the position of the target; in the remaining pixel coordinate sequence I1-P seq In the middle, K is randomly selected. n Each sample point serves as a negative sample point for the target, representing background information; K p Positive sample points of each target and K n Constituting negative sample points
[0021] Step 2.2, Tracking Key Points:
[0022] The N keypoints P0 of the reference frame are input into the point tracking model to track the keypoints of the target in the video data. The movement within the [object], and output the key points in [the object]. The coordinates of the motion trajectory in
[0023] Step 3 includes:
[0024] Step 3.1, Generation of candidate boxes:
[0025] Text description and the t-th frame image I t The data is fed into a text-based object detection model to obtain candidate bounding boxes for the object in the t-th frame. Among them, B t F represents the coordinates of the candidate box. t Features representing candidate bounding boxes;
[0026] Step 3.2, Target Box Selection and Tracking:
[0027] Take the smallest bounding rectangle of the data label M as the ground truth bounding box B of the target. GT Calculate the coordinates B1 of the candidate bounding box and B1 of the ground truth bounding box in the first frame image I1. GT The intersection-union ratio (IUU) is calculated, and the candidate box with the highest IUU is selected, denoted as Q. boxThis represents the bounding box tracking result of the target in the first frame image I1.
[0028] Build a dynamically updated memory in, This represents the selection result of the candidate bounding boxes for the ti-th frame image, and This represents the coordinates of the candidate bounding box in the ti-th frame of the image. The features representing the candidate bounding boxes of the ti-th frame image are represented by... Initialize the memory bank The selection results for each candidate box in the table;
[0029] Computational memory Memory Center Where, mean represents the average value. and They represent memory banks respectively. The coordinate center and feature center;
[0030] Calculate the image I in frame t t candidate boxes Features With memory bank Features cosine similarity in, and Represent the number of targets to be segmented and the image I in frame t, respectively. t candidate boxes Quantity;
[0031] Will The coordinates and features of candidate boxes that are smaller than the set threshold s are set to zero;
[0032] The processed similarity matrix Convert to cost matrix
[0033] The Hungarian algorithm is used to solve for D, and the bipartite graph matching result with the minimum cost is obtained, which is the image I of frame t. t The selection result;
[0034] reserve Image I of frame t t The selection results are updated in the memory according to the first-in-first-out queue method. The remaining t-2 selection results are used to track the target bounding box, and the tracking result is recorded as follows.
[0035] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the video target segmentation method, and the processor is configured to execute the program stored in the memory.
[0036] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the video target segmentation method.
[0037] Compared with existing technologies, the beneficial effects of the present invention are as follows:
[0038] 1. This invention designs a video target segmentation framework based on joint tracking of target key points and bounding boxes, providing a new approach to solving video target segmentation tasks. Compared with mainstream methods, this invention not only eliminates the dependence on large-scale labeled video target segmentation data, but also leverages the powerful zero-shot image segmentation capabilities of the Segment Anything Model (SAM) and category-independent key point and bounding box tracking to achieve robust zero-shot video target segmentation.
[0039] 2. In order to select the most suitable candidate box within the target box tracking branch, this invention innovatively designs a target box selection module based on a memory mechanism. By considering the box selection results of past frames, accurate target box tracking is achieved. Simultaneously, during the box selection process, the box selection problem is transformed into a typical bipartite graph matching problem. The Hungarian algorithm is used to find the bipartite graph with the minimum cost, avoiding multiple selections of the same candidate box.
[0040] 3. To further improve the quality of points and boxes, this invention designs a joint point and box filter to eliminate unreliable key points and target boxes based on the relative positional relationship between points and boxes, thereby obtaining accurate video target segmentation results through SAM.
[0041] 4. This invention utilizes existing point tracking, image description, candidate box generation, and image segmentation models without requiring further training. Furthermore, since key point tracking and target boxes are independent of the target category, this invention is designed to achieve accurate segmentation for unseen targets and videos with different data distributions. Attached Figure Description
[0042] Figure 1 This is a video target segmentation framework diagram based on joint tracking of key points and target boxes provided in an embodiment of the present invention:
[0043] Figure 2 This is a schematic diagram of the target box selection module based on a memory mechanism in the target box tracking branch provided in an embodiment of the present invention;
[0044] Figure 3 This is a flowchart of the target box tracking branch point box joint filter provided in an embodiment of the present invention. Detailed Implementation
[0045] In this embodiment, a video target segmentation method based on joint tracking of keypoints and bounding boxes is characterized by simultaneously deploying keypoint tracking branches and bounding box tracking branches to acquire keypoint information and bounding box information of the target. Furthermore, a memory-based bounding box selection module is designed in the bounding box tracking branch to improve the accuracy of bounding box tracking. Based on this, a joint point-bounding filter is designed to filter out unreliable points and boxes through further interaction between points and boxes, using the retained high-quality keypoints and bounding boxes as input to a Segmentation Aspect Model (SAM) to generate accurate video target segmentation results. Specifically, as... Figure 1 As shown, it includes the following steps:
[0046] Step 1: Obtain the video data to be segmented and the data annotations of the reference frames, and after preprocessing, obtain the text description of the target.
[0047] Step 1.1: Obtain the video data to be segmented, denoted as... Among them, I i Let i represent the i-th frame of the image, and T represent the number of frames in the video data; The first frame of the image is used as the reference frame, and the data of the reference frame is labeled and denoted as M;
[0048] Step 1.2: Obtain the pixel position of the target in the first frame image I1 using the data annotation M of the reference frame, and multiply the first frame image I1 and the annotation M to obtain the cropped image of the target.
[0049] Crop the image The first frame image I1 is input into the image description model for processing, resulting in a background-free text description T. object and text descriptions with background information T bg ;
[0050] T object and T bg To merge, into the target text description In this example, ChatGPT is used to process text T. object and T bg Polish and blend it into a single description.
[0051] Step 2: Construct a keypoint tracking branch to extract and track the key points of the target, obtaining the motion trajectory coordinates of the target key points, denoted as... Where N represents the number of target key points, and in this example, the number of key points is N = 17;
[0052] Step 2.1: Extract N key points from reference frame I1 to represent the target and background, denoted as...
[0053] The pixel coordinate sequence P of the target in the reference frame I1 is obtained using data annotation M. seq ;
[0054] P is selected using the K-centroid clustering algorithm. seq K in p K center points are used as positive sample points of the target, representing the target's position. In this example, the number of positive sample points of the target is K. p =16; in the remaining pixel coordinate sequence I1-P seq In the middle, K is randomly selected. n K sample points are used as negative sample points of the target, representing background information. In this example, the number of positive sample points of the target is K. n =1; by K p Positive sample points of each target and K n Constituting negative sample points
[0055] Step 2.2, Tracking Key Points:
[0056] The N keypoints P0 of the reference frame are input into the point tracking model to track the keypoints of the target in the video data. The movement within the [object], and output the key points in [the object]. The coordinates of the motion trajectory in
[0057] Step 3: Construct the target bounding box tracking branch for... Process the text to generate a description. The candidate bounding boxes are generated; and the candidate bounding box closest to the target position in the reference frame is stored in the memory bank. By comparing the candidate bounding boxes in the current frame image with the target bounding boxes stored in the memory bank, the best target bounding box in the current frame image is selected as the tracking result of the current frame image, so as to achieve target bounding box tracking.
[0058] Step 3.1, Generation of candidate boxes:
[0059] Text description and the t-th frame image I t The data is fed into a text-based object detection model to obtain candidate bounding boxes for the object in the t-th frame. Among them, B t F represents the coordinates of the candidate box. t Features representing candidate bounding boxes;
[0060] Step 3.2, Target Box Selection and Tracking:
[0061] Step 3.2.1: Take the smallest bounding rectangle of the data label M as the true bounding box B of the target. GT Calculate the coordinates B1 of the candidate bounding box and B1 of the ground truth bounding box in the first frame image I1. GT The intersection-union ratio (IUU) is calculated, and the candidate box with the highest IUU is selected, denoted as Q. box This represents the bounding box tracking result of the target in the first frame image I1.
[0062] Step 3.2.2: Construct a dynamically updated memory. in, This represents the selection result of the candidate bounding boxes for the ti-th frame image, and This represents the coordinates of the candidate bounding box in the ti-th frame of the image. The features representing the candidate bounding boxes of the ti-th frame image are represented by... Initialize the memory bank The selection results for each candidate box in the [database / framework].
[0063] Step 3.2.3: For the selection process of the bounding box in the subsequent t-th frame image, as follows... Figure 2 As shown. Computational memory. Memory Center Where, mean represents the average value. and They represent memory banks respectively. The coordinate center and feature center;
[0064] Calculate the image I in frame t t candidate boxes Features With memory bank Features cosine similarity in, and Represent the number of targets to be segmented and the image I in frame t, respectively. t candidate boxes Quantity;
[0065] Will The coordinates and features of candidate boxes that are smaller than the set threshold s are set to zero. In this example, the threshold s = 0.95.
[0066] The processed similarity matrix Convert to cost matrix
[0067] The Hungarian algorithm is used to solve for D, and the bipartite graph matching result with the minimum cost is obtained, which is the image I of frame t. t The selection result;
[0068] reserve Image I of frame t t The selection results are updated in the memory according to the first-in-first-out queue method. The remaining t-2 selection results are used to track the target bounding box, and the tracking result is recorded as follows.
[0069] Step 4, as follows Figure 3 As shown, a joint filter for points and bounding boxes is constructed to determine the relative positional relationship between points and bounding boxes. In this example, when the key points are within the bounding box, key points outside the bounding box are considered noise points and filtered out; when there are fewer than [number missing] key points... When the keypoint is located within the bounding box, in this example, the bounding box is treated as noise and filtered out, thereby affecting the motion trajectory coordinates. The tracking results are filtered out to obtain the filtered motion trajectory coordinates and tracking results. The retained points and bounding boxes are then input into the SAM model to obtain the segmentation results of each frame of the image, thereby completing the target segmentation of the video data.
[0070] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.
[0071] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.
[0072] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A video target segmentation method based on joint tracking of key points and target bounding boxes, characterized in that, Includes the following steps: Step 1: Obtain the video data to be segmented and the data annotations of the reference frames, and after preprocessing, obtain the text description of the target. Step 1.1: Obtain the video data to be segmented, denoted as... Among them, I i Let i represent the i-th frame of the image, and T represent the number of frames in the video data; The first frame of the image is used as the reference frame, and the data of the reference frame is labeled and denoted as M; Step 1.2: Process the reference frame using an image description model to generate a text description of the target, denoted as... Step 2: Construct a keypoint tracking branch to extract and track the key points of the target, obtaining the motion trajectory coordinates of the target key points, denoted as... Where N represents the number of target key points; Step 3: Construct the target bounding box tracking branch for... Process the text to generate a description. The candidate bounding boxes are generated; and the candidate bounding box closest to the target position in the reference frame is stored in the memory bank. By comparing the candidate bounding boxes in the current frame image with the target bounding boxes stored in the memory bank, the best target bounding box in the current frame image is selected as the tracking result of the current frame image, so as to achieve target bounding box tracking. Step 4: Construct a joint filter for the bounding box to analyze the coordinates of the motion trajectory. The tracking results are filtered out to obtain the filtered motion trajectory coordinates and tracking results, which are then input into the SAM model to obtain the segmentation results of each frame of the image, thereby completing the target segmentation of the video data.
2. The video target segmentation method based on joint tracking of key points and target boxes according to claim 1, characterized in that, Step 1.2 includes: The pixel position of the target in the first frame image I1 is obtained using the data annotation M of the reference frame. Then, the first frame image I1 and the annotation M are multiplied to obtain the cropped image of the target. Crop the image The first frame image I1 is input into the image description model for processing, resulting in a background-free text description T. object and text descriptions with background information T bg ; T object and T bg After fusion, the target text description 3. The video target segmentation method based on joint tracking of key points and target boxes according to claim 1, characterized in that, Step 2 includes: Step 2.1: Extract N key points from reference frame I1 to represent the target and background, denoted as... The pixel coordinate sequence P of the target in the reference frame I1 is obtained using data annotation M. seq ; P is selected using the K-centroid clustering algorithm. seq K in p The center points are used as positive sample points of the target, representing the position of the target; in the remaining pixel coordinate sequence I1-P seq In the middle, K is randomly selected. n Each sample point serves as a negative sample point for the target, representing background information; K p Positive sample points of each target and K n Constituting negative sample points Step 2.2, Tracking Key Points: The N keypoints P0 of the reference frame are input into the point tracking model to track the keypoints of the target in the video data. The movement within the [object], and output the key points in [the object]. The coordinates of the motion trajectory in 4. The video target segmentation method based on joint tracking of key points and target boxes according to claim 1, characterized in that, Step 3 includes: Step 3.1, Generation of candidate boxes: Text description and the t-th frame image I t The data is fed into a text-based object detection model to obtain candidate bounding boxes for the object in the t-th frame. Among them, B t F represents the coordinates of the candidate box. t Features representing candidate bounding boxes; Step 3.2, Target Box Selection and Tracking: Take the smallest bounding rectangle of the data label M as the ground truth bounding box B of the target. GT Calculate the coordinates B1 of the candidate bounding box and B1 of the ground truth bounding box in the first frame image I1. GT The intersection-union ratio (IUU) is calculated, and the candidate box with the highest IUU is selected, denoted as Q. box This represents the bounding box tracking result of the target in the first frame image I1. Build a dynamically updated memory in, This represents the selection result of the candidate bounding boxes for the ti-th frame image, and This represents the coordinates of the candidate bounding box in the ti-th frame of the image. The features representing the candidate bounding boxes of the ti-th frame image are represented by... Initialize the memory bank The selection results for each candidate box in the table; Computational memory Memory Center Where, mean represents the average value. and They represent memory banks respectively. The coordinate center and feature center; Calculate the image I in frame t t candidate boxes Features With memory bank Features cosine similarity in, and Represent the number of targets to be segmented and the image I in frame t, respectively. t candidate boxes Quantity; Will The coordinates and features of candidate boxes that are smaller than the set threshold s are set to zero; The processed similarity matrix Convert to cost matrix The Hungarian algorithm is used to solve for D, and the bipartite graph matching result with the minimum cost is obtained, which is the image I of frame t. t The selection result; reserve Image I of frame t t The selection results are updated in the memory according to the first-in-first-out queue method. The remaining t-2 selection results are used to track the target bounding box, and the tracking result is recorded as follows.
5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the video target segmentation methods of claims 1-4, and the processor is configured to execute the programs stored in the memory.
6. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the video target segmentation method according to any one of claims 1-4.