A video object tracking method and system

Abstract

This invention relates to a video object tracking method and system, belonging to the fields of computer vision and artificial intelligence. The method includes: determining a detection time based on the degree of dynamic change in the scene within the video; when the detection time is reached, generating a first mask for each potential object in the current video frame using a preset mask generator; determining the comprehensive confidence level of the first mask based on its segmentation quality confidence level, semantic consistency confidence level, and dynamic stability confidence level; then, modifying the tracking list of the video tracker based on the first masks, injecting new object masks, and pruning masks with persistently low dynamic stability confidence levels in the video tracker; finally, the video tracker continues to track target objects in the video based on the tracking list. This invention improves the tracking performance of the video tracker in long-term, complex scenes by adaptively modifying the mask frequency, injecting new objects, and pruning the mask.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a video object tracking method and system. Background Technology

[0002] Large-scale vision foundational models, represented by Segment Anything Model 2 (SAM2), have demonstrated enormous potential in the field of video object tracking due to their powerful zero-shot segmentation capabilities. Their existing application paradigm typically follows a "segment-and-track" process: First, in the initial frame of the video, an Automatic Mask Generator is used to segment all potential objects in the scene, generating initial masks for all potential objects; then, these initial masks are used as one-time "prompts" and input into a video tracker (such as the SAM2 model), which then propagates and predicts their positions in subsequent frames.

[0003] While this method is effective in simple scenarios, it has profound and fundamental limitations that severely restrict its application in real-world, complex, and long-term monitoring scenarios. These limitations include:

[0004] 1. Insufficient Self-Correction Capability: The tracking process is an open-loop system. Once an object deviates due to occlusion, deformation, or changes in lighting, its mask will become flawed. These flaws accumulate over time (i.e., error drift), eventually leading to tracking failure. While related techniques periodically correct the mask during propagation to suppress error drift, their dynamic adaptability and correction effectiveness are insufficient. Specifically, related techniques typically use an automatic mask generator to regenerate the mask periodically and directly correct the mask in the tracker based on this mask. However, automatic mask generators tend to produce massive amounts of redundant masks, including many meaningless background fragments, noise, or different parts of the same object. Directly using these masks as cues to correct the mask in the video tracker not only greatly increases the computational burden but may also introduce a large number of invalid tracking trajectories, leading to chaos in subsequent processing.

[0005] 2. The limitations of static initialization make it unsuitable for open-world scenarios: Existing methods rely entirely on the segmentation results of the first frame of the video. For targets that appear midway through the video (such as a car entering the frame), the lack of initial cues makes detection and tracking impossible. This makes it unsuitable for open scenarios where entities are constantly entering and exiting in the real world.

[0006] 3. Delayed handling of invalid trajectories: When a target is occluded or deformed, its mask often attaches to the surrounding background, creating "ghost trajectories" that deviate from the true target. The continuous accumulation of a large number of such low-value trajectories (such as waves or leaves) over long periods of operation severely strains computing and storage resources, leading to a significant decline in system performance over time. Current technologies typically passively wait for the trajectory to disappear or its confidence level to drop to zero, without proactive identification and pruning, resulting in untimely pruning. Summary of the Invention

[0007] In view of this, it is necessary to provide a video object tracking method and system to solve the problem of the lack of robustness and self-correction capability in existing video tracking.

[0008] To address the aforementioned problems, in a first aspect, the present invention provides a video object tracking method, comprising:

[0009] During the process of tracking target objects in a video using a video tracker, the detection time is determined based on the degree of dynamic change in the scene within the video.

[0010] When the detection time arrives, a first mask for each potential object in the current video frame is generated using a preset mask generator, and the overall confidence level of the first mask is determined. The overall confidence level is calculated based on segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence. The semantic consistency confidence level is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask. The dynamic stability confidence level is determined based on the deviation between the first predicted future position and the actual position.

[0011] Generate a tracking list; the tracking list includes a second mask of the target object generated by the video tracker in the current video frame;

[0012] When the overall confidence level of the target first mask that matches any second mask position meets the confidence level requirement, any second mask in the tracking list is replaced with the target first mask;

[0013] When the overlap between any first mask and each of the second masks is less than the first overlap threshold, and the overall confidence of any first mask is greater than the injection confidence threshold, the first mask is added to the tracking list.

[0014] When the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames, the mask is removed from the tracking list; wherein, the dynamic stability confidence of the mask at the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated;

[0015] The video tracker continues to track the target object in the video based on the tracking list.

[0016] In one possible implementation, the method further includes:

[0017] The degree of dynamic change in the scene is determined based on the rate of change of optical flow information and the rate of change of motion vector of the target object in the video.

[0018] In one possible implementation, the method further includes:

[0019] The video tracker generates the segmentation quality confidence score of the first mask.

[0020] In one possible implementation, the method further includes:

[0021] Based on the historical trajectory of the first mask, the first predicted future position of the first mask is obtained by using a Kalman filter or a linear regression model.

[0022] In one possible implementation, the method further includes:

[0023] The semantic recognition model is used to determine the semantic information corresponding to the first mask region and the first predicted future position.

[0024] The semantic consistency confidence of the first mask is determined based on the consistency between the first mask region and the semantic information corresponding to the first predicted future position.

[0025] In one possible implementation, determining the overall confidence level of the first mask includes:

[0026] The overall confidence level of the first mask is obtained by multiplying its segmentation quality confidence level, semantic consistency confidence level, and dynamic stability confidence level.

[0027] In one possible implementation, the method further includes:

[0028] After determining the overall confidence level of the first mask, the first mask with an overall confidence level lower than the first overall confidence level threshold is filtered out.

[0029] In one possible implementation, the confidence level requirement is that the overall confidence level is greater than a second overall confidence level threshold, or the difference between the overall confidence level of the target first mask and the confidence level of any second mask is greater than a difference threshold.

[0030] In one possible implementation, the tracking list further includes a mask identifier; the method further includes:

[0031] Add a mask identifier to any of the first masks.

[0032] In a second aspect, the present invention also provides a video object tracking system, comprising:

[0033] The detection time determination module is used to determine the detection time based on the degree of dynamic change of the scene in the video during the process of tracking the target object in the video by the video tracker.

[0034] The first mask generation and comprehensive confidence determination module is used to generate a first mask for each potential object in the current video frame through a preset mask generator when the detection time arrives, and determine the comprehensive confidence of the first mask; wherein, the comprehensive confidence is calculated based on the segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence; the semantic consistency confidence is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask; the dynamic stability confidence is determined based on the deviation between the first predicted future position and the actual position;

[0035] A tracking list generation module is used to generate a tracking list; the tracking list includes a second mask of the target object generated by the video tracker in the current video frame;

[0036] The mask correction module is used to replace any second mask in the tracking list with the target first mask when the overall confidence level of the target first mask that matches any second mask position meets the confidence level requirement.

[0037] The mask injection module is used to add any first mask to the tracking list when the overlap between any retained first mask and each of the second masks is less than the overlap threshold, and the overall confidence of any first mask is greater than the injection confidence threshold.

[0038] The mask pruning module is used to remove any mask from the tracking list when the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames; wherein, the dynamic stability confidence of any mask in the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated;

[0039] The tracking module is used to continue tracking the target object in the video based on the tracking list through the video tracker.

[0040] The beneficial effects of this invention are:

[0041] 1. This invention determines the detection time based on the degree of dynamic change of the scene in the video, which can realize a high detection frequency when the scene changes quickly and a low detection frequency when the scene changes slowly, thus enhancing the dynamic adaptability of the detection.

[0042] 2. When the detection time arrives, this invention first generates a first mask for each potential object in the current video frame using a preset mask generator. Then, it calculates the overall confidence score of the first mask based on its segmentation quality confidence score, semantic consistency confidence score, and dynamic stability confidence score, and filters out first masks with an overall confidence score lower than the confidence score threshold. Specifically, the segmentation quality confidence score indicates whether the first mask represents a complete and clear object; the semantic consistency confidence score indicates whether the first mask matches the user's intent and whether it has future potential (unstable targets that appear briefly, are about to be occluded, or are about to leave the frame have unclear semantic information about their future positions, leading to low semantic consistency confidence scores and thus a lack of future potential); and the dynamic stability confidence score indicates whether the trajectory of the first mask is stable and has sufficient predictability. Then, when the overall confidence score of the target first mask matching any second mask position meets the confidence score requirement, any second mask in the tracking list is replaced with the target first mask, thereby completing the correction of the second mask. This invention can ensure the quality of the mask in the corrected tracking list by using multi-dimensional confidence requirements, thereby improving the correction effect.

[0043] 3. In this invention, if the overlap between any retained first mask and each of the second masks is less than the overlap threshold, then that first mask is determined to be the mask of a new object. At this point, it is further determined whether the overall confidence level of that first mask is greater than the injected confidence threshold to confirm that the first mask truly represents the target object, rather than swaying leaves or lighting noise. If the overall confidence level is greater than the injected confidence threshold, that first mask is added to the tracking list, enabling the video tracker to promptly capture and track new target objects in open scenes.

[0044] 4. In this invention, when the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames, it is determined that its motion trajectory has become unpredictable, and it is very likely that it has disappeared or its tracking has completely failed. At this time, the mask is removed from the tracking list, which can release computing and storage resources in time, reduce the computing pressure of long-term tracking tasks, and also avoid the interference of invalid trajectories on scene analysis.

[0045] In summary, this invention dynamically determines the detection time and generates a high-quality first mask when the detection time is reached. Based on the high-quality first mask, the tracking list of the video tracker is modified, new objects are injected, and pruning is performed. This can improve the application effect of the video tracker in real, complex, and long-term monitoring scenarios without making any underlying modifications to the video tracker. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating an embodiment of the video object tracking method provided by the present invention;

[0048] Figure 2 This is a schematic diagram of an embodiment of the video object tracking system provided by the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] In the description of the embodiments of this invention, unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," etc., used in the embodiments of this invention are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor to indicate or imply their relative importance or implicitly specify the number of indicated technical features. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, and the number of objects is not limited; for example, a first object can be one or more.

[0051] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of the video object tracking method provided by the present invention, the method comprising:

[0053] S101, during the process of tracking the target object in the video using a video tracker, the detection time is determined based on the degree of dynamic change of the scene in the video.

[0054] Video trackers can be any large-scale vision-based model that employs a "segmentation-to-track" paradigm.

[0055] The target object is the object to be tracked, such as a person or a vehicle. During the tracking of the target object in a video using a video tracker, operations such as correction, injection, and pruning of the mask used by the video tracker during tracking can be performed based on an adaptive detection frequency. Specifically, the faster the scene changes dynamically in the video, the higher the detection frequency; conversely, the slower the scene changes dynamically, the lower the detection frequency.

[0056] S102, when the detection time is reached, the first mask of each potential object in the current video frame is generated by the preset mask generator, and the comprehensive confidence of the first mask is determined; wherein, the comprehensive confidence is calculated based on the segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence; the semantic consistency confidence is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask; the dynamic stability confidence is determined based on the deviation between the first predicted future position and the actual position.

[0057] The current video frame is the video frame corresponding to the detection time. For example, it can be the video frame whose timestamp is closest to the detection time.

[0058] Potential objects are objects that may be the target object. After generating the first mask for potential objects, the first mask can be stored in the candidate suggestion pool.

[0059] Segmentation Quality Confidence (SQC) represents the judgment on whether "this mask itself is a complete and clear object".

[0060] Semantic Consistency Confidence (SCC) characterizes the degree of consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask. The first predicted future position of the first mask is the predicted position of the first mask in a future video frame (with a timestamp after the current video frame), for example, the predicted position 5 frames later. The first predicted future position can be predicted based on the historical trajectory of the first mask, which is extracted from past video frames (with timestamps before the current video frame).

[0061] A high degree of semantic information consistency results in a high SCC (Search Conformity Control). A high SCC indicates that the first mask better matches the user's intent and is more forward-looking. However, for unstable targets that appear briefly, are about to be obscured, or are about to leave the frame, their semantic information about their future location may no longer be clear, resulting in a lower SCC.

[0062] Dynamic Stability Confidence (DSC) characterizes the degree of deviation (prediction residual) between the first predicted future position and the actual position. A high degree of deviation results in a low DSC, indicating that the trajectory of the first mask is unstable and has poor predictability; conversely, a low degree of deviation results in a high DSC, indicating that the trajectory of the first mask is stable and has good predictability.

[0063] The determination process of the overall confidence score (P_score) in this embodiment can be triggered by the detection time, last for a certain period of time, and obtained by combining past video frames, current video frames and future video frames.

[0064] S103, Generate a tracking list; the tracking list includes the second mask of the target object generated by the video tracker in the current video frame.

[0065] S104, when the overall confidence level of the target first mask that matches any second mask position meets the confidence level requirement, replace any second mask in the tracking list with the target first mask.

[0066] This step is used to correct the second mask in the tracking list.

[0067] For example, both the first mask and the second mask include multiple masks, where the first mask... i With the second mask j If the positions match (meaning the two are masks of the same object), then the first mask is determined. i The first mask of the target, if the first mask i If the overall confidence level meets the confidence requirement, then the first mask is used. i Replace the second mask in the tracking list j .

[0068] In a more specific implementation, the process of determining the first mask of the target can be as follows:

[0069] Calculate the first mask separately i The Intersection over Union (IoU) ratio with each second mask is calculated using the following formula:

[0070]

[0071] In the formula, Indicates the first mask. This indicates the second mask.

[0072] If IoU> ( The first overlap threshold (which can take values ​​such as 0.5 or 0.6) indicates that... and They correspond to the same object.

[0073] S105, when the overlap between any retained first mask and each of the second masks is less than the first overlap threshold, and the overall confidence of any first mask is greater than the injection confidence threshold, add any first mask to the tracking list.

[0074] This step is used to inject a new object mask into the tracking list.

[0075] For example, the first mask m If the overlap with each of the second masks is less than the first overlap threshold, then it is determined to be the mask of the new target object, and then the IF Score is checked. seg >Inject_Threshold, where Score seg The overall confidence level is represented by `Inject_Threshold`, which represents the injection confidence threshold. The `Inject_Threshold` can be set using methods such as empirical settings or determination based on actual test results from the video tracker. This implementation does not impose specific restrictions on its setting method. If the score... seg >Inject_Threshold, then the first mask m Add to the tracking list.

[0076] In a more specific implementation, the process of determining the new target object can be: calculating the first mask. m The intersection-union comparison is performed with each of the second masks, and the maximum IoU (Max_IoU) is determined. If Max_IoU < ( The second overlap threshold (which can take values ​​such as 0.1 or 0.05) indicates the first mask.m If it does not exist in the current tracking list, it is a new target object mask.

[0077] S106, when the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames, the mask is deleted from the tracking list; wherein, the dynamic stability confidence of any mask in the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated.

[0078] This step is used to prune the masks in the tracking list.

[0079] For example, starting with the current video frame, the mask in the tracking list is calculated once for each video frame. n The confidence level of dynamic stability, if it continues k Frame mask n If the dynamic stability confidence scores are all less than the dynamic stability confidence threshold, then the mask will be... n Remove from the tracking list.

[0080] It is important to note that the dynamic stability confidence is calculated based on the deviation between the predicted future position and the actual position, while the predicted future position is based on historical trajectory prediction. Therefore, for masks that do not yet have historical trajectories in the tracking list, i.e., masks generated for the first time, the dynamic stability confidence of the video frame (target video frame) at the time of generation cannot be calculated. Therefore, the dynamic stability confidence of the target video frame is set to a preset initial value to ensure the completeness of this step.

[0081] S107, the video tracker continues to track the target object in the video based on the tracking list.

[0082] After completing the above corrections, new object injections, and pruning, the video tracker continues to track the target objects in the video based on the tracking list.

[0083] It should be noted that this embodiment does not specify the order of the above-described modification, new object injection, and pruning steps.

[0084] The video object tracking method provided in this embodiment can be applied to a video object tracking system, which can be a software system running on a terminal device. The terminal device can be a tablet computer, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), mobile phone, etc. This embodiment does not impose any restrictions on the specific type of terminal device.

[0085] In summary, this embodiment has the following beneficial effects:

[0086] 1. This embodiment adopts an adaptive frequency detection method, which enhances the dynamic adaptability of the detection.

[0087] 2. This embodiment can ensure the quality of the mask in the corrected tracking list by using multi-dimensional confidence requirements, thereby improving the correction effect.

[0088] 3. In this embodiment, if a first mask of a new target object is found during detection, it is promptly added to the tracking list, enabling the video tracker to capture and track new target objects in open scenes in a timely manner.

[0089] 4. This embodiment can release computing and storage resources in a timely manner by actively identifying pruning, reducing the computing pressure of long-term tracking tasks, and also avoiding interference from invalid trajectories to scene analysis.

[0090] In some embodiments of the present invention, the video tracker is a SAM2 model.

[0091] Existing large-scale model trackers, such as the YOLO model, can detect new objects in a timely manner during tracking. However, the YOLO model is sensitive to "new objects" and is prone to "false positives," such as mistaking a swaying leaf for a car, or a change in lighting for a person. Furthermore, the YOLO model can only provide bounding boxes, lacking pixel-level deformation adaptation and semantic understanding, making it prone to introducing noise and fragmentation. High-quality masks, on the other hand, require complex segmentation networks.

[0092] The SAM2 model features "interactive segmentation / film-level editing," generating masks with extremely high precision, down to the pixel level. Its drawback is that it requires a user prompt and cannot automatically detect new objects, becoming increasingly off-target over time.

[0093] This embodiment combines the advantages of the SAM2 model and achieves the injection of high-quality, semantically clear entity masks through high-quality mask correction and injection operations, resulting in a cleaner tracking trajectory and supplementing YOLO's open-world discovery capabilities.

[0094] In some embodiments of the present invention, the method further includes:

[0095] The degree of dynamic change in the scene is determined by the rate of change of optical flow information and the rate of change of motion vector of the target object in the video.

[0096] Specifically, the rate of change of optical flow information and the rate of change of motion vector of the target object in each video frame can be aggregated to obtain the Scene Kinetic Entropy (SKE). The Scene Kinetic Entropy quantifies the degree of dynamic change of the scene in the video. The rate of change of optical flow information describes the instantaneous change in the direction and velocity of the target object's motion between adjacent video frames, reflecting the stability or abruptness of the motion pattern. The rate of change of motion vector describes how quickly the target object's motion vector (displacement and direction) changes over time, reflecting the degree of acceleration or curvature of the motion trajectory.

[0097] In one example, the scene dynamic entropy can be calculated using the following formula:

[0098] SKE t =Exercise intensity factor exp(λ )

[0099] In the formula, SKE t Indicates the first t The scene dynamic entropy of a video frame, where exp() represents the exponential function. λ This represents an adjustment coefficient (constant) used to control the amplification factor of the final result by the degree of disorder. This represents the motion disorder factor. The motion intensity factor is calculated as follows:

[0100]

[0101] In the formula, N This indicates the total number of tracked objects (or key points) in the scene. m i Indicates the first i The dynamic change vector of the tracked object (i.e., the weighted sum of velocity and acceleration).

[0102] The motion disorder factor is calculated as follows:

[0103]

[0104] In the formula, the motion disorder factor represents the Shannon entropy of the distribution of motion directions of all tracked objects in the scene. This indicates that the direction of motion of a tracked object falls within the first... k The probability of each directional interval. K Indicates the number of directional intervals.

[0105] When a large number of tracked objects in a scene move rapidly and randomly (such as when a crowd suddenly becomes chaotic), the SKE value will increase significantly; conversely, when the scene is static or the tracked objects move slowly and orderly, the SKE value will remain low.

[0106] In some embodiments of the present invention, the dynamic detection interval (ADI) can be adjusted based on SKE. That is, the time interval T for the next activation of the automatic mask generator is no longer fixed, but can be determined based on the inverse function of SKE, T = f(1 / SKE).

[0107] A. High-entropy state (scene chaos): SKE increases, causing T to shorten significantly (e.g., from 100 frames to 30 frames). The system will initiate the "split everything" operation more frequently to capture rapidly changing new objects or dramatic changes in object form.

[0108] B. Low-entropy state (stable scene): SKE decreases, leading to a significant increase in T (e.g., from 100 frames to 300 frames). The system will reduce unnecessary global segmentation to conserve computational resources and rely more on efficient tracking propagation.

[0109] When the current video frame number t When the next dynamic detection time t_next_detection is met, the system calls the automatic mask generator to generate a temporary "hint candidate pool" containing information about all potential objects at that time.

[0110] In some embodiments of the present invention, the method further includes:

[0111] The segmentation quality confidence score of the first mask is generated using a video tracker.

[0112] In some embodiments of the present invention, the method further includes:

[0113] Based on the historical trajectory of the first mask, the first predicted future position of the first mask is obtained by using a Kalman filter or a linear regression model.

[0114] In some embodiments of the present invention, the method further includes:

[0115] The semantic recognition model is used to determine the semantic information corresponding to the first mask region and the first predicted future position.

[0116] The semantic consistency confidence of the first mask is determined based on the consistency of the semantic information corresponding to the first mask region and the first predicted future position.

[0117] Specifically, the first mask region or the first predicted future location region can be input into the semantic recognition model to obtain its semantic label. For example, if the semantic label of both is "person", the labels are the same and the SCC is high.

[0118] In some embodiments of the present invention, the step of determining the overall confidence level of the first mask includes:

[0119] The overall confidence level of the first mask is obtained by multiplying its segmentation quality confidence level, semantic consistency confidence level, and dynamic stability confidence level.

[0120] In this embodiment, any low confidence level will result in a low overall confidence level, thereby ensuring a higher reliability of the overall confidence level.

[0121] In some embodiments of the present invention, the method further includes:

[0122] After determining the overall confidence level of the first mask, the first mask with an overall confidence level lower than the first overall confidence level threshold is filtered out.

[0123] In this embodiment, a first mask with a comprehensive confidence level lower than the first comprehensive confidence level threshold can be filtered out from the candidate suggestion pool to obtain a refined candidate suggestion pool. Subsequent correction and new object injection operations are performed based on the refined candidate suggestion pool, thereby improving the efficiency of correction and new object injection operations.

[0124] Furthermore, the first overall confidence threshold is less than the injection confidence threshold.

[0125] In some embodiments of the present invention, the confidence level requirement is that the overall confidence level is greater than a second overall confidence level threshold, or the difference between the overall confidence level of the target first mask and the confidence level of any second mask is greater than a difference threshold.

[0126] Specifically, the confidence level requirement can be expressed by the following formula:

[0127]

[0128] In the formula, This represents the overall confidence level of the first mask. This indicates the confidence level of the second mask, and High_Threshold represents the second overall confidence threshold. This is a threshold value to prevent frequent jittering.

[0129] The second comprehensive confidence threshold is greater than the first comprehensive confidence threshold. The methods for setting the second comprehensive confidence threshold and the first comprehensive confidence threshold may include: setting based on experience, determining based on the measured effect of the video tracker, etc. This implementation does not impose specific restrictions on the setting methods.

[0130] The confidence level of the second mask can be measured by the stability of the second mask, for example, by the prediction residual corresponding to the second mask.

[0131] In some embodiments of the present invention, the tracking list further includes a mask identifier; the method further includes:

[0132] Add a mask identifier to any first mask.

[0133] Specifically, once any first mask is identified as the mask of a new target object, an ID is assigned to it, for example, ID:103, and it is added to the tracking list. Then, a new video tracker instance is initialized.

[0134] In summary, the video tracker of this invention performs efficient tracking propagation within an adaptive time interval determined by the scene's dynamic entropy. Whenever a dynamic detection time point is reached, a high-confidence refined cue pool, filtered by motion prediction and multi-dimensional confidence, is generated. Subsequently, a lifecycle management engine based on "prediction-verification" and "confidence arbitration" performs precise addition, modification, and deletion operations on the video tracker's tracking object list based on the refined cue pool.

[0135] The core of this invention lies in the intelligent management process of the Prompt information flow, rather than modification of the underlying AI model. It cleverly coordinates the "segmentation" and "tracking" capabilities of a video tracker. All proposed steps can be easily implemented based on existing code and standard computer vision libraries without any model retraining, demonstrating high engineering value and feasibility.

[0136] Reference Figure 2 The diagram illustrates a structural schematic of an embodiment of the video object tracking system provided by the present invention. The system 200 includes:

[0137] The detection time determination module 201 is used to determine the detection time based on the degree of dynamic change of the scene in the video during the process of tracking the target object in the video by the video tracker.

[0138] The first mask generation and comprehensive confidence determination module 202 is used to generate a first mask for each potential object in the current video frame through a preset mask generator when the detection time is reached, and determine the comprehensive confidence of the first mask; wherein, the comprehensive confidence is calculated based on the segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence; the semantic consistency confidence is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask; the dynamic stability confidence is determined based on the deviation between the first predicted future position and the actual position;

[0139] The tracking list generation module 203 is used to generate a tracking list; the tracking list includes the second mask of the target object generated by the video tracker in the current video frame;

[0140] The mask correction module 204 is used to replace any second mask in the tracking list with the target first mask when the overall confidence of the target first mask that matches any second mask position meets the confidence requirement.

[0141] The mask injection module 205 is used to add any first mask to the tracking list when the overlap between any retained first mask and each second mask is less than the overlap threshold and the overall confidence of any first mask is greater than the injection confidence threshold.

[0142] The mask pruning module 206 is used to remove any mask from the tracking list when the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames; wherein, the dynamic stability confidence of any mask in the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated.

[0143] Tracking module 207 is used to continue tracking target objects in the video based on the tracking list via a video tracker.

[0144] It should be noted that the implementation principles or processes of the above modules can be referred to the aforementioned implementation examples of the video object tracking method, and will not be elaborated here.

[0145] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0146] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video object tracking method, characterized in that, include: During the process of tracking target objects in a video using a video tracker, the detection time is determined based on the degree of dynamic change in the scene within the video. When the detection time arrives, a first mask for each potential object in the current video frame is generated using a preset mask generator, and the overall confidence level of the first mask is determined. The overall confidence level is calculated based on segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence. The semantic consistency confidence level is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask. The dynamic stability confidence level is determined based on the deviation between the first predicted future position and the actual position. Generate a tracking list; the tracking list includes a second mask of the target object generated by the video tracker in the current video frame; When the overall confidence level of the target first mask that matches any second mask position meets the confidence level requirement, any second mask in the tracking list is replaced with the target first mask; When the overlap between any first mask and each of the second masks is less than the first overlap threshold, and the overall confidence of any first mask is greater than the injection confidence threshold, the first mask is added to the tracking list. When the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames, the mask is removed from the tracking list; wherein, the dynamic stability confidence of the mask at the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated; The video tracker continues to track the target object in the video based on the tracking list.

2. The video object tracking method according to claim 1, characterized in that, The method further includes: The degree of dynamic change in the scene is determined based on the rate of change of optical flow information and the rate of change of motion vector of the target object in the video.

3. The video object tracking method according to claim 1, characterized in that, The method further includes: The video tracker generates the segmentation quality confidence score of the first mask.

4. The video object tracking method according to claim 1, characterized in that, The method further includes: Based on the historical trajectory of the first mask, the first predicted future position of the first mask is obtained by using a Kalman filter or a linear regression model.

5. The video object tracking method according to claim 1, characterized in that, The method further includes: The semantic recognition model is used to determine the semantic information corresponding to the first mask region and the first predicted future position. The semantic consistency confidence of the first mask is determined based on the consistency between the first mask region and the semantic information corresponding to the first predicted future position.

6. The video object tracking method according to claim 1, characterized in that, Determining the overall confidence level of the first mask includes: The overall confidence level of the first mask is obtained by multiplying its segmentation quality confidence level, semantic consistency confidence level, and dynamic stability confidence level.

7. The video object tracking method according to claim 1, characterized in that, The method further includes: After determining the overall confidence level of the first mask, the first mask with an overall confidence level lower than the first overall confidence level threshold is filtered out.

8. The video object tracking method according to claim 1, characterized in that, The confidence level requirement is that the overall confidence level is greater than the second overall confidence level threshold, or the difference between the overall confidence level of the first target mask and the confidence level of any second mask is greater than the difference threshold.

9. The video object tracking method according to claim 1, characterized in that, The tracking list also includes a mask identifier; the method further includes: Add a mask identifier to any of the first masks.

10. A video object tracking system, characterized in that, include: The detection time determination module is used to determine the detection time based on the degree of dynamic change of the scene in the video during the process of tracking the target object in the video by the video tracker. The first mask generation and comprehensive confidence determination module is used to generate a first mask for each potential object in the current video frame through a preset mask generator when the detection time arrives, and determine the comprehensive confidence of the first mask; wherein, the comprehensive confidence is calculated based on the segmentation quality confidence, semantic consistency confidence, and dynamic stability confidence; the semantic consistency confidence is determined based on the consistency between the semantic information of the first mask region and the semantic information of the first predicted future position of the first mask; the dynamic stability confidence is determined based on the deviation between the first predicted future position and the actual position; A tracking list generation module is used to generate a tracking list; the tracking list includes a second mask of the target object generated by the video tracker in the current video frame; The mask correction module is used to replace any second mask in the tracking list with the target first mask when the overall confidence level of the target first mask that matches any second mask position meets the confidence level requirement. The mask injection module is used to add any first mask to the tracking list when the overlap between any retained first mask and each of the second masks is less than the overlap threshold, and the overall confidence of any first mask is greater than the injection confidence threshold. The mask pruning module is used to remove any mask from the tracking list when the dynamic stability confidence of any mask in the tracking list is less than the dynamic stability confidence threshold for multiple consecutive frames; wherein, the dynamic stability confidence of any mask in the target video frame is a preset initial value, and the target video frame is the video frame when the mask is first generated; The tracking module is used to continue tracking the target object in the video based on the tracking list through the video tracker.

Images