Zero-shot multi-object tracking method and device, electronic equipment and storage medium

Abstract

The application provides a zero-shot multi-target tracking method and device, electronic equipment and storage medium, and belongs to the technical field of target tracking. The method comprises the following steps: acquiring continuous video frame sequence data and semantic query data; extracting a space-time feature sequence of the continuous video frame sequence data, and performing semantic space mapping on each graphic-text query data to obtain at least one semantic vector; determining at least one candidate detection box corresponding to each video frame based on the space-time feature sequence and the at least one semantic vector; calculating the smooth confidence of a specific target in each candidate detection box in each video frame; in response to the smooth confidence meeting a trigger condition, determining an enhancement level of trajectory information corresponding to the candidate detection box; determining a tracking area and an enhancement mode of the specific target based on the enhancement level of the trajectory information, and tracking the specific target based on the tracking area and the enhancement mode of the specific target. The application can improve the stability of target tracking.

Description

Technical Field

[0001] This application belongs to the field of target tracking technology, and more specifically, relates to a zero-shot multi-target tracking method and apparatus, electronic device, and storage medium. Background Technology

[0002] Multiple object tracking (MOT) is widely used in video surveillance, autonomous driving, and intelligent security. Its task is to detect multiple objects in a video and maintain consistency of identity across frames. Existing MOT methods mostly rely on training with large-scale labeled data, and their applicability is limited to known categories. When new object categories are introduced, detection and tracking performance drops significantly. Summary of the Invention

[0003] The purpose of this application is to provide a zero-shot multi-target tracking method, device, electronic device, and storage medium, which enables users to detect and track targets in videos without providing labeled training samples for target categories, and only need to input text descriptions or reference images, thereby improving the stability of target tracking.

[0004] A first aspect of this application provides a zero-shot multi-target tracking method, comprising:

[0005] Acquire continuous video frame sequence data and semantic query data, the semantic query data including at least one image / text query data;

[0006] Extract the spatiotemporal feature sequence of continuous video frame sequence data, and perform semantic space mapping on each image and text query data to obtain at least one semantic vector. Based on the spatiotemporal feature sequence and at least one semantic vector, determine at least one candidate detection box corresponding to each video frame.

[0007] Calculate the smooth confidence of a specific target in each candidate detection box across all video frames;

[0008] In response to the smooth confidence level meeting the trigger condition, the enhancement level of the trajectory information corresponding to the candidate detection box is determined based on the smooth confidence level and the environmental quality signal affecting the extraction of candidate detection boxes;

[0009] The enhancement level based on trajectory information determines the tracking area and enhancement method for a specific target, and the target is tracked based on the tracking area and enhancement method.

[0010] A second aspect of this application provides a zero-shot multi-target tracking device, comprising:

[0011] The data acquisition unit is used to acquire continuous video frame sequence data and semantic query data, wherein the semantic query data includes at least one image and text query data.

[0012] The feature extraction unit is used to extract the spatiotemporal feature sequence of continuous video frame sequence data, and to perform semantic space mapping on each image and text query data to obtain at least one semantic vector, and to determine at least one candidate detection box corresponding to each video frame based on the spatiotemporal feature sequence and at least one semantic vector.

[0013] The calculation unit is used to calculate the smooth confidence of a specific target in each candidate detection box across various video frames;

[0014] The conditional triggering unit is used to determine the enhancement level of the trajectory information corresponding to the candidate detection box based on the smooth confidence level and the environmental quality signal that affects the extraction of the candidate detection box, in response to the smooth confidence level meeting the triggering condition.

[0015] The target tracking unit is used to determine the tracking area and enhancement method of a specific target based on the enhancement level of trajectory information, and to track the specific target based on the tracking area and enhancement method.

[0016] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the zero-shot multi-target tracking method described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the zero-shot multi-target tracking method described above.

[0018] The advantages of the zero-shot multi-target tracking method, apparatus, electronic device, and storage medium provided in this application are as follows:

[0019] This application embodiment extracts spatiotemporal feature sequences from continuous video frame data and combines them with semantic vectors obtained by mapping image and text query data to determine candidate detection boxes. This enables more accurate location of potential targets in the video, providing a more reliable foundation for subsequent tracking. Furthermore, this embodiment calculates the smoothness confidence of specific targets within the candidate detection boxes and determines the enhancement level of trajectory information based on the smoothness confidence and environmental quality signals. This, in turn, determines the tracking area and enhancement method. This multi-dimensional information utilization approach effectively reduces tracking errors caused by environmental interference or changes in the target itself, improving the accuracy of tracking specific targets. This application embodiment uses zero-shot tracking and utilizes semantic query data for target tracking, without relying on pre-training with specific samples. This allows it to adapt to target tracking needs in more diverse types and scenarios, exhibiting broader applicability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a zero-shot multi-target tracking method provided in an embodiment of this application.

[0022] Figure 2 A schematic diagram of multiple consecutive video frames provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram illustrating semantic space mapping of graphic data provided in an embodiment of this application;

[0024] Figure 4 This is a structural block diagram of a zero-shot multi-target tracking device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] To facilitate understanding, the main terms used in this application will be explained first.

[0028] Zero-shot: refers to the fact that no labeled samples are provided for a specific category during the training phase, but the model can still complete the detection and tracking of new categories or open scene targets based on text descriptions or image examples during the inference / application phase.

[0029] Multi-object tracking (MOT) is an important task in computer vision, aiming to continuously track multiple targets (such as pedestrians, vehicles, and animals) in a video frame sequence, outputting information such as the position and motion trajectory of each target in different video frames, and maintaining the consistency of target identity (i.e., the same target is assigned a unique ID throughout the video to avoid identity confusion). MOT is widely used in video surveillance, autonomous driving, and intelligent security.

[0030] Intersection over Union (IoU) is a metric in computer vision that measures the degree of spatial overlap between two regions. It is calculated by dividing the area of ​​the intersection of the two regions by the area of ​​their union.

[0031] Non-Maximum Suppression (NMS) is a commonly used post-processing algorithm in the field of object detection. Its basic principle is to first sort all candidate boxes according to the similarity scores of candidate objects, and select the candidate box with the highest score as the retained result; then calculate the overlap between the remaining candidate boxes and the retained box. When the overlap exceeds a set threshold, it is judged as redundant detection and directly deleted; then repeat the process from the remaining candidates until all candidates have been processed.

[0032] Soft Non-Maximum Suppression (Soft-NMS) is a post-processing algorithm in object detection. Traditional NMS directly deletes low-scoring boxes with an overlap exceeding a threshold when processing candidate boxes, especially in dense object scenes, which may lead to missed detections. Therefore, to address this issue, Soft-NMS does not directly delete overlapping low-scoring boxes, but instead achieves suppression by reducing their confidence scores. Specifically, for candidate boxes with high overlap with high-scoring boxes, their confidence scores are reduced proportionally based on the degree of overlap. If the candidate box's score is still higher than a preset threshold after reducing the confidence score, it can still proceed to subsequent steps. This preserves potentially useful candidate boxes while reducing the impact of duplicate boxes, effectively improving performance in scenarios such as dense object detection.

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0034] Traditional multi-target tracking methods employ a pipeline architecture of "detection + re-identification + association," which, due to its multi-stage processing, leads to problems such as low inference efficiency, error accumulation, and insufficient robustness.

[0035] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a zero-shot multi-target tracking method provided in an embodiment of this application. The method can be executed by an electronic device and may include steps S101 to S105.

[0036] This method is a zero-shot multi-object tracking approach driven by a multimodal large model. It can automatically locate and continuously track semantically matched targets based solely on a continuous sequence of video frames input by the user and one or more semantic query data, without requiring prior training on target categories. Furthermore, it supports parallel processing of multiple targets, significantly improving the robustness of the large model and the stability of target tracking. The specific steps are as follows:

[0037] S101: Obtain continuous video frame sequence data and semantic query data, the semantic query data including at least one image / text query data.

[0038] In this embodiment, the format of the continuous video frame sequence data can be [batch size B, number of frames T, number of channels C, height H, width W], for example, a 600-frame 1080P video (reference). Figure 2 Semantic query data includes at least one image-text query. Image-text query data can be text, such as "a person carrying a backpack," or an image, such as a photo of a bicycle, or a mixture of image and text data. Semantic query data can have multiple image-text queries, such as multiple text queries "track that person" and "track that backpack."

[0039] S102: Extract the spatiotemporal feature sequence of continuous video frame sequence data, and perform semantic space mapping on each image and text query data to obtain at least one semantic vector. Based on the spatiotemporal feature sequence and at least one semantic vector, determine at least one candidate detection box corresponding to each video frame.

[0040] In this embodiment, both the acquired video frame sequence data and semantic query data need to be preprocessed. Specifically, the video frame sequence data can be input into a "video understanding backbone network," such as a Video Masked Autoencoder (VideoMAE) or a Time-Space Transformer (TimeSformer), to extract spatial features frame by frame. Then, through temporal modeling, the information from adjacent video frames is concatenated to obtain a "spatiotemporal feature sequence." This spatiotemporal feature sequence contains the spatial structure and temporal context of each frame, providing a basic reference for the next step of generating candidate detection boxes.

[0041] For each text / image query, a pre-trained large-scale text / image understanding model can be invoked, such as a Vision Language (VL) model or a Contrastive Language-Image Pretraining (CLIP) model, to uniformly convert the user's input text or image into a fixed-length semantic vector. This semantic vector can be understood as the "digital fingerprint" of the query target.

[0042] refer to Figure 3 For text input, the large-scale image-text understanding model will output a vector representing the overall semantics after understanding the entire sentence. Figure 3 (Upper part); For image input, the large image-text understanding model extracts the visual semantic features of the entire image ( Figure 3 (Lower half); Regardless of whether the input is text or an image, they are ultimately mapped to the same semantic space to ensure fair comparisons can be made later.

[0043] In one embodiment, determining at least one candidate detection box corresponding to each video frame based on a spatiotemporal feature sequence and at least one semantic vector includes:

[0044] Based on the spatiotemporal feature sequence, multiple first candidate target boxes are obtained for each video frame using a target detection algorithm; at least one candidate detection box for each video frame is obtained by filtering from the multiple first candidate target boxes based on at least one semantic vector.

[0045] In this embodiment, for the spatiotemporal feature sequence corresponding to a video frame, an object detection algorithm can be used to detect each video frame separately, automatically generating several first candidate target boxes in each frame, and extracting the visual feature vector corresponding to each first candidate target box. (See reference...) Figure 5 The image shows a series of possible targets (only a portion are shown in the example; the actual number of candidate boxes generated is much greater).

[0046] In this embodiment, the object detection algorithm can be a classless detection head, similar to the DERT structure, but without the class classification branch found in traditional detectors. An object detection algorithm is essentially an object detection model; the "detection head" is the model's output module. Traditional detection heads typically contain two core branches:

[0047] Category classification branch: Predict the category of each target (such as "person", "car", "cat" etc.).

[0048] Bounding box regression branch: Predicts the location and size of each object (e.g., coordinates (x, y, w, h)).

[0049] The classless detection head in this application retains only the bounding box regression branch and removes the category classification branch. The output only contains the target's location information (bounding box) and does not include any category label (e.g., it no longer predicts "this is a cat" or "this is a dog"). After the above processing, the resulting first candidate target boxes are themselves "classless" and can be matched with semantic vectors to filter the first candidate target boxes, obtaining at least one candidate detection box for each video frame, thereby realizing cross-modal open-world detection and tracking, and better adapting to new categories and complex scenes.

[0050] In one embodiment, at least one candidate detection box corresponding to each video frame is obtained by filtering from multiple first candidate target boxes based on at least one semantic vector, including:

[0051] Each semantic vector is compared with multiple visual feature vectors to obtain multiple semantic similarities. Based on the comparison results of multiple semantic similarities with similarity thresholds, multiple second candidate target boxes are obtained.

[0052] Based on a sliding time window with the target frame length, multiple second candidate target boxes are filtered to obtain at least one candidate detection box corresponding to each video frame.

[0053] In this embodiment, the similarity between each semantic vector and the visual feature vector can be calculated based on a similarity calculation formula, such as the cosine similarity formula. Prior to this, to ensure the consistency of dimensions and numerical stability between the semantic vector and the visual feature vector, the vectors are first subjected to L2 normalization. The cosine similarity formula is as follows:

[0054]

[0055] Where q represents the semantic vector. This represents the i-th visual feature vector. This represents the similarity between the semantic vector and the visual feature vector.

[0056] In this embodiment, the similarity threshold can be preset. When the similarity score between the semantic vector and the visual feature vector, i.e., the similarity score of the first candidate target box, is lower than the similarity threshold, the target is directly eliminated, thereby preventing irrelevant targets from entering the subsequent processing flow. Preferably, the similarity threshold can be a fixed value, such as any value between 0.25 and 0.35 rounded to two decimal places.

[0057] In this embodiment, the similarity threshold can also be adaptively set based on the similarity score distribution of each video frame. For example, the 80th percentile can be used as the threshold. This approach can improve robustness in different video frame scenarios.

[0058] In this embodiment, in order to suppress transient false detections, the second candidate target box within the sliding time window of the target frame length can be retained, while the second candidate target box outside the sliding time window of the target frame length can be removed, thus obtaining at least one candidate detection box corresponding to each video frame.

[0059] In this embodiment, when multiple image-text query data are input, the large model calculates the similarity between each image-text query and the first candidate target box in parallel, and performs steps such as similarity threshold filtering and adaptive candidate target box retention. Preferably, each image-text query has its own independently set similarity threshold and target frame length to adapt to different semantic retrieval difficulties and target densities. Through the above steps, the large model can output multiple independent matching results in the same video frame, and maintain the continuity of the trajectory ID of each independent matching result in subsequent operations.

[0060] S103: Calculate the smooth confidence of a specific target in each candidate detection box across all video frames.

[0061] In this embodiment, continuous video frame sequences may exhibit situations where the target is partially occluded, deformed, or experiences sudden changes in illumination. These changes can cause a sharp drop in the smooth confidence of a specific target within a candidate detection box. To address this, this application reinforces the existing target tracking process based on smooth confidence as needed. Specifically, the smooth confidence of a candidate detection box in the current video frame can be calculated based on the original similarity score of that candidate detection box and the smooth confidence of the candidate detection box corresponding to the same specific target in the previous video frame. The calculation formula is as follows:

[0062] .

[0063] in, The original similarity score. This represents the smooth confidence level in the previous video frame. The smooth confidence level of the current video frame.

[0064] In this embodiment, by updating the similarity scores corresponding to each candidate detection box using an exponential moving average, single-frame fluctuations can be suppressed, making the selected candidate detection boxes more accurate.

[0065] S104: In response to the smooth confidence level meeting the trigger condition, determine the enhancement level of the trajectory information corresponding to the candidate detection box based on the smooth confidence level and the environmental quality signal affecting the extraction of the candidate detection box.

[0066] In this embodiment, the triggering condition includes any one of the following:

[0067] The absolute value of the difference between the smoothness confidence of the current video frame and the smoothness confidence of the previous video frame is greater than or equal to the first value;

[0068] The smoothness confidence of the current video frame is less than the second value.

[0069] In this embodiment, after determining at least one candidate detection box corresponding to each video frame, the candidate detection boxes of a specific target in different video frames are associated to obtain the initial trajectory corresponding to the specific target.

[0070] For each initial trajectory, starting from the first time-series video frame, calculate the smooth confidence of the specific target in each candidate detection box in each video frame. If any of the following conditions are met, it is determined that the initial trajectory corresponding to the specific target needs to be corrected:

[0071] (1) The absolute value of the difference between the smooth confidence of the current video frame and the smooth confidence of the previous video frame is greater than or equal to the first value, that is, the decrease in the smooth confidence of the current video frame and the smooth confidence of the previous video frame is greater than or equal to the first value.

[0072] (2) The smoothness confidence of the current video frame is less than the second value, and the second value is greater than the first value.

[0073] If the smoothness confidence of consecutive H frames is greater than or equal to the preset smoothness threshold, then no processing is required for subsequent video frames. Based on the enhancement level of the trajectory information corresponding to the candidate detection box, the enhancement level is gradually reduced until it returns to the normal mode, that is, the mode that does not process the initial trajectory.

[0074] In one embodiment, the enhancement level of the trajectory information corresponding to the candidate detection box is determined based on the smoothness confidence level and the environmental quality signal affecting the extraction of the candidate detection box, including:

[0075] The environmental quality signal affecting the extraction of candidate detection boxes is normalized to obtain the normalized environmental quality signal;

[0076] If the smoothing confidence is less than the first smoothing confidence and greater than or equal to the second smoothing confidence, or if the normalized environmental quality signal is less than the first environmental quality signal threshold and greater than or equal to the second environmental quality signal threshold, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be mild enhancement.

[0077] If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be moderate enhancement.

[0078] If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is not in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be intensity enhancement.

[0079] In this embodiment, the environmental quality signals that affect the extraction of candidate detection boxes include, but are not limited to, the following four:

[0080] Target occlusion: A comprehensive evaluation based on the change in the area of ​​the candidate detection box, the edge-fitting ratio of the candidate detection box, and the sharpness of the target within the candidate detection box.

[0081] Illumination variation: determined based on the brightness of video frame images or the histogram distance between adjacent frames.

[0082] Style variation: determined based on histogram distance or structural similarity between adjacent frames.

[0083] Motion deviation: The deviation between the center position of the candidate detection box and the observation box is calculated based on the motion model.

[0084] The influence value can be obtained by normalizing the above four environmental quality signals and then summing them by weight, or by choosing one of them.

[0085] S105: Determine the tracking area and enhancement method of a specific target based on the enhancement level of the trajectory information, and track the specific target based on the tracking area and enhancement method.

[0086] In this embodiment, the tracking area and enhancement method of a specific target are determined based on the enhancement level of the trajectory information. For example, when the enhancement level is mild, a steady-state conservative approach is adopted, and the tracking area of ​​the specific target remains unchanged. The enhancement method can be selected from one of the following four methods:

[0087] (1) Threshold fine-tuning: Reduce the similarity threshold of candidate detection boxes.

[0088] (2) Template update: The target occlusion degree is greater than the preset occlusion degree threshold or the smooth confidence degree is less than the second value.

[0089] (3) Lightweight robust scoring: Perform training-free augmentation on the candidate detection box or the target in it, such as increasing brightness or contrast, reducing light blur or modifying the size of the candidate detection box, and take the maximum similarity score after augmentation as the final score.

[0090] (4) Matching reweighting: Based on the motion anomaly signal, appropriately increase the weight of the motion consistency term to reduce the single-point dependence on the template appearance score.

[0091] In one embodiment, template updates are performed when the target occlusion degree is greater than a preset occlusion degree threshold or the smoothness confidence score is less than a second value. The update rate can be adaptively updated according to the target occlusion degree and the smoothness confidence score to avoid low-quality candidate detection box samples contaminating the template. Specifically, when the template needs to be updated to adapt to changes in the appearance of a specific target, such as turning or posture changes, the template update can be performed based on the following formula:

[0092]

[0093] in, This is the updated template for the q-th second candidate target box in the current video frame. The weighting coefficients are adaptively calculated from the smoothness confidence and target occlusion degree of a specific target. This is the updated template for the q-th second candidate target bounding box in the previous video frame. The candidate features are the q-th second candidate target box in the current video frame.

[0094] When the enhancement level is moderate, a local expansion approach is adopted to locally expand the tracking region. The enhancement method can be one of the following two methods:

[0095] (1) Local search expansion: Expand the region proportionally (e.g., 1.5×) with the candidate detection box of the previous video frame as the center, and re-perform the classless detection head within the expanded region to improve the candidate coverage.

[0096] (2) Template integration matching: Use both short-term and long-term templates for scoring, and take the maximum value or weighted result.

[0097] In one embodiment, to address rapidly changing target poses and maintain a stable appearance, short-term and long-term templates can be maintained, and their weights can be dynamically adjusted based on the stability of the appearance. The short-term template is used to quickly adapt to rapid changes in the target, while the long-term template is used to preserve the target's stable characteristics and avoid drift. The update formula is as follows:

[0098]

[0099] in, and These are weighting coefficients. , and These represent the short-term template and long-term template of the current video frame, respectively. and These represent the short-term template and long-term template corresponding to the previous video frame, respectively.

[0100] (3) Semantic query data augmentation:

[0101] Text query: Generate a small amount of semantic rewriting (such as "in low light", "partially occluded...") and merge it with the original semantic query data;

[0102] Image search: Enhance the contrast, color temperature, and blur of the reference image, and select the image with the highest similarity for matching.

[0103] When the enhancement level is intensity enhancement, a global expansion approach is adopted, and the tracking area is the entire frame range. The enhancement method can be selected from one of the following four methods:

[0104] (1) Full-frame re-detection: Increase the similarity threshold and perform classless re-detection across the entire frame;

[0105] (2) Appearance memory retrieval: For each initial trajectory, maintain a memory bank (limited-length queue) of historical high-confidence frames, calculate the maximum cosine similarity between the candidate and the memory vector, and use it as the re-identification score;

[0106] (3) Multiple hypothesis retention and verification: Select a preset number of the highest similarity scores (such as the top 5) to verify across 3 to 5 frames to prevent mistakenly rewriting other people's targets as the current ID;

[0107] (4) Cross-query suppression: When the same candidate detection box is hit with high scores by multiple semantic query conditions, "negative similarity to other queries" is introduced as a penalty to suppress mutual interference.

[0108] In one embodiment, if a specific target fails to match within n frames, the specific target will be cleared, releasing the ID resource corresponding to the specific target and avoiding excessive memory consumption.

[0109] In one embodiment, adaptive template updates are performed only when the enhancement level is mild or moderate. When the enhancement level is strong, the threshold for establishing new trajectory information can be raised to avoid generating new IDs due to relocation. If a specific target loses contact in a video frame, and the lost trajectory and the newly generated trajectory have multiple similarity matches within a specific time window, the IDs of the original lost trajectory are merged and retained.

[0110] The aforementioned mechanisms can automatically compensate for and recover targets when there are drastic changes in appearance or imaging conditions in the video, significantly reducing the risk of losing specific targets and ID switching, and improving the stability and continuity of multi-target tracking. For example, starting from frame 501 of the video, the similarity of a specific target with ID 001 drops sharply due to occlusion. At this time, the system automatically enters the intensity enhancement level, and the large model can still maintain stable tracking of the specific target.

[0111] In one embodiment, after the large model obtains the final trajectory information of a specific target, it outputs structured information for each video frame, specifically including:

[0112] The coordinates of the candidate detection box, the trajectory ID (a unique identifier that remains unchanged across frames), the query ID (indicating which image and text query data the specific target belongs to), and the matching confidence score (the semantic similarity score of the current video frame).

[0113] The output format is compatible with common tracking data standards (such as the MOT Challenge format) and can be directly used for downstream tasks such as security, video editing, and autonomous driving.

[0114] As can be seen from the above, on the one hand, the embodiments of this application, by extracting the spatiotemporal feature sequence of continuous video frame sequence data and combining it with the semantic vector obtained by mapping image and text query data to determine candidate detection boxes, can more accurately locate the possible target positions in the video, providing a more reliable foundation for subsequent tracking. In addition, this embodiment calculates the smoothness confidence of a specific target in the candidate detection box and determines the enhancement level of trajectory information based on the smoothness confidence and environmental quality signals, thereby determining the tracking area and enhancement method. This multi-dimensional information utilization method can effectively reduce tracking errors caused by environmental interference or changes in the target itself, improving the accuracy of tracking specific targets.

[0115] On the other hand, the embodiments of this application use zero samples and semantic query data for target tracking, which does not rely on pre-training of specific samples, and can adapt to target tracking needs in more different types and scenarios, thus having wider applicability.

[0116] In one embodiment of this application, after determining at least one candidate detection box corresponding to each video frame based on spatiotemporal feature sequences and semantic vectors, the zero-shot multi-object tracking method further includes:

[0117] Calculate the spatial overlap of a specific target in adjacent video frames and the similarity of its semantic features;

[0118] Based on similarity, candidate detection boxes of a specific target in different video frames are associated to obtain the initial trajectory corresponding to the specific target, and the ID of the initial trajectory corresponding to the specific target is determined.

[0119] In this embodiment, after determining at least one candidate detection box for each video frame based on the spatiotemporal feature sequence and semantic vector, each candidate detection box is still isolated. Therefore, it is necessary to associate the candidate detection boxes of the same specific target in different video frames to obtain the initial trajectory corresponding to the specific target, and assign a unique ID to the initial trajectory corresponding to the specific target. The initial trajectory obtained in this embodiment is not output, but needs to be corrected through zero-shot augmentation to output the corrected final trajectory.

[0120] Through the above steps, this embodiment can stably track a specific target in a dynamic scene and ensure the consistency of the target ID.

[0121] In one embodiment of this application, multiple second candidate target boxes are obtained based on comparison results of multiple semantic similarities with a similarity threshold, including:

[0122] All first candidate bounding boxes with semantic similarity greater than or equal to the similarity threshold are selected as third candidate bounding boxes; the third candidate bounding boxes are then filtered based on the following filtering operation to obtain multiple second candidate bounding boxes;

[0123] Filtering operations include:

[0124] Determine the candidate upper limit value for each video frame;

[0125] When the number of third candidate bounding boxes corresponding to a video frame is greater than the upper limit of the number of candidates, the selection is based on the spatial location and semantic similarity of all third candidate bounding boxes corresponding to the video frame.

[0126] In this embodiment, when the number of third candidate bounding boxes corresponding to a video frame exceeds the upper limit value, filtering is performed based on the spatial location and semantic similarity of all third candidate bounding boxes corresponding to the video frame. Specifically, the semantic similarity scores of multiple third candidate bounding boxes are sorted from highest to lowest. For each video frame, an upper limit value is set. When the number of candidates for multiple third candidate bounding boxes exceeds the upper limit value, redundancy removal is performed using NMS or Soft-NMS based on the spatial location and semantic similarity scores of the third candidate bounding boxes.

[0127] This embodiment uses adaptive filtering to make the selection of multiple second candidate target boxes more accurate, which is beneficial for subsequent target tracking based on multiple second candidate target boxes.

[0128] In one embodiment of this application, filtering multiple second candidate target boxes based on a sliding time window with a target frame length to obtain at least one candidate detection box corresponding to each video frame includes:

[0129] Determining the spatial neighborhood range based on continuous video frame sequence data;

[0130] Multiple second candidate target boxes within the same spatial neighborhood and a time window that is greater than or equal to the target frame length are determined as candidate target boxes after lightweight matching. Lightweight matching is used to determine whether a specific target appears consecutively within the target frame length.

[0131] Calculate the smooth similarity score of the candidate target boxes, and based on the comparison result of the smooth similarity score and the similarity threshold, select at least one candidate detection box corresponding to each video frame from the candidate target boxes.

[0132] In this embodiment, the spatial neighborhood range is determined based on continuous video frame sequence data. For example, if a person's image appears in the middle of a video frame, the middle area can be used as the spatial neighborhood range.

[0133] In this embodiment, the target frame length is set to m. When similarity scores greater than the similarity threshold repeatedly appear within the same spatial neighborhood and within a time window of ≥m frames, multiple second candidate target boxes can be confirmed as candidate target boxes. In one embodiment, lightweight matching can be performed on the candidate target boxes to determine whether they represent consecutive occurrences of the same specific target. The specific operation is as follows:

[0134] First, construct the cost function:

[0135]

[0136] Where cost represents the cost function. This represents the intersection-union ratio (IoU) of the spatial positions of the current video frame t and the previous video frame t-1. Indicates the weighting coefficient. Let p be the candidate features of the p-th second candidate target box in the current video frame. Let λ be the candidate feature of the q-th second candidate target bounding box in the previous video frame. If cost ≤ γ, it is determined that the same specific target appears consecutively, where γ is a preset value. Preferably, λ = 0.5 and γ = 0.6. Here, the candidate feature represents the feature vector represented by the corresponding target bounding box.

[0137] This embodiment effectively suppresses the judgment error caused by single-frame fluctuations by constructing a cost function and performing a moving average on the similarity scores of the candidate target boxes, making the final determined candidate detection box more accurate.

[0138] Corresponding to the zero-shot multi-target tracking method in the above embodiments, Figure 4 This is a structural block diagram of a zero-shot multi-target tracking device provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 4 The zero-sample multi-target tracking device 20 includes: a data acquisition unit 21, a feature extraction unit 22, a calculation unit 23, a condition triggering unit 24, and a target tracking unit 25.

[0139] The data acquisition unit 21 is used to acquire continuous video frame sequence data and semantic query data, wherein the semantic query data includes at least one image and text query data.

[0140] The feature extraction unit 22 is used to extract the spatiotemporal feature sequence of continuous video frame sequence data, and to perform semantic space mapping on each image and text query data to obtain at least one semantic vector, and to determine at least one candidate detection box corresponding to each video frame based on the spatiotemporal feature sequence and at least one semantic vector.

[0141] The calculation unit 23 is used to calculate the smooth confidence of a specific target in each candidate detection box in each video frame;

[0142] The condition triggering unit 24 is used to determine the enhancement level of the trajectory information corresponding to the candidate detection box based on the smooth confidence and the environmental quality signal that affects the extraction of the candidate detection box in response to the smooth confidence meeting the triggering condition;

[0143] The target tracking unit 25 is used to determine the tracking area and enhancement method of a specific target based on the enhancement level of the trajectory information, and to track the specific target based on the tracking area and enhancement method of the specific target.

[0144] In one embodiment of this application, the feature extraction unit 22 is specifically used for:

[0145] Based on the spatiotemporal feature sequence, multiple first candidate target boxes corresponding to each video frame are obtained using a target detection algorithm;

[0146] At least one candidate detection box is obtained for each video frame by filtering from multiple first candidate target boxes based on at least one semantic vector.

[0147] In one embodiment of this application, each first candidate target box corresponds to a visual feature vector. The feature extraction unit 22 is specifically used for:

[0148] Each semantic vector is compared with multiple visual feature vectors to obtain multiple semantic similarities. Based on the comparison results of multiple semantic similarities with similarity thresholds, multiple second candidate target boxes are obtained.

[0149] Based on a sliding time window with the target frame length, multiple second candidate target boxes are filtered to obtain at least one candidate detection box corresponding to each video frame.

[0150] In one embodiment of this application, the feature extraction unit 22 is specifically used for:

[0151] All first candidate bounding boxes whose semantic similarity is greater than or equal to the similarity threshold are used as third candidate bounding boxes;

[0152] The third candidate bounding box is filtered based on the following filtering operation to obtain multiple second candidate bounding boxes;

[0153] Filtering operations include:

[0154] Determine the candidate upper limit value for each video frame;

[0155] When the number of third candidate bounding boxes corresponding to a video frame is greater than the upper limit of the number of candidates, the selection is based on the spatial location and semantic similarity of all third candidate bounding boxes corresponding to the video frame.

[0156] In one embodiment of this application, the feature extraction unit 22 is specifically used for:

[0157] Determining the spatial neighborhood range based on continuous video frame sequence data;

[0158] Multiple second candidate target boxes within the same spatial neighborhood and a time window that is greater than or equal to the target frame length are determined as candidate target boxes after lightweight matching. Lightweight matching is used to determine whether a specific target appears consecutively within the target frame length.

[0159] Calculate the smooth similarity score of the candidate target boxes, and based on the comparison result of the smooth similarity score and the similarity threshold, select at least one candidate detection box corresponding to each video frame from the candidate target boxes.

[0160] In one embodiment of this application, the zero-shot multi-target tracking device 20 further includes a trajectory generation unit, specifically used for:

[0161] Calculate the spatial overlap of a specific target in adjacent video frames and the similarity of its semantic features;

[0162] Based on similarity, candidate detection boxes of the feature target in different video frames are associated to obtain the initial trajectory corresponding to the feature target, and the ID of the initial trajectory corresponding to the feature target is determined.

[0163] In one embodiment of this application, the condition triggering unit 24 is specifically used for:

[0164] The environmental quality signal affecting the extraction of candidate detection boxes is normalized to obtain the normalized environmental quality signal;

[0165] If the smoothing confidence is less than the first smoothing confidence and greater than or equal to the second smoothing confidence, or if the normalized environmental quality signal is less than the first environmental quality signal threshold and greater than or equal to the second environmental quality signal threshold, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be mild enhancement.

[0166] If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be moderate enhancement.

[0167] If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is not in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be intensity enhancement.

[0168] In summary, based on the embodiments of the above-described zero-shot multi-target tracking method and apparatus, the key innovations to be protected by this application are as follows:

[0169] (1) This application can achieve cross-modal semantic alignment through multimodal large models, thereby improving adaptability to new categories and complex scenarios;

[0170] (2) This application adopts a single-stage detection + target tracking integrated scheme. Unlike the traditional two-stage detection which first generates candidate regions and then performs secondary classification, this application can directly predict candidate boxes and their visual feature vectors on the spatiotemporal feature sequence corresponding to video frames, achieving the goal in one step and improving recognition efficiency and model robustness.

[0171] (3) This application adopts a zero-sample augmentation mechanism, which improves the stability of tracking specific targets in long-term and complex scenarios by updating the template online.

[0172] (4) This application achieves cross-frame tracking of specific targets by judging cross-modal similarity and temporal consistency, thus ensuring the uniqueness of the ID.

[0173] (5) This application provides a zero-shot multi-target tracking method based on a large model, which allows users to detect and track targets in videos without having to provide labeled training samples for target categories. The user only needs to input text descriptions or reference images, making the operation more convenient.

[0174] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the units in the above-described device embodiments, for example... Figure 4 The functions of the data acquisition unit 21, feature extraction unit 22, calculation unit 23, condition triggering unit 24, and target tracking unit 25 are shown.

[0175] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0176] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0177] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory.

[0178] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the zero-shot multi-target tracking method provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0179] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0180] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0181] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0182] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A zero-shot multi-target tracking method, characterized in that, include: Acquire continuous video frame sequence data and semantic query data, wherein the semantic query data includes at least one image / text query data; Extract the spatiotemporal feature sequence of the continuous video frame sequence data, and perform semantic space mapping on each of the image and text query data to obtain at least one semantic vector. Based on the spatiotemporal feature sequence and the at least one semantic vector, determine at least one candidate detection box corresponding to each video frame. Calculate the smooth confidence of a specific target in each candidate detection box across all video frames; In response to the smooth confidence level satisfying the trigger condition, the enhancement level of the trajectory information corresponding to the candidate detection box is determined based on the smooth confidence level and the environmental quality signal affecting the extraction of the candidate detection box; Based on the enhancement level of the trajectory information, the tracking area and enhancement method of the specific target are determined, and the specific target is tracked based on the tracking area and enhancement method.

2. The zero-shot multi-object tracking method of claim 1, wherein, The step of determining at least one candidate detection box corresponding to each video frame based on the spatiotemporal feature sequence and the at least one semantic vector includes: Based on the spatiotemporal feature sequence, multiple first candidate target boxes corresponding to each video frame are obtained using a target detection algorithm; Based on the at least one semantic vector, at least one candidate detection box corresponding to each video frame is obtained by filtering from the plurality of first candidate target boxes.

3. The zero-shot multi-target tracking method as described in claim 2, characterized in that, Each first candidate bounding box corresponds to a visual feature vector. The step of filtering from the plurality of first candidate bounding boxes to obtain at least one candidate detection box corresponding to each video frame based on the at least one semantic vector includes: Each semantic vector is compared with multiple visual feature vectors to obtain multiple semantic similarities. Based on the comparison results of the multiple semantic similarities with similarity thresholds, multiple second candidate target boxes are obtained. Based on a sliding time window with a target frame length, at least one candidate detection box is obtained for each video frame by filtering the multiple second candidate target boxes.

4. The zero-shot multi-object tracking method of claim 3, wherein, The process of obtaining multiple second candidate target boxes based on the comparison results of the multiple semantic similarities with similarity thresholds includes: All first candidate bounding boxes whose semantic similarity is greater than or equal to the similarity threshold are used as third candidate bounding boxes; The third candidate bounding box is filtered based on the following filtering operation to obtain multiple second candidate bounding boxes; The filtering operation includes: Determine the candidate upper limit value for each video frame; When the number of third candidate target boxes corresponding to the video frame is greater than the upper limit value of the candidates, the selection is performed based on the spatial location and semantic similarity of all third candidate target boxes corresponding to the video frame.

5. The zero-shot multi-object tracking method of claim 3, wherein, The step of filtering the multiple second candidate target boxes based on a sliding time window with a target frame length to obtain at least one candidate detection box corresponding to each video frame includes: The spatial neighborhood range is determined based on the continuous video frame sequence data; Multiple second candidate target boxes within the same spatial neighborhood and a time window that is greater than or equal to the target frame length are determined as candidate target boxes after lightweight matching. The lightweight matching is used to determine whether a specific target appears continuously within the target frame length. Calculate the smooth similarity score of the candidate target boxes, and based on the comparison result of the smooth similarity score and the similarity threshold, select at least one candidate detection box corresponding to each video frame from the candidate target boxes.

6. The zero-shot multi-object tracking method of claim 1, wherein, After determining at least one candidate detection bounding box corresponding to each video frame based on the spatiotemporal feature sequence and the semantic vector, the method further includes: Calculate the spatial overlap and semantic similarity of the specific target in adjacent video frames; Based on the similarity, candidate detection boxes of the specific target in different video frames are associated to obtain the initial trajectory corresponding to the specific target, and the ID of the initial trajectory corresponding to the specific target is determined.

7. The zero-sample multi-target tracking method of claim 1, wherein, The step of determining the enhancement level of the trajectory information corresponding to the candidate detection box based on the smoothness confidence and the environmental quality signal affecting the extraction of the candidate detection box includes: The environmental quality signal affecting the extraction of candidate detection boxes is normalized to obtain the normalized environmental quality signal; If the smooth confidence level is less than the first smooth confidence level and greater than or equal to the second smooth confidence level, or if the normalized environmental quality signal is less than the first environmental quality signal threshold and greater than or equal to the second environmental quality signal threshold, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be mild enhancement. If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be moderate enhancement. If the normalized environmental quality signal is less than the second environmental quality signal threshold, and the specific target corresponding to the candidate detection box is not in the same spatial neighborhood, then the enhancement level of the trajectory information corresponding to the candidate detection box is determined to be intensity enhancement.

8. A zero-shot multi-object tracking device, comprising: include: The data acquisition unit is used to acquire continuous video frame sequence data and semantic query data, wherein the semantic query data includes at least one image and text query data. The feature extraction unit is used to extract the spatiotemporal feature sequence of the continuous video frame sequence data, and to perform semantic space mapping on each of the image and text query data to obtain at least one semantic vector, and to determine at least one candidate detection box corresponding to each video frame based on the spatiotemporal feature sequence and the at least one semantic vector. A calculation unit is used to calculate the smooth confidence of a specific target in each candidate detection box in each video frame; A conditional triggering unit is configured to, in response to the smooth confidence level satisfying the triggering condition, determine the enhancement level of the trajectory information corresponding to the candidate detection box based on the smooth confidence level and the environmental quality signal affecting the extraction of the candidate detection box; The target tracking unit is used to determine the tracking area and enhancement method of the specific target based on the enhancement level of the trajectory information, and to track the specific target based on the tracking area and enhancement method.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images