Generative open vocabulary multi-object trajectory tracking method, system, benchmark construction and evaluation method

Abstract

This invention discloses a generative open-vocabulary multi-target trajectory tracking method, system, benchmark construction, and evaluation method. This method projects visual features onto a language model space to generate natural language descriptive words, reconstructing a generative recognition paradigm. It utilizes a self-consistency matrix to quantify inter-frame matching deviation, dynamically adjusting the consistency-aware focus loss accordingly to perform adaptive association, outputting trajectory data containing bounding boxes, identities, and descriptive words. This invention also involves the construction of a balanced benchmark based on a greedy algorithm and the introduction of a multi-granularity semantic evaluation system using a category hierarchy tree. These solutions address the challenges of vocabulary dependence in recognition tasks and unstable associations in complex scenarios, significantly improving the system's generalization ability and tracking robustness for unseen categories, providing a complete technical path and scientific evaluation criteria for open-vocabulary tracking tasks.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to target localization, open-vocabulary recognition, and multi-target trajectory association techniques in video sequences. More specifically, this invention relates to a generative open-vocabulary multi-target trajectory tracking method and system that reconstructs a traditional classification paradigm into a language generation paradigm, as well as related dedicated benchmark construction and multi-granularity semantic evaluation methods for quantitatively measuring generative descriptions. Background Technology

[0002] Multi-object tracking, a core task in computer vision, aims to simultaneously detect and continuously track multiple moving targets in a video sequence, assigning a unique identity to each target to maintain consistency across the entire timeline. In traditional technological evolution, multi-object tracking typically follows a detection-then-tracking paradigm. This involves first extracting target bounding boxes in each frame using a detector, and then using association modules such as Kalman filtering, the Hungarian algorithm, and re-identification feature matching to achieve cross-frame identity binding. However, with the continuous expansion of real-world applications, the closed-set category limitation of traditional solutions has become increasingly apparent. The models can only recognize a limited number of categories preset during training, making it difficult to cope with the ever-increasing number of new targets in the real world.

[0003] To break free from the constraints of closed sets, open-vocabulary multi-object tracking schemes have gradually become a research focus. These schemes, by incorporating a vision-language model, allow users to utilize custom textual vocabulary as category descriptions, thereby enabling inference about unseen categories. In the current typical workflow, the system embeds the detected bounding box regions into a visual encoder and performs similarity matching with a category embedding layer generated by a text encoder. Based on this approach, various improvement schemes have emerged in existing technologies. For example, OVMOT proposes a data illusion strategy driven by a denoising diffusion probability model to enhance robustness. SLAck attempts to fuse semantic, positional, and appearance cues to construct a spatiotemporal object graph for association learning. OVTR achieves alignment between objects and the base model through a dual-branch structure. Furthermore, VOVTrack proposes a self-supervised association learning method based on consistency constraints.

[0004] While the aforementioned technologies have made progress in open-vocabulary recognition, significant limitations remain in practical engineering applications. Current open-vocabulary solutions still heavily rely on predefined category lists, which often results in models lacking truly superior recognition capabilities when faced with completely unknown categories. Furthermore, existing mainstream datasets, such as TAO and LVIS, suffer from issues like extremely imbalanced category distribution, data sparsity, and inconsistent labeling granularity, making fair and accurate performance evaluation of the models impossible. Moreover, existing evaluation metrics struggle to provide effective quantitative measures for phenomena such as semantic mismatches commonly found in the output results.

[0005] In summary, existing multi-object tracking technologies face significant challenges in both technical architecture and evaluation systems. A unified end-to-end generative approach is still lacking in the three core dimensions of localization, cross-frame association, and semantic recognition. Consequently, the technical problems of existing technologies manifest as follows: the recognition process is constrained by predefined category lists, preventing true generative open-vocabulary recognition; a lack of large-scale, category-balanced dedicated benchmark datasets and multi-granular semantic evaluation metrics are required; and the failure to deeply integrate localization, recognition, and association tasks into a unified generative framework limits the overall tracking performance of the system in open-world scenarios. Summary of the Invention

[0006] This invention proposes a generative open-vocabulary multi-target trajectory tracking method, system, benchmark construction method, and evaluation method. To address the dependence of recognition tasks on predefined category lists, this invention first establishes a novel generative open-vocabulary multi-target tracking paradigm. This paradigm reconstructs traditional target recognition tasks from a classification logic based on limited categories to a generative language generation logic. Under this paradigm, the system translates visual features into natural language, outputting fine-grained category names, corresponding bounding boxes, and cross-frame identities for all targets in an image at once. The generative categories are presented in natural language form containing multiple semantic levels and do not rely on any externally pre-defined vocabulary, thus achieving a technological leap from closed-world recognition to open-world reasoning.

[0007] To support the training and performance measurement of generative recognition models, this invention relates to a method for constructing a large-scale and class-balanced benchmark using the GOVTrackB dataset. This method first performs class unification and compatibility processing on multiple original datasets, mapping heterogeneous annotations to a unified standard class space and constructing a class hierarchy tree. Then, a greedy algorithm is used for class-balanced extraction, ensuring a balanced distribution of video samples for each class. Simultaneously, this invention proposes a multi-granularity semantic evaluation system. By introducing multi-granularity semantic recognition metrics based on a noun network hierarchy and a comprehensive evaluation metric that considers localization, association, and classification accuracy, this system scientifically quantifies and matches the natural language descriptions output in the generative paradigm, solving the evaluation challenge arising from synonym and hypernym matching in generative output.

[0008] At the implementation level of the tracking system, this invention designs a unified framework integrating a detection head, a generative recognition head, and a cross-frame association module. This architecture utilizes a Transformer-based object detector as the localization head, responsible for generating candidate bounding boxes and extracting visual features. Subsequently, the generative recognition head maps the visual features to the input space of a pre-trained language model through linear projection, and the language model generates fine-grained attributes and category descriptors of the target. In the cross-frame association stage, this invention relates to a computational logic for consistency-aware focus loss. This logic quantifies the deviation between appearance consistency and semantic consistency between video frames from a global perspective by constructing a feature similarity matrix and a corresponding self-consistency matrix, and dynamically adjusts the hard sample factor in the focus loss accordingly, thereby guiding the model to adaptively learn more discriminative association features in complex matching environments.

[0009] Specifically, the first aspect of this invention provides a generative open-vocabulary multi-target trajectory tracking method. This method receives an input video sequence and extracts visual features of targets in each video frame. Candidate bounding boxes are generated using these visual features. The visual features are then linearly projected onto the input space of a pre-trained language model, driving the model to perform generation operations to produce natural language descriptive words independent of a preset category list. A self-consistency matrix is ​​constructed by combining the feature similarity matrix of targets between adjacent frames to quantify the consistency deviation between video frames. Finally, cross-frame identity association is performed based on this consistency deviation, and trajectory data containing bounding boxes, tracking identities, and descriptive words is output. This technical solution achieves a paradigm shift from closed-world recognition to open-world reasoning, significantly improving the model's generalization ability for recognizing unseen category targets and the stability of identity association under complex conditions.

[0010] Building upon this method, for the calculation of the feature similarity matrix, this invention first constructs the original feature similarity matrix by calculating the vector inner product between the visual features of the keyframe and the visual features of the reference frame. Then, it performs normalization processing on this matrix, converting the unbounded similarity scores into a matching probability distribution within a preset interval with a fixed sum. This approach eliminates dimensional differences through global regularization, ensuring the scale consistency and comparability of similarity assessments across different video sequences.

[0011] Furthermore, this invention obtains a self-consistent matrix reflecting the bidirectional matching verification relationship by performing matrix multiplication on the matching probability distribution and its transpose. The diagonal elements of this matrix are then extracted to determine its deviation from the identity matrix under ideal matching conditions. Finally, this deviation is processed using a nonlinear mapping function to obtain an appearance consistency factor characterizing the level of appearance consistency. This method can accurately quantify the logical self-consistency of cross-frame matching, providing a scientific feedback basis for identifying unreliable matches during the tracking process.

[0012] When performing cross-frame identity association, this invention determines a difficult sample factor that comprehensively reflects the matching difficulty based on an appearance consistency factor and a semantic consistency factor used to quantify semantic attribute offsets. This difficult sample factor is then used to dynamically adjust the association loss, adaptively increasing the contribution ratio of inconsistent matching samples during model training to perform cross-frame identity association. This dynamic adjustment mechanism enables the model to automatically focus on complex matching scenarios caused by occlusion or drastic appearance changes, significantly enhancing the system's robustness in extreme environments.

[0013] To further optimize the association performance, this invention injects a difficult sample factor as a regulation index into the consistency-aware focus loss function, and uses this consistency-aware focus loss and auxiliary regularization loss to construct a total loss function, thereby jointly optimizing the association module. Through multi-objective collaborative training, the feature extraction network maintains its discriminative power while possessing better convergence speed and adaptability to long-range tracking tasks.

[0014] Furthermore, this invention also involves mapping the generated natural language descriptors to a preset standard category space, and determining the final category of each target in the trajectory data by calculating the semantic similarity between the descriptors and candidate category nodes. This process breaks down the barrier between generative open semantics and structured standard categories, ensuring the practicality of the system output and its compatibility with existing industry standards.

[0015] A second aspect of this invention provides a multi-granularity semantic performance evaluation method for generative outputs. The system acquires the predicted trajectory and its corresponding natural language descriptors, calculates a positioning accuracy index (measuring location precision) and an association accuracy index (measuring identity consistency), and, based on a pre-defined category hierarchy tree, determines the hierarchical relationship between descriptors and real tags to calculate multi-granularity semantic recognition indicators. Finally, the system merges these indicators to generate a comprehensive evaluation index. This evaluation system solves the problem of quantifying generative descriptions at the semantic level, establishing a fair and scientific measurement criterion for open vocabulary tracking tasks.

[0016] Preferably, the comprehensive evaluation metric is the TRETA (Tracking and Recognition Accuracy Index), which considers multi-granularity semantic recognition. This metric is obtained by performing an arithmetic average of the positioning accuracy index, the association accuracy index, and the multi-granularity semantic recognition index. This balanced index aggregation method comprehensively considers the perception and cognitive capabilities of the tracking system, providing a multi-dimensional quantitative reference for the iterative optimization of the algorithm.

[0017] Furthermore, a third aspect of this invention provides a method for constructing a benchmark dataset. This method first performs cross-dataset category mapping to unify multi-source heterogeneous annotations into a standard category space and construct an initial hierarchical tree. Then, a greedy algorithm is used to iteratively extract data from the original video library to achieve a balanced ratio of base class to new class samples. Finally, a label system supporting multi-granularity semantic matching is constructed based on a noun network, serving as a source of training and validation samples to support the development of tracking methods. This scheme solves the problems of extremely imbalanced category distribution and semantic granularity conflicts in existing datasets, laying a high-quality data foundation for robust training of generative models.

[0018] Finally, a fourth aspect of the present invention provides a generative open-vocabulary multi-target trajectory tracking system. This system generates candidate bounding boxes and extracts visual features through a localization unit, performs linear projection of visual features onto a language space using a generative recognition unit, and drives a language model to generate natural language descriptive words. An adaptive association unit performs cross-frame identity association based on the global consistency level of appearance and semantics. Finally, an output unit summarizes and outputs complete trajectory data including bounding boxes, identity IDs, and descriptive words. Through deep integration of functional units, this system achieves a high-performance, end-to-end generative tracking hardware platform, significantly improving target processing efficiency in scenarios such as automated inspection and intelligent monitoring.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention achieves a fundamental technological leap from a closed to an open world by reconstructing the target recognition task from the traditional classification paradigm to a language generation paradigm. Through this generative recognition mechanism, the model can directly utilize the universal alignment between visual features and linguistic semantics to proactively generate natural language descriptions of targets, thus eliminating strong reliance on predefined category lists. This paradigm shift endows the model with extremely strong generalization ability and robustness towards any category of targets never seen during the training phase, accurately capturing fine-grained attribute information such as the target's color, material, and state. This rich semantic information, in turn, provides more discriminative features for target association in complex scenes, significantly reducing the identity switching problem caused by similar target appearances.

[0020] At the data support and evaluation system level, the dedicated benchmark constructed in this invention, through semantic unification and balanced selection across datasets, completely solves the problems of long-tailed category distribution and data source bias commonly found in existing benchmarks, providing a balanced and semantically rich material foundation for the training of generative models. The accompanying multi-granularity evaluation metrics, such as TRETA, introduce hierarchical semantic matching logic, achieving for the first time the scientific quantification of ambiguous referentials, synonyms, and hyponyms in generative outputs. This solves the technical problem of traditional metrics completely failing in open-vocabulary scenarios, establishing a reliable measurement standard for the technological evolution in this field.

[0021] Regarding model optimization and association performance, the consistency-aware focus loss function proposed in this invention upgrades model training from a traditional extensive mode to an adaptive intelligent mode by introducing a global consistency perspective. This mechanism dynamically calculates the hard sample factor using appearance self-consistency matrices and semantic self-consistency matrices, enabling the model to automatically adjust its learning focus based on the overall difficulty of inter-frame matching in the video, thereby significantly improving association stability in complex environments such as severe occlusion and drastic appearance changes. Experimental data shows that this invention achieves an association accuracy of 62.8% without any predefined vocabulary assistance, and its TETA metrics on both the base class and new class are significantly superior to existing mainstream technology combinations, demonstrating the excellent practical feasibility and technological advancement of this solution in handling complex real-world tracking tasks. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of a generative open vocabulary multi-target trajectory tracking method provided in an embodiment of the present invention. Detailed Implementation

[0023] To better understand the technical solutions of the present invention, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that those skilled in the art can implement the present invention in other forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0024] This invention provides a generative open-vocabulary multi-target trajectory tracking method, system, benchmark construction, and evaluation method. Its core concept lies in completely reconstructing the multi-target tracking task from the traditional closed-set classification or matching paradigm relying on predefined lists to a natural language generation paradigm based on visual features. In practical applications, since target categories are infinite and constantly evolving, this solution introduces the language understanding capabilities of a multimodal large-scale model, enabling the system to break free from dependence on fixed category labels and proactively generate fine-grained natural language descriptions directly based on the visual features of the targets. This processing logic not only achieves a leap from closed-world to open-world recognition but also provides more discriminative feature support for target association in complex dynamic environments through deep semantic information mining.

[0025] like Figure 1 As shown, the overall technical architecture provided by this embodiment of the invention presents an end-to-end processing flow that integrates localization, generative recognition, and cross-frame association. During execution, the system receives consecutive input video frames. First, the localization module integrated within the framework scans each frame to determine the physical location of potential targets and extract corresponding candidate bounding box visual features. Subsequently, these visual features are fed to the generative recognition module, where a language model generates natural language descriptive words independent of category lists based on the visual information, thereby endowing the target with rich semantic attributes. Simultaneously, the association module receives feature information between adjacent frames and uses a consistency-aware focus loss mechanism to learn and determine the consistency of the target's identity across frames. Finally, the system outputs tracking results including bounding boxes, tracking IDs, and detailed category descriptions. Combined with a specially constructed multi-granularity semantic evaluation system, the system achieves a quantitative evaluation of tracking performance through matching analysis between the generated text and the category hierarchy.

[0026] The technical system covered in this embodiment forms a logical closed loop, encompassing not only the online tracking algorithm but also the underlying methodologies supporting its training and validation. Specifically, before implementation, a dedicated benchmark dataset is constructed through cross-dataset category unification and equalization screening to ensure the model learns evenly distributed and semantically rich visual-linguistic alignment knowledge. During the inference phase, by adaptively calculating the global consistency of appearance and semantics, the model intelligently focuses on challenging samples caused by occlusion or violent movement, thereby significantly improving the stability of the target's identity during long-distance movement while ensuring recognition generalization. This design, which deeply integrates localization, cognitive generation, and spatiotemporal association into a single framework, provides a complete implementation path for solving the multi-target tracking challenge in open-vocabulary scenarios.

[0027] Example 1 This embodiment details the process of constructing the GOVTrackB dataset benchmark to support generative model training and evaluation. To address evaluation bias issues caused by sparse class distribution, highly imbalanced distribution, and inconsistent annotation granularity in existing datasets, this embodiment first performs class unification and compatibility processing. Addressing the common synonyms, polysemous words, and granularity differences found in the TAO and LV-VIS datasets, this embodiment employs a combination of manual review and automated tools to precisely map all original categories to the 1203 standard categories defined in the LVIS dataset. Based on this, a hierarchical class tree structure is constructed, providing a standardized semantic space for subsequent semantic expansion and generative recognition.

[0028] After class normalization, this embodiment optimizes the distribution structure of video samples by executing a class balancing extraction process based on a greedy algorithm. During the extraction process, the system strictly adheres to specific screening criteria, such as ensuring that each target class corresponds to at least two video sequences, while iteratively calculating to minimize the total number of selected videos and striving to achieve a balance between the distribution of base and new classes. Through this greedy strategy's iterative screening, a large-scale dataset benchmark containing 892 classes and a total of 1635 video sequences is finally constructed. Of these, 732 video sequences are reserved specifically for subsequent final testing of generative tracking performance, thus ensuring the representativeness and fairness of the evaluation results.

[0029] To meet the recognition requirements of the generative paradigm, this embodiment further constructs a semantically consistent multi-granularity labeling system to calculate the multi-granularity semantic recognition index mgReA. Specifically, this embodiment utilizes the noun network in the WordNet neural network to establish deep hierarchical relationships. Considering that WordNet nodes often have multiple paths leading to uncertainties in relationships, this embodiment employs a greedy algorithm to execute the parent node selection logic. That is, during the iteration process, it prioritizes selecting the parent node that maximizes the total number of child nodes, thereby organizing all categories into a clearly structured hierarchical system. Under this system, the evaluation mechanism is set so that as long as the predicted label generated by the model belongs to any relevant category in the aggregated multi-granularity category hierarchy, it is considered a successful recognition under extended semantics. This logic effectively solves the common problem of synonym and hypernym matching in generative output.

[0030] Example 2 This embodiment combines Figure 1 This document details the specific execution flow of a generative open-vocabulary multi-target trajectory tracking method. Taking a video sequence as input, this method uses end-to-end processing logic to collaboratively generate bounding boxes, cross-frame identities, and generative category names for each moving target in the video, independent of a predefined list of categories.

[0031] In the location and identification stage, such as Figure 1 As shown, the input video frame first enters the localization head, which is composed of a Transformer-based Deformable Detector (DETR). This module generates candidate target boxes and corresponding visual features by performing feature extraction and attention mechanism modeling on the video frame. To achieve open-vocabulary recognition capability, this embodiment maps the visual features output by the detector to the input space of a pre-trained language model via linear projection, and uses Flan-T5-base as the generative recognition head. In this process, the model does not perform discrete label classification, but actively generates several category descriptive words based on the mapped visual cues. These descriptive words are output in natural language form, which can cover the fine-grained semantics and attributes such as color and appearance of the target, such as generating descriptions like "black dog, animal, sheepdog". Subsequently, these generated texts are aligned with a large-scale vocabulary instance segmentation dataset (such as LVIS) through a semantic matching mechanism to determine the final category of the target.

[0032] In the cross-frame association stage, this embodiment achieves target identity binding by constructing a feature similarity matrix and introducing a consistency-aware focus loss. For example... Figure 1 As shown in the associated header input, the system first calculates the keyframe features F. key With reference frame feature F ref The eigensimilarity matrix S between them is expressed mathematically as follows: Among them, F key F represents the feature vectors of M objects in a keyframe. ref Let S represent the feature vectors of N objects in the reference frame. Each element in matrix S represents the original feature similarity between the keyframe object and the reference frame object. To ensure the similarity scores are globally interpretable, this embodiment performs Softmax regularization on matrix S, transforming it into a bounded normalized probability distribution. In the above formula, r and c represent the row and column indices of the matrix, respectively, and τ is a preset temperature parameter used to adjust the smoothness of the probability distribution. This regularization operation transforms the unbounded similarity scores into matching probabilities that contain global information.

[0033] To quantify the reliability of the tracking process, this embodiment further constructs a self-consistency matrix Y. This matrix is ​​used to evaluate the degree of mutual corroboration of video target matching, that is, to check whether the matching relationship from the keyframe to the reference frame can match the matching relationship from the reference frame back to the keyframe. Its calculation formula is as follows: Under ideal stable tracking conditions, matrix Y should approximate the identity matrix I. A greater deviation indicates a more complex tracking environment or a less reliable matching. Based on this physical meaning, this embodiment defines an appearance consistency factor γ. app This is used to measure the overall reliability of target appearance matching in a video: In the formula, diag(Y) and diag(I) represent the diagonal elements of the self-consistency matrix and the identity matrix, respectively, and the tanh function maps the deviation values ​​to a specific interval. When γ app A smaller value indicates a relatively simple and reliable tracking environment. Based on this, this embodiment further calculates the hard sample factor γ, which combines appearance consistency and semantic consistency γ. sem Dynamic impact: Here, β1 and β2 are parameters that adjust the consistency weights for different dimensions. This factor γ is injected as a core parameter into the consistency-aware focus loss L. c In this context, it is used to dynamically adjust the model's focus on difficult samples: In the loss function L c In this context, α is the balancing factor, and p is the model's predicted matching probability. A larger γ value significantly reduces the loss weight for simple samples, forcing the network to focus on learning features of difficult-to-match samples caused by occlusion, drastic changes in lighting, or alterations in appearance.

[0034] Finally, this embodiment performs joint optimization of the model using the total loss function L: Among them, L aux To assist L2 loss, additional regularization constraints are provided during training to stabilize feature extraction capabilities.

[0035] Through the complete process described above, from localization and generative recognition to global consistency-based perception association, this embodiment achieves accurate and robust trajectory tracking of multiple targets in open vocabulary scenarios.

[0036] Example 3 This embodiment provides a generative open-vocabulary multi-target trajectory tracking system. This system, as a hardware or logic carrier for executing the aforementioned tracking method, achieves the transformation from raw video signals to high-dimensional semantic trajectory results through the coordinated operation of multiple functional units. For example... Figure 1As shown, the system mainly consists of a video input interface, a positioning unit, a generative recognition unit, a cross-frame association unit, and a result output interface. The units interact with each other through a data bus or a high-speed signal link.

[0037] The localization unit, as the system's front-end perception module, integrates a deformable detector based on the Transformer architecture. Upon receiving a sequence of video frames from the video input interface, this unit performs parallel feature extraction and spatial attention computation to predict target bounding boxes and extract visual features of the corresponding image regions. The candidate boxes and their visual feature vectors output by this unit are transmitted to the generative recognition unit in real time. The generative recognition unit is internally configured with a linear feature projection layer and a pre-trained language model, such as Flan-T5-base. This unit maps the received visual features to the embedding space of the language model through the projection layer and drives the language model to perform decoding operations based on visual cues, thereby generating natural language descriptive words reflecting the target's attributes, category, and fine-grained features. Because this unit adopts a generative architecture, its operational logic is not limited to any pre-defined category list, enabling dynamic semantic description of any object in the video.

[0038] The cross-frame association unit is the core of the system's spatiotemporal synchronization, integrating a consistency-aware computation module. This unit receives multi-frame feature information from the localization unit and measures the spatial correlation of objects between frames by constructing a feature similarity matrix S. To achieve highly reliable identity binding, this unit further constructs a self-consistency matrix Y, quantifying the appearance consistency factor γ of the current matching environment in real time by calculating the deviation of the diagonal of the Y matrix from the identity matrix I. app Based on this factor, the association unit dynamically adjusts the hard sample factor γ in the focus loss, and uses this as a feedback signal to guide the feature extraction network to adaptively learn more discriminative association features under complex conditions such as occlusion and drastic changes in appearance. By fusing appearance consistency and semantic consistency calculations, the system ensures the stability of the association logic during long-term tracking and can effectively suppress identity switching phenomena.

[0039] The result output interface is connected to the cross-frame association unit and the generative recognition unit, and is used to aggregate the processing results of various functional modules. For example... Figure 1As shown, this interface maps the text information output by the generative recognition unit to a pre-defined large-scale vocabulary instance segmentation dataset space by performing semantic alignment operations, and integrates and encapsulates the final determined category description, bounding box coordinates, and unique tracking ID assigned by the association unit. Finally, the system outputs dynamic trajectory data with rich semantic information, which can be sent to the comprehensive evaluation module for performance verification in conjunction with the aforementioned multi-granularity semantic evaluation metrics. This modular system design not only achieves a high degree of integration of localization, recognition, and association tasks, but also endows the system with superior perception and cognitive capabilities in open-world environments through the coupling of generative paradigms and adaptive association mechanisms.

[0040] Example 4 This embodiment provides an evaluation method specifically designed for generative open-vocabulary multi-target tracking. Addressing the characteristics of synonymy, polysemy, and varying semantic granularity in the output natural language descriptions of generative paradigms, this embodiment constructs a multi-dimensional quantitative indicator system to comprehensively measure the model's localization accuracy, cross-frame association stability, and semantic understanding depth. In the basic performance evaluation dimension, this embodiment first uses the localization accuracy indicator LocA (Localization Accuracy) to measure the degree of spatial matching between the model's predicted bounding boxes and the ground truth bounding boxes. The formula for calculating this indicator is: Wherein, |TPL| represents the number of correctly extracted True Positive Localizations, i.e., elements whose Intersection over Union (IoU) with the ground truth bounding box reaches a preset threshold; |FPL| represents the number of incorrectly localized True Positive Localizations; and |FNL| represents the number of undetected True Positive Localizations, reflecting the system's missed detection rate. This metric physically isolates the influence of category attributes and is purely used to evaluate the model's basic localization capability.

[0041] In terms of cross-frame association stability, this embodiment introduces the association accuracy metric AssocA (Association Accuracy) based on correctly locating objects. Its mathematical expression is as follows: In the above formula, |TPA(b)| represents the number of samples with correctly associated cross-frame identities for a correctly located real identity; |FPA(b)| represents the number of misassociated identities, i.e., the number of identities assigned to other objects; and |FNA(b)| represents the number of identity loss due to no association with any predicted identity. AssocA reflects the model's ability to maintain target identity consistency over long time series by averaging the matching quality of all TPL samples.

[0042] In addition to quantifying cross-frame association stability, this embodiment also retains the traditional classification accuracy metric ClsA (Classification Accuracy) to measure the correctness of the model's prediction of the target class in a basic classification task. The formula for calculating this metric is: Wherein, |TPC| represents the number of samples whose predicted class matches the true result (True Positive Classification); |FPC| represents the number of samples whose predicted class does not match the true result (False Positive Classification); and |FNC| represents the number of samples where the true target class exists, but the model does not provide a predicted class or cannot match (False Negative Classification). This metric reflects the model's discrimination accuracy in a discrete label space in a physical sense and is a fundamental dimension for measuring recognition ability.

[0043] Based on the evaluation results of the above three dimensions of location, association, and classification, this embodiment defines the overall tracking capability evaluation index TETA (Tracking-Everything Accuracy), which is calculated as follows: This metric, by averaging the three dimensions, provides a comprehensive quantitative feedback on the overall tracking ability of the model, and provides a basis for judging the merits of different algorithms under the standard classification paradigm.

[0044] Furthermore, this embodiment focuses on the proposed multi-granularity semantic recognition metric mgReA for generative open-vocabulary tasks. Because the natural language descriptions output in generative paradigms exhibit synonymy, polysemy, and varying semantic granularity, the traditional ClsA metric struggles to handle fuzzy semantic matching. Therefore, this embodiment introduces mgReA to replace the traditional classification metric. The implementation of this metric relies on a hierarchical relationship constructed using a WordNet neural network. In practice, the system uses a greedy algorithm to iteratively select the parent node that maximizes the total number of child nodes among WordNet nodes with multiple paths, thereby organizing all involved categories into a clear hierarchical structure. Under this evaluation logic, as long as the natural language description generated by the model belongs to any relevant category in this aggregated multi-granularity category hierarchy, it is considered a successful recognition. This design physically solves the common problem in generative outputs where hypernym matching or synonym matching is misjudged.

[0045] Finally, this embodiment uses the comprehensive evaluation index TRETA (Tracking & Recognizing-Everything Accuracy) to jointly evaluate the overall performance of the generative system. This index equally weights positioning accuracy, association stability, and multi-granularity semantic recognition capability, and its formula is defined as: By calculating the TRETA metric, this embodiment can quantitatively measure the model's comprehensive capabilities in multiple aspects, including pointer expression comprehension, category recognition, and attribute generation. This three-pronged evaluation approach not only solves the problem of traditional evaluation metrics failing in generative tasks but also provides a scientific and objective measurement protocol for evaluating the model's generalization performance in open-world scenarios.

[0046] Example 5 It should be understood that the above embodiments are merely preferred embodiments of the present invention and do not constitute a limitation on the scope of protection of the present invention. In practical applications, those skilled in the art can make equivalent substitutions or improvements to the above solutions according to specific needs, and these design changes should all be included within the technical concept of the present invention. Regarding the selection of the generative language model, although the embodiments of the present invention use a Flan-T5 model with an encoding-decoding structure as the generative recognition head, in specific alternative solutions, the recognition module can be flexibly adjusted according to computing power costs and real-time requirements. For example, an autoregressive language model, a visual-language integrated multimodal large model with image-text co-encoding capabilities, or a lightweight Transformer language model optimized by pruning, or even an RNN or GRU model that performs stably in specific sequence modeling tasks, can be used. Its core technology lies in the ability to effectively map the visual features output by the localization module to the language model input space, and thereby generate a natural language description reflecting the target attributes. This change in model architecture does not change the basic logic of the recognition task of the present invention.

[0047] To address the diverse needs for generation formats, the noun phrases generated in this embodiment of the invention can be further expanded into richer expressions. In alternative solutions, the generated results can be set according to task complexity as a single core category word, a list of multiple candidate category labels, or a complete descriptive sentence containing the target state, actions, and environmental relationships. In subsequent processing, as long as these multi-form generated results are mapped to a preset category hierarchy using appropriate semantic matching algorithms or vector embedding alignment mechanisms to complete category determination, such variations in the length and form of the generated content constitute equivalent implementations of this invention.

[0048] To balance the efficiency of base class recognition with the generalization ability of unseen classes, the proposed solution can be integrated with traditional closed-set classification methods. Specific hybrid strategies include, but are not limited to, using a predefined closed-set classification head for rapid determination of high-frequency base classes in the training set, while only activating the generative recognition head of this invention for samples with low detection confidence or high recognition uncertainty. Furthermore, the system can also adopt a cascaded structure where a generative model first generates several semantic candidates, followed by a secondary selection by a small, refined classifier within the candidate set. This adaptive selection of recognition paths in this hybrid mode effectively improves the overall operational efficiency of the system in complex and variable environments.

[0049] Furthermore, the multi-granularity semantic evaluation system proposed in this invention also possesses high scalability. In the calculation of mgReA and TRETA metrics, the WordNet semantic resources used to construct hierarchical relationships can be replaced with a self-built industry-specific ontology knowledge base, or a dynamic category tree generated by automatic clustering from a pre-trained language model. In terms of evaluation dimensions, semantic matching weights for attributes such as color, material, and action can be further superimposed on the existing multi-granularity semantic recognition, thereby achieving finer-grained performance measurement. Simultaneously, the weight allocation of each sub-item in the TRETA metric can be dynamically adjusted according to different application focuses. For example, a higher weight can be assigned to association accuracy in security monitoring scenarios, or supplementary sub-metrics such as trajectory integrity and trajectory stability can be introduced. As long as its evaluation logic follows the basic idea of ​​jointly evaluating localization, association, and semantic recognition in the context of generative output, it is considered a reasonable improvement to the evaluation system of this invention.

[0050] The technical solutions of the present invention have been described in detail above with reference to specific embodiments. However, the above embodiments are only used to illustrate the technical concept of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or substitutions can be made to the technical solutions involved in the present invention without departing from the overall technical concept of the present invention. All equivalent substitutions, modifications, or improvements made to the present invention by those skilled in the art based on their understanding of the technical solutions of the present invention, as long as they do not depart from the technical concept and essence of the present invention, should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A generative open-vocabulary multi-object trajectory tracking method, characterized in that, The method includes: Receive the input video sequence and extract the visual features of the target in each video frame; Candidate bounding boxes are generated using the visual features, and the visual features are linearly projected onto the input space of a pre-trained language model; The pre-trained language model is driven to perform a generation operation based on the linearly projected visual features to generate natural language descriptive words that do not depend on a preset category list; Calculate the feature similarity matrix of targets between adjacent frames, and construct a self-consistency matrix based on the feature similarity matrix to quantify the degree of consistency deviation between video frames; Based on the degree of consistency deviation, cross-frame identity association is performed, and trajectory data containing the target bounding box, cross-frame tracking identity, and the natural language descriptive words is output.

2. The method according to claim 1, characterized in that, The calculation of the feature similarity matrix of targets between adjacent frames includes: The original feature similarity matrix is ​​constructed by calculating the vector inner product between the visual features of the keyframe and the visual features of the reference frame; The original feature similarity matrix is ​​normalized to convert the similarity scores into a matching probability distribution.

3. The method according to claim 2, characterized in that, The step of constructing a self-consistency matrix based on the feature similarity matrix to quantify the degree of consistency deviation between video frames includes: Perform matrix multiplication on the matching probability distribution and its transpose to obtain a self-consistent matrix that reflects the bidirectional matching confirmation relationship; Extract the diagonal elements of the self-consistent matrix and determine the deviation of the diagonal elements from the corresponding elements of the identity matrix under the ideal matching state; The deviation is processed by a nonlinear mapping function to obtain an appearance consistency factor that characterizes the appearance consistency level.

4. The method according to claim 3, characterized in that, The step of performing cross-frame identity association based on the degree of consistency deviation includes: Based on the appearance consistency factor and the semantic consistency factor used to quantify semantic attribute offset, a difficult sample factor that comprehensively reflects the matching difficulty is determined. The association loss is dynamically adjusted using the difficult sample factor to adaptively increase the contribution of inconsistent matching samples in model training, thereby performing the cross-frame identity association.

5. The method according to claim 4, characterized in that, The method of dynamically adjusting the association loss using the difficult sample factor includes: The difficult sample factor is injected as a moderating index into the consistency-perceived focus loss function; A total loss function is constructed using the consistency-aware focus loss and auxiliary regularization loss, and the association module that performs the cross-frame identity association is jointly optimized based on the total loss function.

6. The method according to claim 1, characterized in that, The generated natural language descriptors are mapped to a preset standard category space; The classification of each target in the trajectory data is determined by calculating the semantic similarity between the natural language descriptor and the candidate category node.

7. A method for evaluating the multi-granularity semantic performance of generative output, characterized in that, The evaluation method is used to evaluate the generative open-vocabulary multi-target trajectory tracking method of claim 1, and the evaluation method includes: Obtain the predicted trajectory data and its corresponding natural language descriptive terms; Calculate the positioning accuracy metric, which measures the accuracy of target location matching, and the association accuracy metric, which measures the consistency of identity binding across frames. Based on a preset category hierarchy tree, the membership relationship between the natural language descriptive words and the real labels on the hierarchical path is determined, and multi-granularity semantic recognition indicators are calculated. By integrating the positioning accuracy index, the association accuracy index, and the multi-granularity semantic recognition index, a comprehensive evaluation index for measuring the overall performance of the system is generated.

8. The method according to claim 7, characterized in that, The comprehensive evaluation index is a comprehensive tracking and recognition accuracy index that considers multi-granularity semantic recognition. The comprehensive evaluation index is obtained by performing an arithmetic average on the positioning accuracy index, the association accuracy index, and the multi-granularity semantic recognition index.

9. A method for constructing a generative open-vocabulary multi-objective tracking benchmark dataset, characterized in that, The construction method includes: Perform cross-dataset category mapping to unify the annotations from multiple original datasets into a standard category space and construct an initial hierarchical tree; A greedy algorithm is used to perform iterative extraction in the original video library to select target video sequences that achieve a preset balance ratio between the number of video samples in the baseline category and the new category. A labeling system supporting multi-granularity semantic matching is constructed based on a noun network to serve as a source of training and / or validation samples for training and / or validating the method described in claim 1.

10. A generative open-vocabulary multi-target trajectory tracking system, characterized in that, The system includes: The localization unit is used to generate candidate bounding boxes for each target and extract visual features; A generative recognition unit, connected to the localization unit, is used to perform linear projection of visual features into the language space and drive a pre-trained language model to generate natural language descriptive words for the target. An adaptive association unit, connected to both the localization unit and the generative recognition unit, is used to perform cross-frame identity association based on the global consistency level of appearance and semantics. The output unit is connected to the adaptive association unit and the generative recognition unit respectively, and is used to output trajectory data containing target bounding boxes, cross-frame tracking identities and natural language descriptive words.

Images