A method and system for cooperative tracking of multiple dynamic targets
By employing a teacher-student paradigm and hierarchical distillation loss, combined with a query responsibility separation mechanism, the problems of insufficient real-time performance and accuracy in multi-target tracking are solved, achieving efficient multi-target tracking in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-01-19
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies cannot simultaneously achieve both real-time performance and accuracy when tracking multiple targets, and are prone to tracking failures or insufficient real-time response, especially in complex scenarios.
An initial target tracking model is constructed using a teacher-student paradigm. Through hierarchical distillation loss and query responsibility separation mechanism, combined with knowledge transfer from the teacher model and updates to the lightweight model, collaborative tracking of multiple dynamic targets is achieved.
Without increasing computational costs, it improves the accuracy and real-time performance of multi-target tracking, effectively tracking existing targets and identifying new targets, thus achieving efficient multi-target tracking.
Smart Images
Figure CN122156248A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and specifically to a collaborative tracking method and system applicable to multiple dynamic targets. Background Technology
[0002] Object tracking is a fundamental task in the field of computer vision, while multi-object tracking is a basic application of object tracking in real-world scenarios. It aims to uniquely identify and continuously locate multiple objects in complex scenes with crowding, occlusion, and other challenging conditions.
[0003] To achieve effective multi-target tracking, existing technologies primarily optimize and update algorithms in two ways: one approach focuses on enhancing the feature representation and correlation capabilities of the tracking algorithm model, such as constructing a joint paradigm between detection and embedding to simultaneously learn detection and Re-ID features in a single network. The other approach attempts to optimize tracking capabilities by introducing more robust model architectures, such as incorporating an end-to-end Transformer architecture and utilizing a query mechanism to model the target's lifecycle.
[0004] While both methods offer certain technical advantages, they cannot simultaneously achieve both accuracy and real-time performance in multi-target tracking. Specifically, the first method, although improving target tracking efficiency, suffers from tracking failures when encountering targets with similar appearances or severe occlusion because it only extracts and identifies single features. The second method, while advantageous in tracking existing targets, requires a large and deep network architecture, making it difficult to ensure real-time tracking response. In particular, the second method cannot balance tracking known targets with detecting new targets, resulting in limitations in accurately and reliably tracking all targets. Summary of the Invention
[0005] Firstly, a cooperative tracking method suitable for multiple dynamic targets is provided, including: An initial target tracking model is constructed, and based on the teacher-student paradigm, the initial target tracking model is used as the student model to construct an associated teacher model; Video samples are simultaneously input into the teacher model and the initial target tracking model, and processed sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module, and ID Head module in both models; and the teacher visual features F of the feature layer are output sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in ES The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer. T Student ID Logits Z S ; The Transformer module of the initial target tracking model includes input queries for explicitly partitioned and concatenable trajectory queries and new target queries. The trajectory query is used to associate the target features of the current frame with the target features that already exist in the previous frame. The new target query is used to search for new targets in the full range of the current frame. Based on the teacher's visual characteristics F T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher ID Logits Z T Student IDLogits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation; The loss from the stratified distillation is: ;in, α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ is the coefficient for time-sequential distillation losses; where, ;in, For the initial target tracking model, the first i Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T The number of channels in the teacher model; ;in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function; ; A loss function is constructed based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model; The video clips are input into the final target tracking model in real time for multi-target real-time tracking.
[0006] Furthermore, the loss function is: ;in, L task To detect the loss.
[0007] Furthermore, the loss from the stratified distillation is: ;in, λ kd (t) represents the dynamic weights; where, ;in, λ max t is the upper limit of the weight of the loss in stratified distillation. current t represents the current global training steps. warmup The weights increase linearly from 0 to... λ max Total number of preheating steps.
[0008] Furthermore, the step of inputting video clips into the final target tracking model in real time for multi-target real-time tracking includes: The center point coordinates of existing targets in the previous frame are mapped to the current frame as the initial reference point for trajectory query. The feature embedding of existing targets in the previous frame is used as the initial content embedding for trajectory query. At the same time, a fixed set of new queries is introduced to capture new targets in the current frame. During forward propagation, trajectory queries and new queries are concatenated according to the channel dimension to form a complete query set; The complete query set is input into the decoder of the Transformer module.
[0009] Furthermore, the step of inputting video clips into the final target tracking model in real time for multi-target real-time tracking includes: The prediction head module based on the final target tracking model obtains the detection box and class confidence of the current frame as the prediction result of the current frame; The trajectory state of a new target is initialized, and the trajectory state of an existing target is updated when the similarity between the predicted result and the historical trajectory of the previous frame is less than the similarity threshold. The initial trajectory state of new targets in the current frame and the updated trajectory state of existing targets are passed to the next frame based on the feedback loop.
[0010] Secondly, a cooperative tracking system suitable for multiple dynamic targets is provided, including: The model building module is used to build an initial target tracking model and, based on the teacher-student paradigm, use the initial target tracking model as the student model to build an associated teacher model. The representation output module is used to simultaneously input video samples into the teacher model and the initial target tracking model, and process them sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module, and ID Head module in the two models; and output the teacher visual features F of the feature layer sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in E S The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer. T Student ID Logits Z S ; The Transformer module of the initial target tracking model includes input queries for explicitly partitioned and concatenable trajectory queries and new target queries. The trajectory query is used to associate the target features of the current frame with the target features that already exist in the previous frame. The new target query is used to search for new targets in the full range of the current frame. The loss acquisition module is used to obtain the loss based on the teacher's visual features F. T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher IDLogitsZ T Student ID Logits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation; The loss from the stratified distillation is: ;in, α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ is the coefficient for time-sequential distillation losses; where, ;in, For the initial target tracking model, the firsti Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T The number of channels in the teacher model; ;in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function; ; The model update module is used to construct a loss function based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model; The target tracking module is used to input video clips into the final target tracking model in real time for multi-target real-time tracking.
[0011] Furthermore, the target tracking module includes: The query unit is used to map the center point coordinates of existing targets in the previous frame to the current frame as the initial reference point for trajectory query, and to embed the features of existing targets in the previous frame as the initial content for trajectory query; at the same time, a fixed set of new queries is introduced to capture new targets in the current frame. The splicing unit is used to splice the trajectory query and the new query according to the channel dimension to form a complete query set during forward propagation; A decoding input unit is used to input the complete query set into the decoder of the Transformer module.
[0012] Furthermore, the target tracking module includes: The prediction unit is used to obtain the detection box and class confidence of the current frame based on the prediction head module of the final target tracking model, so as to serve as the prediction result of the current frame. The update unit is used to initialize the trajectory state of a new target, and at the same time, it updates the trajectory state of an existing target when the similarity between the predicted result and the historical trajectory of the previous frame is less than the similarity threshold. The transmission unit is used to transmit the initial trajectory state of the new target in the current frame and the updated trajectory state of the existing target to the next frame based on the feedback loop.
[0013] Thirdly, an electronic device is provided, including at least one processor coupled to a memory storing a computer program configured to be executed by the processor when run.
[0014] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the method.
[0015] Beneficial effects: As can be seen from the above technical solutions, the technical solution of the present invention provides a cooperative tracking method applicable to multiple dynamic targets to solve the technical defects of the prior art in that it is impossible to simultaneously take into account real-time performance and accuracy when performing multi-target tracking.
[0016] Overall, considering the insufficient accuracy of existing lightweight target tracking models in complex scenarios and the real-time limitations of large model architectures, this technical solution uses an initial target tracking model as a student model, and then introduces a teacher model based on a teacher-student paradigm. In this case, the lightweight initial target tracking model can acquire expert-level prior knowledge from the large teacher model through knowledge transfer, thus possessing sufficient knowledge for accurate target tracking without increasing computational costs.
[0017] Specifically, to ensure that the initial target tracking model, serving as the student model, can fully and accurately mimic and absorb the teacher model's target tracking experience in complex scenarios, this technical solution designs three parallel knowledge distillation paths corresponding to the entire thought chain from visual perception to decision logic and temporal reasoning, thereby constructing a hierarchical distillation loss. This ensures that the initial target tracking model can repeatedly internalize the teacher model's advanced reasoning patterns.
[0018] Furthermore, considering the impact of balancing existing target tracking and new target identification on recognition accuracy, a query responsibility separation mechanism was introduced into the Transformer module of the initial target tracking model. This allows for tracking of existing targets through trajectory queries and capture of new targets through novelty queries. By decoupling the resource contention between tracking and detection tasks, accurate tracking of various target types is ensured.
[0019] At this point, when tracking targets within a video segment based on the final target tracking model, accurate tracking of each target that appears can be performed with high real-time performance.
[0020] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.
[0021] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description
[0022] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the cooperative tracking method for multiple dynamic targets described in this embodiment; Figure 2 A flowchart for performing a target query; Figure 3 A flowchart for closed-loop tracking of various targets; Figure 4 This is a structural block diagram of the cooperative tracking system for multiple dynamic targets described in this embodiment; Figure 5 This is a structural block diagram of the electronic device described in this embodiment. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.
[0024] The terms "first," "second," and similar words used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0025] Multi-target tracking aims to continuously locate multiple targets in a video sequence while maintaining their unique identities. In existing technologies, to achieve more effective tracking of dynamic multi-targets in complex scenes, two main technical improvement paths exist. One path focuses on enhancing the feature representation and correlation capabilities of the tracking algorithm model, while the other focuses on introducing a more powerful model architecture to optimize tracking capabilities. However, both of these technical improvement paths have varying degrees of shortcomings in terms of real-time performance and accuracy in practical applications, making it difficult to comprehensively optimize multi-target tracking simultaneously. Therefore, this embodiment aims to provide a cooperative tracking method suitable for multiple dynamic targets to address the aforementioned technical deficiencies.
[0026] The following description, in conjunction with the accompanying drawings, details the cooperative tracking method applicable to multiple dynamic targets described in this embodiment.
[0027] The method described in this embodiment is based on a server equipped with an NVIDIA A10 GPU and an Intel Xeon CPU. Combined with... Figure 1 As shown, the specific steps include the following: Step S202: Construct an initial target tracking model, and based on the teacher-student paradigm, use the initial target tracking model as the student model to construct the associated teacher model.
[0028] In this embodiment, the teacher model is a parameter-frozen ResNet-101 (R-101) based model, and the initial target tracking model is a ResNet-50 (R-50) based model to be trained. In specific implementation, the teacher model adopts a Deformable DETR detector with R-101 as the backbone, and has been fully trained on the DanceTrack training set.
[0029] Step S204: Input the video samples into the teacher model and the initial target tracking model simultaneously, and process them sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module and ID Head module in the two models; and output the teacher visual features F of the feature layer sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in E S The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer. T Student ID Logits Z S .
[0030] In practical implementation, to optimize the internal attention mechanism of the initial target tracking model, a query responsibility separation mechanism is introduced into its Transformer module. Specifically, its input queries include an explicitly partitioned and concatenable set of trajectory queries and a set of new target queries. The trajectory queries are used to associate the target features of the current frame with the target features that already exist in the previous frame; thereby realizing the state update and trajectory maintenance of existing targets. The new target queries are used to search for new targets across the entire range of the current frame.
[0031] More specifically, the set of trajectory queries includes N track A learnable embedding, the set of new queries includes N... newborn There are 10 learnable embeddings; the total number of them is the total number of queries.
[0032] Step S206: Based on the teacher's visual features F T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher ID LogitsZ T Student ID Logits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation.
[0033] In this embodiment, the loss from the stratified distillation is: ; in,α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ These are the coefficients for the time-series distillation loss; they are used to balance the fixed hyperparameters of the characteristic distillation loss, identity distillation loss, and time-series distillation loss. In practical applications, the coefficients for the characteristic distillation loss and the time-series distillation loss are set to 1.0, 0.5, and 0.5, respectively.
[0034] The characteristic distillation loss is: ; in, For the initial target tracking model, the first i Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T This represents the number of channels in the teacher model.
[0035] At this point, the feature distillation loss can be used to regularize the visual feature space of the student model using the prior knowledge of the teacher model, so that the student model can improve the stability and reliability of the extracted visual features when facing complex scenes such as severe occlusion and drastic changes in lighting.
[0036] Identity distillation loss is: ; in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function.
[0037] At this point, the student model can learn from the rich inter-category relationship knowledge within the teacher model based on identity distillation loss. When faced with similar targets, it can obtain a more refined decision-making logic with confidence, thereby achieving accurate discrimination of targets with similar appearances.
[0038] The time-series distillation loss is: .
[0039] The time-series distillation loss specifically employs cosine similarity loss, which allows for reasonable prediction of the location of the tracked target even after it briefly disappears, thus maintaining the uniqueness of its identity and continuous tracking of its trajectory when it reappears.
[0040] In summary, based on the aforementioned hierarchical distillation loss, knowledge distillation can be performed simultaneously on each link within the entire thought chain, ensuring a more comprehensive knowledge transfer from the teacher model to the initial target tracking model.
[0041] In practical implementation, the loss function is expressed as: ; in, L task To detect loss, it is used to enable the initial target tracking model to master the basic tracking task.
[0042] Considering the initial shortcomings of the target tracking model in basic task understanding during the early training stages, detection loss should be the primary method, with distillation loss as a secondary method. As training iterates and the initial target tracking model's task understanding gradually improves, the weight of distillation loss should be gradually increased to allow it to learn sufficient prior knowledge. Therefore, this embodiment introduces a linearly increasing dynamic weight into the hierarchical distillation loss. At this point, the hierarchical distillation loss is updated as follows: ; in, λ kd (t) represents the dynamic weight.
[0043] Specifically, ; in, λ max t is the upper limit of the weight of the loss in stratified distillation. current t represents the current global training steps. warmup The weights increase linearly from 0 to... λ max The total number of warm-up steps. In practice, the weight cap is set to 1.0, and the number of training rounds in the warm-up phase is set to 3.
[0044] Step S208: Construct a loss function based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model.
[0045] In the specific training process, all models were trained end-to-end using the AdamW optimizer. The initial learning rate was set to 1e-4, and the training lasted for 50 epochs. The learning rate was reduced to 1e-5 in the 40th epoch.
[0046] Step S210: Input the video clips into the final target tracking model in real time to perform multi-target real-time tracking.
[0047] In specific multi-target tracking, in order to reduce the search space of the model, it is necessary to combine... Figure 2As shown, it also includes: Step S21002: Map the center point coordinates of the existing target in the previous frame to the current frame as the initial reference point for trajectory query, and embed the features of the existing target in the previous frame as the initial content embedding for trajectory query; at the same time, introduce a fixed set of new queries to capture new targets in the current frame.
[0048] Step S21004: During forward propagation, the trajectory query and the new query are concatenated according to the channel dimension to form a complete query set.
[0049] Step S21006: Input the complete query set into the decoder of the Transformer module.
[0050] At this point, by combining steps S21002 to S21006, the trajectory of the jump confirmed in the previous frame can be used as prior information. Then, the initialization strategy based on motion prior can effectively reduce the search space, thereby efficiently realizing the tracking of existing targets and the confirmation of new targets.
[0051] As a specific implementation method, combined with Figure 3 As shown, closed-loop tracking of multiple targets is performed in the following manner: Step S21022: The prediction head module based on the final target tracking model obtains the detection box and class confidence of the current frame as the prediction result of the current frame.
[0052] Step S21024: Initialize the trajectory state of the new target, and at the same time, if the similarity between the prediction result and the historical trajectory of the previous frame is less than the similarity threshold, update the trajectory state of the existing target.
[0053] Step S21026: The initial trajectory state of the new target in the current frame and the updated trajectory state of the existing target are passed to the next frame based on the feedback loop.
[0054] In summary, the method described in this embodiment employs a teacher-student paradigm to build the model for training. This ensures tracking accuracy without incurring significant computational overhead during subsequent actual tracking, thereby guaranteeing real-time tracking. Furthermore, a hierarchical distillation technique corresponding to the full reasoning process is introduced to ensure that the initial target tracking model, serving as the student model, possesses sufficient expert knowledge. Moreover, a query responsibility separation mechanism is introduced to simultaneously achieve effective tracking of existing targets and the search for new targets, thus enabling accurate tracking of various types of targets.
[0055] The aforementioned program can run in a processor or be stored in memory (or a computer-readable storage medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.
[0056] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.
[0057] This embodiment also provides a cooperative tracking system suitable for multiple dynamic targets. Combined with... Figure 4 As shown, the system includes the following functional modules: The model building module is used to build an initial target tracking model and, based on the teacher-student paradigm, use the initial target tracking model as the student model to build an associated teacher model.
[0058] The representation output module is used to simultaneously input video samples into the teacher model and the initial target tracking model, and process them sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module, and ID Head module in the two models; and output the teacher visual features F of the feature layer sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in E S The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer.T Student ID Logits Z S .
[0059] The Transformer module of the initial target tracking model includes input queries for explicitly partitioned and concatenable trajectory queries and new target queries. The trajectory query is used to associate the target features of the current frame with the target features that already exist in the previous frame. The new target query is used to search for new targets in the full range of the current frame.
[0060] The loss acquisition module is used to obtain the loss based on the teacher's visual features F. T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher IDLogitsZ T Student ID Logits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation.
[0061] The loss from the stratified distillation is: ; in, α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ This is the coefficient for time-series distillation losses.
[0062] ;in, For the initial target tracking model, the first i Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T This represents the number of channels in the teacher model.
[0063] ;in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function.
[0064] .
[0065] The model update module is used to construct a loss function based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model.
[0066] The target tracking module is used to input video clips into the final target tracking model in real time for multi-target real-time tracking.
[0067] Since the system is built based on the method described above, the points already explained will not be repeated here.
[0068] For example, the target tracking module includes: The query unit is used to map the center point coordinates of existing targets in the previous frame to the current frame as the initial reference point for trajectory query, and to embed the features of existing targets in the previous frame as the initial content for trajectory query; at the same time, a fixed set of new queries is introduced to capture new targets in the current frame.
[0069] The splicing unit is used to splice the trajectory query and the new query according to the channel dimension to form a complete query set during forward propagation.
[0070] A decoding input unit is used to input the complete query set into the decoder of the Transformer module.
[0071] For example, the target tracking module includes: The prediction unit is used to obtain the detection box and class confidence of the current frame based on the prediction head module of the final target tracking model, so as to serve as the prediction result of the current frame.
[0072] The update unit is used to initialize the trajectory state of a new target, and at the same time, it updates the trajectory state of an existing target when the similarity between the predicted result and the historical trajectory of the previous frame is less than the similarity threshold.
[0073] The transmission unit is used to transmit the initial trajectory state of the new target in the current frame and the updated trajectory state of the existing target to the next frame based on the feedback loop.
[0074] Additionally, this embodiment also provides an electronic device. Combined with... Figure 5 As shown, it includes at least one processor coupled to a memory storing a computer program configured to be executed by the processor when run.
[0075] This embodiment also provides a computer-readable storage medium. A computer program is stored thereon, the computer program being executed by a processor to implement the method described thereon.
[0076] Since the system, the electronic device, and the storage medium are all used to implement the method or to build upon the method, the real-time performance and accuracy of the tracking can be simultaneously considered when tracking multiple dynamic targets based on them.
[0077] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. A cooperative tracking method suitable for multiple dynamic targets, characterized in that, include: An initial target tracking model is constructed, and based on the teacher-student paradigm, the initial target tracking model is used as the student model to construct an associated teacher model; Video samples are simultaneously input into the teacher model and the initial target tracking model, and processed sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module, and ID Head module in both models; and the teacher visual features F of the feature layer are output sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in E S The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer. T Student ID Logits Z S ; The Transformer module of the initial target tracking model includes input queries for explicitly partitioned and concatenable trajectory queries and new target queries. The trajectory query is used to associate the target features of the current frame with the target features that already exist in the previous frame. The new target query is used to search for new targets in the full range of the current frame. Based on the teacher's visual characteristics F T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher ID Logits Z T Student IDLogits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation; The loss from the stratified distillation is: ;in, α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ is the coefficient for time-sequential distillation losses; where, ;in, For the initial target tracking model, the first i Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T The number of channels in the teacher model; ;in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function; ; A loss function is constructed based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model; The video clips are input into the final target tracking model in real time for multi-target real-time tracking.
2. The cooperative tracking method for multiple dynamic targets according to claim 1, characterized in that, The loss function is: ;in, L task To detect the loss.
3. The cooperative tracking method for multiple dynamic targets according to claim 2, characterized in that, The loss from the stratified distillation is: ;in, λ kd (t) represents the dynamic weights; where, ;in, λ max t is the upper limit of the weight of the loss in stratified distillation. current t represents the current global training steps. warmup The weights increase linearly from 0 to λ max Total number of preheating steps.
4. The cooperative tracking method for multiple dynamic targets according to claim 1, characterized in that, The step of inputting video clips into the final target tracking model in real time for multi-target real-time tracking includes: The center point coordinates of existing targets in the previous frame are mapped to the current frame as the initial reference point for trajectory query. The feature embedding of existing targets in the previous frame is used as the initial content embedding for trajectory query. At the same time, a fixed set of new queries is introduced to capture new targets in the current frame. During forward propagation, trajectory queries and new queries are concatenated according to the channel dimension to form a complete query set; The complete query set is input into the decoder of the Transformer module.
5. The cooperative tracking method for multiple dynamic targets according to claim 1, characterized in that, The step of inputting video clips into the final target tracking model in real time for multi-target real-time tracking includes: The prediction head module based on the final target tracking model obtains the detection box and class confidence of the current frame as the prediction result of the current frame; The trajectory state of a new target is initialized, and the trajectory state of an existing target is updated when the similarity between the predicted result and the historical trajectory of the previous frame is less than the similarity threshold. The initial trajectory state of new targets in the current frame and the updated trajectory state of existing targets are passed to the next frame based on the feedback loop.
6. A cooperative tracking system suitable for multiple dynamic targets, characterized in that, include: The model building module is used to build an initial target tracking model and, based on the teacher-student paradigm, use the initial target tracking model as the student model to build an associated teacher model. The representation output module is used to simultaneously input video samples into the teacher model and the initial target tracking model, and process them sequentially through the corresponding backbone network module, Transformer module, trajectory modeling module, and ID Head module in the two models; and output the teacher visual features F of the feature layer sequentially through the corresponding backbone network module. T Student visual characteristics F S The corresponding trajectory modeling module outputs the time-series teacher trajectory embedding E. T Student trajectory embedded in E S The corresponding ID Head module outputs the teacher ID Logits Z for the identity layer. T Student ID Logits Z S ; The Transformer module of the initial target tracking model includes input queries for explicitly partitioned and concatenable trajectory queries and new target queries. The trajectory query is used to associate the target features of the current frame with the target features that already exist in the previous frame. The new target query is used to search for new targets in the full range of the current frame. The loss acquisition module is used to obtain the loss based on the teacher's visual features F. T With the student's visual characteristics F S Constructing characteristic distillation loss L feat Based on teacher trajectory embedding E T Student trajectory embedded in E S Constructing time-series distillation loss L temporal Based on teacher IDLogitsZ T Student ID Logits Z S Constructing Identity Distillation Loss L id This leads to losses from stratified distillation; The loss from the stratified distillation is: ;in, α The coefficient representing the characteristic distillation loss. β The coefficient for identity distillation loss. γ is the coefficient for time-sequential distillation losses; where, ;in, For the initial target tracking model, the first i Feature maps at each scale layer; This is the feature map at the i-th scale layer in the teacher model; O For the set of levels to be distilled, ϕ i It is a 1x1 convolutional layer. C S The number of channels in the initial target tracking model. C T The number of channels in the teacher model; ;in, D KL Let KL divergence be τ, and τ be the distillation temperature hyperparameter. σ () represents the Softmax function; ; The model update module is used to construct a loss function based on the hierarchical distillation loss to iteratively update the initial target tracking model, thereby obtaining the final target tracking model; The target tracking module is used to input video clips into the final target tracking model in real time for multi-target real-time tracking.
7. The cooperative tracking system for multiple dynamic targets according to claim 6, characterized in that, The target tracking module includes: The query unit is used to map the center point coordinates of existing targets in the previous frame to the current frame as the initial reference point for trajectory query, and to embed the features of existing targets in the previous frame as the initial content for trajectory query; at the same time, a fixed set of new queries is introduced to capture new targets in the current frame. The splicing unit is used to splice the trajectory query and the new query according to the channel dimension to form a complete query set during forward propagation; A decoding input unit is used to input the complete query set into the decoder of the Transformer module.
8. The cooperative tracking system for multiple dynamic targets according to claim 6, characterized in that, The target tracking module includes: The prediction unit is used to obtain the detection box and class confidence of the current frame based on the prediction head module of the final target tracking model, so as to serve as the prediction result of the current frame. The update unit is used to initialize the trajectory state of a new target, and at the same time, it updates the trajectory state of an existing target when the similarity between the predicted result and the historical trajectory of the previous frame is less than the similarity threshold. The transmission unit is used to transmit the initial trajectory state of the new target in the current frame and the updated trajectory state of the existing target to the next frame based on the feedback loop.
9. An electronic device, characterized in that, It includes at least one processor coupled to a memory storing a computer program configured to be executed by the processor to perform the method of any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which is used to be executed by a processor to implement the method of any one of claims 1-5.