A method, apparatus, device, and medium for multi-target tracking based on Transformer and trajectory query group

Abstract

This invention provides a multi-target tracking method, apparatus, device, and medium based on Transformer and trajectory query group, relating to the field of computer vision technology. It includes: inputting the current image frame into a multi-target tracking model to be trained; processing the current image frame through a backbone network and a Transformer encoder to obtain current image encoding features; if a target is detected in the previous image frame, inputting the current image encoding features, detection query features, and trajectory query group into a Transformer decoder to obtain the output embedding result corresponding to the target; determining the sample multi-target tracking result based on the output embedding result corresponding to the target; and training the multi-target tracking model based on the sample multi-target tracking result and the multi-target tracking result label to obtain a trained multi-target tracking model for use in multi-target tracking, thereby improving the tracking accuracy of multi-target tracking in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-target tracking method, apparatus, device, and medium based on Transformer and trajectory query group. Background Technology

[0002] In recent years, deep learning-based algorithms have been widely applied to solve many real-world problems, especially in the field of computer vision. Tasks such as image classification, object detection, and object tracking now rely on deep learning algorithms, and related technologies continue to iterate rapidly. As a downstream task of object detection, object tracking is a fundamental but important computer vision task. Its core objective is to temporally correlate and match detected objects. Object tracking techniques are often used in conjunction with object detection techniques, not only to integrate the motion trajectories of objects and predict their motion trends, but also to effectively filter out false detections through trajectory matching.

[0003] Multi-object tracking (MOT), as a downstream task of object detection, aims to detect multiple targets of interest in a scene, assign them unique IDs, and maintain their identities over time within the video stream. MOT technology has wide applications in security monitoring, intelligent transportation, military applications, and autonomous driving. However, despite the rapid development of MOT algorithms, it still faces many challenges in practical applications, affecting tracking accuracy. For example, there is the problem of target occlusion: most MOT methods rely on image sequences acquired by sensorless cameras, and in crowded scenes, multiple targets occluding each other can significantly interfere with detection and tracking. Furthermore, the appearance and disappearance of targets in the scene, frequent ID switching between targets, and the uncertainty of target trajectories also bring considerable difficulties to trajectory processing. Solving these problems has become a key direction for the development of MOT technology.

[0004] With DETR (an end-to-end object detector based on Transformer) bringing the Transformer architecture to the field of computer vision, Transformer-based algorithms have provided a truly end-to-end solution for multi-object tracking tasks. In current Transformer-based tracking methods, a Track Query is typically associated with a tracked object. However, in crowded scenes, the target's movement frequently changes, causing the Track Query to fail to update its features in a timely manner, resulting in target loss. Therefore, relying solely on a single Track Query for tracking in complex scenes is insufficient and cannot meet the challenges posed by feature transformations. Thus, improving the tracking accuracy of multi-object tracking in complex scenes is a key technical problem that this invention aims to solve. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a multi-target tracking method, apparatus, device, and medium based on Transformer and TrackQuery Groups, aiming to improve the tracking accuracy of multi-target tracking in complex scenarios and avoid target loss due to the inability of a single TrackQuery to adapt to feature changes.

[0006] The first aspect of this invention provides a multi-target tracking method based on Transformer and trajectory query groups, the method comprising:

[0007] The current image frame in the sample video stream is input into the multi-object tracking model to be trained. The multi-object tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder.

[0008] The current image frame is processed by the backbone network and the Transformer encoder to obtain the current image coding features;

[0009] If a target is detected in the previous image frame of the current image frame, the current image encoding features, detection query features, and trajectory query groups corresponding to the tracked targets in the current image frame are input into the Transformer decoder to obtain one or more prediction embedding results corresponding to multiple targets respectively; each tracked target corresponds to a trajectory query group, and the trajectory query group includes: multiple trajectory query features representing different occlusion levels;

[0010] Determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets;

[0011] Based on the output embedding results corresponding to multiple targets, the sample multi-target tracking result corresponding to the current image frame is determined;

[0012] Based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, the multi-target tracking model to be trained is trained to obtain the trained multi-target tracking model.

[0013] The video stream to be detected is input into the trained multi-object tracking model to obtain the multi-object tracking result corresponding to the video stream to be detected.

[0014] A second aspect of the present invention provides a multi-target tracking device based on Transformer and trajectory query group, the device comprising:

[0015] An image input module is used to input the current image frame from the sample video stream into the multi-target tracking model to be trained. The multi-target tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder.

[0016] The image encoding module is used to process the current image frame through the backbone network and the Transformer encoder to obtain the current image encoding features;

[0017] The embedding prediction module is used to input the current image encoding features, detection query features, and trajectory query groups corresponding to the tracked targets in the current image frame into the Transformer decoder when a target is detected in the previous image frame of the current image frame, to obtain one or more prediction embedding results corresponding to multiple targets respectively; each tracked target corresponds to a trajectory query group, and the trajectory query group includes: multiple trajectory query features representing different occlusion levels;

[0018] The embedding output module is used to determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets respectively;

[0019] The tracking and prediction module is used to determine the multi-target tracking result of the sample corresponding to the current image frame based on the output embedding results corresponding to multiple targets respectively.

[0020] The model training module is used to train the multi-target tracking model to be trained based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, so as to obtain the trained multi-target tracking model.

[0021] The tracking and detection module is used to input the video stream to be detected into the trained multi-target tracking model to obtain the multi-target tracking result corresponding to the video stream to be detected.

[0022] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the multi-target tracking method based on Transformer and trajectory query group as described in the first aspect of the present invention.

[0023] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-target tracking method based on Transformer and trajectory query group of the first aspect of the present invention.

[0024] In the multi-target tracking method based on Transformer and trajectory query group provided by the present invention, each image frame in the video stream is processed sequentially by Transformer encoder and Transformer decoder in order to track and detect multiple targets in the image frame. In the Transformer decoder, multiple targets are associated and detected through Track Query Groups (TGs) and Detect Query features. Tracking is performed on targets using Track Query Groups as the tracking unit. A Track Query Group is designed for each tracked target, and each Track Query includes multiple Track Query features representing different occlusion levels. Each Track Query feature is responsible for identifying the target's state under a specific occlusion level. This allows the Track Query Group to identify the same tracked target under different occlusion levels, enabling each Track Query Group to handle the association of the same target under different occlusion levels. This enhances the categorical representation of targets under different occlusion levels in complex scenes. This invention addresses the issues of target occlusion and state changes by proposing a mechanism that dynamically adjusts target tracking and designs a categorical representation method that adapts to different target occlusion levels. This adapts to the frequent state transitions of targets in complex scenes, promptly updating the target's content features and enhancing the perception of target states in complex scenes. This improves the tracking accuracy of multi-target tracking in complex scenes and avoids the limitations imposed by a single Track Query Group. The query fails to adapt to changes in features, resulting in the loss of the target. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the steps of a multi-target tracking method based on Transformer and trajectory query group according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of a trajectory query group according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of a trajectory query group updater according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of a position predictor according to an embodiment of the present invention;

[0030] Figure 5 This is a flowchart illustrating a multi-target tracking model based on trajectory query groups according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram illustrating the application of a multi-target tracking task in street scene crowds, according to an embodiment of the present invention.

[0032] Figure 7 This is a flowchart illustrating a model training process according to an embodiment of the present invention;

[0033] Figure 8 This is a flowchart illustrating a model reasoning process according to an embodiment of the present invention;

[0034] Figure 9 This is a schematic diagram illustrating a street view crowd tracking visualization according to an embodiment of the present invention;

[0035] Figure 10 This is a schematic diagram illustrating the application of a multi-target tracking task in a dance scene according to an embodiment of the present invention;

[0036] Figure 11 This is a schematic diagram illustrating the tracking and visualization of a dance scene according to an embodiment of the present invention;

[0037] Figure 12 This is a structural block diagram of a multi-target tracking device based on Transformer and trajectory query group provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multi-target tracking method based on Transformer and trajectory query groups, as shown in an embodiment of the present invention. Figure 1 As shown, the multi-target tracking method based on Transformer and trajectory query group provided in this embodiment includes at least the following steps:

[0040] Step S11: Input the current image frame from the sample video stream into the multi-target tracking model to be trained. The multi-target tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder.

[0041] This embodiment provides a sample dataset for training a multi-object tracking model. This dataset includes multiple sample video streams, each with a label representing the multi-object tracking result for each image frame. The sample video streams are used for training the multi-object tracking model.

[0042] In this embodiment, the current image frame in the sample video stream can be input into the multi-target tracking model to be trained for tracking and detection processing, so that each image frame in the sample video stream can be input into the multi-target tracking model to be trained.

[0043] Step S12: Process the current image frame through the backbone network and the Transformer encoder to obtain the current image coding features.

[0044] In this embodiment, after the current image frame is input into the multi-object tracking model to be trained, the current image frame is processed by the backbone network and the Transformer encoder to extract two-dimensional features, obtaining the current image encoded features, which are used as one of the inputs to the Transformer decoder. In a specific example, the current image frame can be first input into the backbone network for feature extraction to obtain initial image features, and then the initial image features can be input into the Transformer encoder to obtain the current image encoded features output by the Transformer encoder.

[0045] Step S13: If a target is detected in the previous image frame of the current image frame, the current image encoding features, detection query features, and trajectory query group corresponding to the tracked target in the current image frame are input into the Transformer decoder to obtain one or more prediction embedding results corresponding to multiple targets respectively.

[0046] In this embodiment, if a target is detected in the previous image frame of the current image frame, that is, if there is a tracked target in the current image frame, the current image encoding features, detection query features, and trajectory query group corresponding to the tracked target in the current image frame can be input into the Transformer decoder to obtain one or more prediction embedding results corresponding to multiple targets output by the Transformer decoder.

[0047] In this embodiment, a fixed number (which can be freely set) of detection query features are input to the Transformer decoder in each image frame to detect newly appearing targets (i.e., new targets) in the current image frame. It can be understood that the detection query features are learnable embeddings.

[0048] In traditional methods, each tracked target is associated with only one track query feature, which is effective when the target's occlusion level is stable. However, in complex scenarios involving different occlusion levels, this method often fails because a single track query cannot handle the challenge, leading to numerous ID switching. Therefore, in this embodiment, each tracked target corresponds to a track query group, and each track query group includes multiple track queries representing different occlusion levels. Furthermore, within each track query group, multiple track queries jointly predict the same target.

[0049] In this embodiment, the Transformer decoder can output corresponding prediction embeddings for multiple detected targets. Each target can correspond to one prediction embedding or multiple prediction embeddings.

[0050] Step S14: Determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets respectively.

[0051] In this embodiment, after obtaining one or more prediction embeddings corresponding to multiple targets, an output embedding for each target can be determined from these prediction embeddings. Specifically, when a target corresponds to multiple prediction embeddings, the most suitable prediction embedding representing the target's current state can be selected as the target's output embedding. When a target corresponds to only one prediction embedding, that prediction embedding is directly used as the target's output embedding.

[0052] Step S15: Based on the output embedding results corresponding to multiple targets, determine the sample multi-target tracking result corresponding to the current image frame.

[0053] In this embodiment, the sample multi-object tracking result corresponding to the current image frame can be determined based on the output embedding results corresponding to multiple targets. The sample multi-object tracking result is the multi-object tracking result output by the multi-object tracking model during training, and the multi-object tracking result includes at least the confidence scores and bounding boxes corresponding to multiple targets. In a specific example, the trajectory query features in the trajectory query group are input into the Transformer decoder to obtain the embedding, and then processed by MLP to predict the corresponding confidence scores and bounding boxes.

[0054] Step S16: Based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, train the multi-target tracking model to be trained to obtain the trained multi-target tracking model.

[0055] In this embodiment, each image frame in the sample video stream can be processed according to the above steps to obtain the sample multi-target tracking result corresponding to each image frame in the sample video stream. Then, the multi-target tracking model to be trained is trained based on the sample multi-target tracking result and the multi-target tracking result label corresponding to each image frame in the sample video stream. This updates the network parameters of the backbone network, the Transformer encoder, and the Transformer decoder to be trained, thereby obtaining the trained multi-target tracking model.

[0056] Step S17: Input the video stream to be detected into the trained multi-object tracking model to obtain the multi-object tracking result corresponding to the video stream to be detected.

[0057] In this embodiment, the trained multi-object tracking model is used for multi-object tracking of a single image or video stream. During the application of the trained multi-object tracking model, in the case of multi-object tracking of a video stream, the video stream to be detected can be input into the trained multi-object tracking model to obtain the multi-object tracking result corresponding to the video stream to be detected, output by the trained multi-object tracking model. This multi-object tracking result includes the multi-object tracking result corresponding to each image frame in the video stream to be detected. The video stream to be detected is the video stream for which multi-object tracking is required.

[0058] In this embodiment, multiple targets are associated and detected through trajectory query groups and detection query features. Using trajectory query groups as the tracking unit, a trajectory query group is designed for each tracked target. Each trajectory query includes multiple trajectory query features representing different occlusion levels; that is, different trajectory query features are designed for targets under different occlusion levels. Each trajectory query feature is responsible for identifying the target's state under a specific occlusion level, thus identifying the same tracked target under different occlusion levels through trajectory query groups. This allows the Track Query contained in each trajectory query group to handle the association problem of the same target under different occlusion levels, thereby increasing the categorical representation of targets under different occlusion levels in complex scenes. Compared with the traditional method of tracking using a single Track Query, this embodiment uses trajectory query groups for association, which is more robust because the multi-target tracking model can further utilize multiple appearance features of different occlusion levels in the trajectory. By selecting the most suitable prediction embedding result, it adapts to changes in the target's occlusion state, thereby improving the accuracy of multi-target tracking results in complex scenes and alleviating the challenges brought by complex scenes.

[0059] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-target tracking method based on Transformer and trajectory query groups, wherein, in addition to the above steps, the method may further include steps S21 to S23:

[0060] Step S21: If no target is detected in the previous image frame of the current image frame, or if the current image frame is the first image frame, input the current image encoding features and the detection query features into the Transformer decoder to obtain the prediction embedding result corresponding to the new target in the current image frame.

[0061] In this embodiment, if no target was detected in the previous image frame, or if the current image frame is the first image frame (i.e., there is no tracked target in the current image frame), then only the current image encoding features and detection query features are input to the Transformer decoder. The result is the predicted embedding result corresponding to the new target in the current image frame, output by the Transformer decoder. This is because the target detected based on the detection query features is a new target; therefore, the Transformer decoder outputs the predicted embedding result corresponding to the new target in the current image frame.

[0062] Step S22: Based on the prediction embedding result corresponding to the new target, determine the sample multi-target tracking result corresponding to the current image frame.

[0063] In this embodiment, there is only one predicted embedding result for each new target. Therefore, the output embedding result corresponding to the new target is the predicted embedding result corresponding to the new target. Thus, the multi-target tracking result of the sample corresponding to the current image frame can be determined directly based on the predicted embedding result corresponding to the new target.

[0064] Step S23: In the next image frame of the current image frame, the new target is taken as the tracked target, and the trajectory query group corresponding to the tracked target in the next image frame is initialized based on the prediction embedding result corresponding to the new target.

[0065] In this embodiment, for each newly detected target, a trajectory query group is initialized as a dedicated query set to track the new target. Specifically, after a new target is detected in the current image frame, in the next image frame, the detected new target is treated as a tracked target. Based on the prediction embedding result corresponding to the new target in the current image frame, the trajectory query group corresponding to the tracked target in the next image frame is initialized. That is, the prediction embedding result corresponding to the new target in the current image frame is effectively converted into trajectory query features in the trajectory query group for subsequent tracking tasks. In one example, the trajectory query group is initialized by combining output content embedding and position embedding.

[0066] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-target tracking method based on Transformer and trajectory query groups. In this method, the multiple targets include: tracked targets and new targets, or multiple tracked targets. Specifically, when the target is a tracked target, each target corresponds to multiple prediction embedding results, and each prediction embedding result is a prediction embedding result corresponding to trajectory query features of different occlusion levels; when the target is a new target, each target corresponds to one prediction embedding result. Furthermore, step S14 above, "determining one output embedding result corresponding to each of the multiple prediction embedding results corresponding to the multiple targets," specifically includes steps S31 and S32:

[0067] Step S31: Determine the loss between the multiple predicted embedding results corresponding to each target and the label embedding results corresponding to each target.

[0068] In this embodiment, each tracked target corresponds to multiple predicted embedding results. At this time, the loss between each of the multiple predicted embedding results corresponding to each target and the label embedding result corresponding to each target can be determined.

[0069] Step S32: Determine the prediction embedding result with the minimum loss as an output embedding result for each target.

[0070] In this embodiment, the prediction embedding result corresponding to the calculated minimum loss is determined as an output embedding result for each tracked target.

[0071] For example, if each tracked target corresponds to three trajectory query features with different occlusion levels, then the predicted embedding results for each of these three trajectory query features are calculated. A first loss value is calculated between the predicted embedding result of the trajectory query feature at the first occlusion level and the corresponding label embedding result. Similarly, a first loss value is calculated between the predicted embedding result of the trajectory query feature at the second occlusion level and the corresponding label embedding result. Finally, a third loss value is calculated between the predicted embedding result of the trajectory query feature at the third occlusion level and the corresponding label embedding result. The minimum loss value is then determined from these three values, and the predicted embedding result corresponding to the minimum loss value is identified as an output embedding result for that tracked target.

[0072] In an optional example, when calculating the loss between multiple predicted embeddings for each target and the label embeddings for each target, the total loss between the predicted and label embeddings is calculated, where the total loss includes classification loss and bounding box loss. However, since different occlusion levels are determined based on confidence levels, simply applying equal weights to the classification loss (calculated based on confidence levels) and the bounding box loss is unfair. Therefore, to reduce the impact of classification loss, this embodiment can assign a higher weight to the bounding box loss to prioritize trajectory query features (queries) that can predict more accurate bounding boxes. Further, in an optional example, the bounding box loss can include L1 loss and GIoU loss; therefore, different weights can be assigned to each loss type, for example: L1 = 5, GIoU = 5, class = 1.

[0073] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-target tracking method based on Transformer and trajectory query groups. In this embodiment, the multi-target tracking model generates a trajectory query group for each tracked target (i.e., trajectory). The trajectory query group stores the content features of the tracked target under different occlusion levels, and the different occlusion levels include at least three occlusion levels.

[0074] In an optional example, this embodiment categorizes different occlusion levels into three types: complete exposure, slight occlusion, and severe occlusion; each of these three occlusion levels corresponds to a different occlusion threshold. Since the occlusion level of the target changes continuously in complex scenes, the detection confidence of the trajectory query features within the trajectory query group also fluctuates. Typically, when the target is completely exposed, the confidence (the query confidence output by the Transformer decoder, i.e., the confidence score of the query prediction) is high; with slight occlusion, the confidence decreases; and with severe occlusion, the confidence is lowest. Therefore, this embodiment sets different occlusion thresholds for the three occlusion levels of the trajectory query group, meaning each of the three occlusion levels corresponds to a different occlusion threshold.

[0075] For example, for the three types of queries in the trajectory query group: fully exposed trajectory query features, slightly occluded trajectory query features, and severely occluded trajectory query features, the corresponding occlusion thresholds are respectively... , and During tracking, the output embedding results can be classified into low-scoring, medium-scoring, and high-scoring embeddings based on the confidence score and occlusion threshold corresponding to the target, respectively representing complete exposure, slight occlusion, and severe occlusion of the target. For example, the three occlusion thresholds are: = 0.85、 = 0.7、 = 0.5. The confidence level of the output embedding result corresponding to the new target is 0.75. <0.75< If the new target is classified as slightly occluded, the output embedding result corresponding to the new target will be saved to the trajectory query feature corresponding to slightly occluded in the trajectory query group corresponding to the target.

[0076] In this embodiment, during the process of the trained multi-target tracking model tracking and detecting the video stream to be detected, when the predicted embedding results corresponding to the three occlusion levels of the target are obtained, the predicted embedding result with the highest confidence is determined as the output embedding result corresponding to the target.

[0077] In other words, during the model inference process of the trained multi-object tracking model, given the predicted embedding results corresponding to the three occlusion levels of the target, this embodiment prioritizes selecting the predicted embedding result with the highest confidence as the output. That is, the predicted embedding result with the highest confidence is determined as the output embedding result corresponding to the target, and this output embedding result will be used for subsequent updates to the trajectory query group. However, if the confidence of the output embedding result falls within the occlusion range corresponding to other occlusion levels, the occlusion level representing the target has changed. In this case, if the new occlusion level is not yet represented in the trajectory query group, it is added to the group; otherwise, the previously existing embedding for the new occlusion level is replaced as the trajectory query feature for that new occlusion level.

[0078] For example, during the inference process, the confidence level is highest in the fully exposed prediction embeddings corresponding to the target, at 0.75, while... <0.75< At this point, the target falls within the occlusion range corresponding to slight occlusion. The occlusion level of the target in the current image frame changes from complete exposure in the previous image frame to slight occlusion in the current image frame. If it is determined that there are no slightly occluded trajectory query features in the target's trajectory query group, the predicted embedding result is added to the trajectory query group as a slightly occluded trajectory query feature. If it is determined that there are slightly occluded trajectory query features in the target's trajectory query group, the predicted embedding result replaces the slightly occluded trajectory query features in that group.

[0079] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating a trajectory query group according to an embodiment of the present invention. Figure 2 In the multi-target tracking model (TGFormer), a set of track queries is generated for each target i. The track query set TG... iThe internal storage contains content features of the tracked target under different occlusion levels, with each group containing a maximum of three Track Queries. Specifically, this method categorizes occlusion levels into three types: fully exposed scene, slightly occluded scene, and heavily occluded scene. Each type of track query feature is responsible for detecting targets at a specific occlusion level. Because the target's occlusion level constantly changes in complex scenes, the detection confidence of the queries within a group also fluctuates. Typically, the confidence is high when the target is fully exposed; it decreases with slight occlusion; and it is lowest with heavy occlusion.

[0080] In related technologies, since DETR introduced the Transformer model into the field of computer vision, especially for object detection, the Query (the embedding input to the Transformer Decoder in DETR, used for object detection) typically combines the content features and location information of the target. While this combination is convenient, it may prevent the model from fully utilizing the independent information of different features to optimize localization accuracy. Therefore, in conjunction with the above embodiments, in one implementation, this invention also provides a multi-target tracking method based on Transformer and trajectory query groups. In this embodiment, the content features and location information in the query features (including detection query features and trajectory query features) are separated. Specifically, these trajectory query features share the same location information but have independent content embeddings adapted to different occlusion levels. That is, the trajectory query features include: query content embedding and query location embedding, and the output embedding result includes: output content embedding and output location embedding; different query content embeddings corresponding to the same target share the same query location embedding.

[0081] Because this embodiment contains multiple trajectory query features within the trajectory query group, it is not possible to directly use traditional methods for updating trajectory query features (in previous Transformer-based trackers, if a target is detected, the Detect Query of the previous frame is passed as the Track Query to the next frame, gradually updating the content features of the Query). Therefore, this embodiment sets up a dedicated updater for the trajectory query group: the Track Query Group Updater (TG Updater), to update the embedded query content in the trajectory query group, thereby further completing the update of the trajectory query group. Specifically, in this embodiment, in addition to the above steps, steps S41 to S44 may also be included:

[0082] Step S41: For each of the multiple targets, the output content embedding of the target in the current image frame, the query content embedding of the target in the trajectory query group in the current image frame, and the long-term memory embedding of the target are input into the trajectory query group updater to obtain the updated query content embedding of the target.

[0083] In this embodiment, for each of the multiple targets detected in the current image frame, the output content embedding of the target in the current image frame, the query content embedding of the target in the trajectory query group in the current image frame, and the long-term memory embedding of the target can be input to the trajectory query group updater, so that the trajectory query group updater updates the query content embedding of the trajectory query group corresponding to the target, and obtains the updated query content embedding of the target output by the trajectory query group updater.

[0084] Step S42: For each of the multiple targets, at least the output position of the target in the current image frame is embedded into the position predictor to obtain the predicted query position embedding of the target in the next image frame.

[0085] In this embodiment, for each of the multiple targets detected in the current image frame, the output position of the target corresponding to the current image frame can be embedded into the position predictor to obtain the predicted query position embedding of the target in the next image frame.

[0086] Step S43: Based on the updated query content embedding corresponding to the target and the predicted query position embedding corresponding to the target in the next image frame, obtain the updated trajectory query group corresponding to the target.

[0087] In this embodiment, after obtaining the updated query content embedding corresponding to the target and the predicted query position embedding corresponding to the target in the next image frame, the updated trajectory query features corresponding to the target can be obtained based on the updated query content embedding corresponding to the target and the predicted query position embedding corresponding to the target in the next image frame, thereby obtaining the updated trajectory query group corresponding to the target.

[0088] Step S44: Use the updated trajectory query group as the trajectory query group corresponding to the tracked target in the next image frame.

[0089] In this embodiment, after obtaining the updated trajectory query group corresponding to the target, the updated trajectory query group corresponding to the target can be used as the trajectory query group corresponding to the target (i.e., the tracked target) in the next image frame.

[0090] In this embodiment, by processing content features and location information independently, the independence of the two is fully utilized to further improve tracking performance. Separating the content features and location information in the query and providing explicit prior location information to the decoder helps to determine the target's location more quickly. This improvement makes the target localization process more explicit and efficient.

[0091] In one optional implementation, when the target is a tracked target, the long-term memory embedding corresponding to the target is the sum of the output content embeddings corresponding to the target in the current image frame and the previous image frames; when the target is a new target, the long-term memory embedding corresponding to the target is the output content embedding corresponding to the target in the current image frame.

[0092] Specifically, this method uses long-term memory embedding. This represents the target's stable characteristics to maintain information about the tracked target for a longer period. When a new target is detected, the output of the new target is embedded. Initialize as long-term memory And update the long memory using a running average method with exponentially decaying weights:

[0093] ;

[0094] Here, λ represents the weight decay exponent, which can be set to 0.01, because it is assumed that the memory of the target changes smoothly and steadily in consecutive frames.

[0095] In conjunction with the above embodiments, in one implementation, the present invention also provides a multi-target tracking method based on Transformer and trajectory query group. In this embodiment, the step S41 above, "embedding the output content corresponding to the target in the current image frame, embedding the query content in the trajectory query group corresponding to the target in the current image frame, and embedding the long-term memory corresponding to the target, and inputting them into the trajectory query group updater to obtain the updated query content embedding corresponding to the target," specifically includes steps S51 to S54:

[0096] Step S51: The query content embeddings of different occlusion levels in the trajectory query group corresponding to the target in the current image frame are concatenated with the output content embeddings of the target in the current image frame and then input into the short-term memory aggregator to obtain the query enhancement content embeddings of different occlusion levels corresponding to the target.

[0097] In this embodiment, the embedding of the query content of the target in the trajectory query group corresponding to the current image frame includes: embedding of the query content of the target at different occlusion levels in the trajectory query group corresponding to the current image frame. The embeddings of the query content of the target at different occlusion levels in the trajectory query group corresponding to the current image frame can be concatenated with the output content embedding of the target in the current image frame to obtain concatenated results corresponding to different occlusion levels. Then, the concatenated results corresponding to different occlusion levels are input into a Short-Term Memory Aggregator for feature enhancement, resulting in the output of the Short-Term Memory Aggregator: enhanced query content embeddings of the target at different occlusion levels. These embeddings serve the same target but contain unique features under different occlusion levels.

[0098] In one optional implementation, the short-term memory aggregator is composed of a multilayer perceptron (MIP), which can obtain the query enhancement content embedding corresponding to different occlusion levels of the target through the following formula. It aims to enhance the target feature representation by concatenating features from multiple frames, and at the same time, combine high-scoring, medium-scoring, and low-scoring embeddings to extract and fuse target features under different scenarios.

[0099] ;

[0100] in, , , These are defined as fully exposed query augmentation embedding, slightly occluded query augmentation embedding, and severely occluded query augmentation embedding; MIP is a short-term memory aggregator. To embed the target in the output content corresponding to the current image frame, , , These are, respectively, the query content embedding when the target is fully exposed in the trajectory query group corresponding to the current image frame, the query content embedding when the target is slightly occluded in the trajectory query group corresponding to the current image frame, and the query content embedding when the target is severely occluded in the trajectory query group corresponding to the current image frame.

[0101] Step S52: Concatenate the query enhancement content embeddings of different occlusion levels to obtain concatenated content embeddings. Use the concatenated content embeddings as keys and values, and use the query enhancement content embeddings of different occlusion levels as queries respectively. Perform occlusion perception memory attention operation to obtain intermediate content embeddings corresponding to different occlusion levels.

[0102] In this embodiment, in order to distinguish these unique features ( , , A multi-head attention module, called Occlusion-Aware Memory-Attention, was designed. After obtaining query-enhanced content embeddings at different occlusion levels, these embeddings were concatenated to obtain the concatenated content embedding. and embed the spliced ​​content As keys and values. Furthermore, in this embodiment, query enhancement content with different occlusion levels is embedded ( , , Each key, value, and query is used as a query. Occlusion-aware memory attention operations are performed based on the key, value, and query to obtain intermediate content embeddings corresponding to different occlusion levels. These intermediate content embeddings represent the unique features of the target under different occlusion levels.

[0103] Step S53: Combine the intermediate content embeddings corresponding to different occlusion levels with the long-term memory embeddings corresponding to the target to obtain the updated content embeddings corresponding to different occlusion levels.

[0104] In this embodiment, after obtaining the intermediate content embeddings corresponding to different occlusion levels, the intermediate content embeddings corresponding to different occlusion levels can be combined with the long-term memory embedding corresponding to the target to obtain the updated content embeddings corresponding to different occlusion levels. Specifically, residual connections can be used to combine the intermediate content embeddings corresponding to different occlusion levels with the long-term memory embedding corresponding to the target. For example, the updated content embeddings corresponding to different occlusion levels can be obtained according to the following formula:

[0105] ;

[0106] in, , , These represent the updated content embeddings for complete exposure, slight occlusion, and severe occlusion, respectively; Attention refers to the occlusion-aware memory attention operation. To embed content, , , These are defined as fully exposed query augmentation content embedding, slightly occluded query augmentation content embedding, and severely occluded query augmentation content embedding. Embed the long-term memory corresponding to the target.

[0107] Step S54: Based on the updated content embedding corresponding to different occlusion levels, obtain the updated query content embedding corresponding to the target.

[0108] In this embodiment, after obtaining the updated content embeddings corresponding to different occlusion levels, the updated query content embeddings corresponding to the target can be obtained based on the updated content embeddings corresponding to different occlusion levels: .

[0109] Furthermore, the updated query content embedded for the target shares the same predicted query position embedding for the target in the next image frame, and together they form an updated trajectory query group, which is input into the next image frame for tracking.

[0110] In this embodiment, the design of the trajectory query group updater significantly improves the multi-target tracking model's ability to associate targets in complex scenarios, enabling it to more accurately identify and track occluded targets or targets in different states. Furthermore, by assigning unique feature representations to targets at different occlusion levels, this design effectively reduces ID switching issues caused by occlusion and appearance changes, thereby significantly improving the stability and continuity of tracking.

[0111] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a trajectory query group updater according to an embodiment of the present invention. Figure 3 In the context of object i, the content embeddings for the target include: the output content embedding of the target in the current image frame. The target is embedded in the query content of the trajectory query group corresponding to the current image frame. and the long-term memory embedding corresponding to the target ( Based on (Updated). Each type of occlusion , , Embedded with the output content corresponding to the target in the current image frame The concatenated input is fed into the short-term memory aggregator to obtain... , , These are: fully exposed query augmentation embedding, slightly occluded query augmentation embedding, and severely occluded query augmentation embedding.

[0112] Then, , , splicing Using these as keys and values, query enhancement content with different occlusion levels is then embedded. , , The following parameters are used as queries: Q1, Q2, and Q3, to promote feature interaction in occlusion-aware memory attention, resulting in intermediate content embeddings corresponding to different occlusion levels. Finally, the intermediate content embeddings corresponding to different occlusion levels are compared with the long-term memory embedding corresponding to the target. By combining these methods, we can obtain the updated content embeddings corresponding to different occlusion levels. .

[0113] For tracking moving targets, directly using the position information of the previous frame as the localization prior for the next frame is inaccurate. To address this issue, in one embodiment, based on the above embodiments, this invention also provides a multi-target tracking method based on Transformer and trajectory query groups. In this method, a simple and efficient position predictor is used. The position predictor combines the predicted position with content embedding to form a trajectory query feature, thereby enabling the multi-target tracking model to more accurately locate targets in dynamic scenes. Specifically, in this embodiment, step S42 may include steps S61 and S62:

[0114] Step S61: Embed the target's output position in the current image frame and the target's historical output position in the previous image frame into the position predictor to obtain the target's predicted query position embedding in the next image frame and the target's historical output position embedding in the current image frame.

[0115] In this embodiment, the target's output position embedding in the current image frame and the target's historical output position embedding in the previous image frame are input to the position predictor to obtain the position predictor outputs: the target's predicted query position embedding in the next image frame and the target's historical output position embedding in the current image frame. The historical output position embedding in the previous image frame is the set of output position embeddings for the target in the previous image frame and all preceding image frames. In an optional implementation, the position prediction can be implemented using an LSTM network, an RNN (Recurrent Neural Network), or a Transformer.

[0116] Step S62: The target is embedded in the historical output position corresponding to the current image frame to provide a continuous temporal context, so as to assist the position predictor in making subsequent prediction query position embedding outputs in the next image frame.

[0117] In this embodiment, the target's historical output position embedding in the current image frame is a set of target output position embeddings in the current image frame and all previous image frames. This target's historical output position embedding in the current image frame is used to provide a continuous temporal context to assist the position predictor in performing subsequent prediction query position embedding output in the next image frame.

[0118] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram illustrating a location predictor according to an embodiment of the present invention. Figure 4 During tracking, the model continuously stores the predicted target position for each iteration. When the number of stored historical target positions exceeds a certain value (arbitrarily set, such as 5 frames, without limitation), the position predictor can begin using this data to predict the target's movement trend. Specifically, these position sequences are input into the position predictor... Figure 6 In this model, the position predictor is implemented using an LSTM network to predict the target's position information in the next frame. The LSTM generates two outputs: and .in, This is to predict the target's location information in the next frame (i.e., embedding the predicted query location of the target in the next image frame), and This stores information implicit in the previous sequence to provide a continuous temporal context, i.e. The target is embedded at the historical output position corresponding to the current image frame. In the next frame, It will assist in target detection. The output position corresponding to the next image frame is embedded in X. t+1 The data is then fed into the LSTM for further prediction. The advantage of this design is that it eliminates the need to continuously store all target locations; only key information needs to be retained to complete the prediction task.

[0119] In this embodiment, a position predictor is used to predict the target's position information in the next frame using historical trajectories. This prediction is then provided to the Query as a position prior to enhance detection capabilities. The design of the position predictor significantly improves the localization accuracy of moving targets during tracking, enabling the model to more effectively handle dynamic changes in the target, particularly demonstrating higher accuracy and robustness in tracking moving targets in complex scenes. By providing a more efficient position prior information update strategy, the position information of dynamically moving targets can reflect the actual motion state of the target in a timely manner, reducing the decrease in tracking accuracy caused by motion prediction lag, thereby optimizing target localization accuracy. This avoids directly using the position information of the previous frame as the localization prior for the next frame, as tracked targets are usually dynamically moving, and this approach is often inaccurate, especially for fast-moving targets, where such prior information may lag behind the target's actual motion, leading to a decrease in tracking accuracy.

[0120] In one embodiment, such as Figure 5 As shown, Figure 5 This is a flowchart illustrating a multi-target tracking model based on trajectory query groups, as shown in an embodiment of the present invention. Figure 5 In the video input stream, for frames... First, the data is fed into the backbone network and the Transformer encoder to extract two-dimensional features, which serve as one of the inputs to the decoder. The multi-object tracking model uses a fixed number of learnable embeddings, or Detect Queries, to detect newly emerging targets in the current frame, with each tracked target corresponding to a trajectory query group. Within each group, Track Queries collectively predict the same target. Specifically, these queries share the same position embeddings but have independent content embeddings adapted to different occlusion levels. Simultaneously, the multi-target tracking model predicts the target's position information from historical trajectories using a position predictor and combines this information with the content embeddings to form track query groups. The inputs are fed into the decoder. The decoder generates embeddings for all queries within the group (including DetectQueries and trajectory query groups). These embeddings are processed by the prediction network to obtain confidence scores and bounding boxes. The model selects the most appropriate prediction based on these results and uses the corresponding decoder output as the output embedding. In the post-processing stage, the output embeddings, long-term memory, and trajectory query groups are fed together into the TG Updater to update the queries within the trajectory query groups.

[0121] To ensure long-term stable association, this embodiment proposes a novel updater, TG Updater, applicable to Track Query Groups (TG). All Track Query queries within the group are continuously updated in each frame using the proposed TG Updater. For the TG Updater, a Short-Term Memory Aggregator is used to embed and fuse the queries within the group with the output content obtained in the current frame's output embedding. Since targets exhibit different features under different occlusion levels, an Occlusion-Aware Memory Attention mechanism is applied to distinguish the target's features at each occlusion level. Simultaneously, a long-term memory is maintained for the target, and this memory is connected to the output of the memory attention mechanism via residual connections to help the model learn more discriminative representations. The Track Query Group (TG) is continuously updated in each frame using the above method, ensuring that the target is always accurately captured in complex scenes. Furthermore, this embodiment introduces a PositionPredictor, enabling the tracker to predict motion trends and helping the model more accurately locate moving targets.

[0122] For example, via I t-1 The updated content embedding from the frame TG Updater output and the position embedding from the position predictor output are used to update the trajectory query features in the target's trajectory query group for use in I t The frame is input into the Transformer decoder. Figure 5 For I t-1 The process between the Transformer decoder and the TG Updater and position predictor is omitted; in fact, it is related to I... t The process between the Transformer decoder, TG Updater, and position predictor in a frame is the same: first, the predicted embeddings output by the decoder are obtained, and then the output embeddings are selected from the predicted embeddings. The output content in the output embeddings is then embedded into the input TG Updater to update the query content embedding. Finally, the output position in the output embeddings is embedded into the input position predictor to output the corresponding predicted query position embedding in the next image frame.

[0123] In one alternative embodiment, multi-target tracking plays an important role in security monitoring applications, such as detecting and tracking pedestrians in street scenes, shopping malls, and roads. Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the application of a multi-target tracking task in street scene crowds, as shown in one embodiment of the present invention. Street scenes contain dense crowds, and security monitoring requires the detection and continuous tracking of these pedestrians. To accomplish this security monitoring task, this embodiment requires training a multi-target tracking model adapted to the specific task, as follows: Figure 7 As shown, Figure 7 This is a flowchart illustrating a model training process according to an embodiment of the present invention. The detailed steps of the model training are as follows:

[0124] 1. Constructing a training dataset. For street scene monitoring, publicly available datasets like MOT17 and MOT20 are typically used as training datasets. However, if monitoring is required in a specific scenario, staff need to construct their own training dataset. This involves capturing scene videos over a certain time period using an RGB monocular camera, and then labeling each task appearing in the video, including its bounding box and ID. The specific data format is as follows: <id>,<bb_left> ,<bb_top> ,<bb_width> ,<bb_height> ,<trajectory_conf> ,<trajectory_type> ,<visibility_ratio> Wherein represents the frame number. <id>The trajectory ID representing the target.<bb_left> and<bb_top> This represents the coordinates of the top-left corner of the bounding box.<bb_width> and<bb_height> This indicates the width and height of the bounding box.<trajectory_conf> A flag indicating whether the trajectory is considered (0 indicates ignored, 1 indicates active).<trajectory_type> Indicates the category of the target.<visibility_ratio> This indicates the visibility percentage of the target, thus completing the construction of a specific dataset.

[0125] 2. Configure training parameters. For example, the multi-object tracking model uses ResNet50 as the backbone network and DAB-Deformable-DETR (pre-trained on the COCO dataset) as the detector. Training the model is recommended on a graphics card with 24GB or more of VRAM, such as an NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 4090, A100, or A800. The optimizer used is AdamW with an initial learning rate of 2.0 × 10^(-4). During training, the model filters out tracked targets with scores and IoU below a threshold of 0.5. In the model, each target corresponds to a Track Query Group. When a Detect Query detects a new target, a Track Query Group is initialized, and the query is categorized into high-score embeddings, mid-score embeddings, and low-score embeddings based on confidence. Other embeddings are initially set to zero vectors. During tracking, the detection confidence of the output embeddings may change. When the target's occlusion level changes significantly, causing the detection confidence to fall within the threshold range of the other two embeddings, the output embedding is saved as the corresponding embedding. For example, the confidence thresholds are: high = 0.85, mid = 0.7, and low = 0.5.

[0126] 3. Start Training. Before training the model, you need to configure the runtime environment. The PyTorch module is required, specifically the following versions: pytorch==1.13.1, torchvision==0.14.1, torchaudio==0.13.1, pytorch-cuda=11.7.

[0127] 4. Read image data. Read the corresponding image data and its ground truth data from the training dataset. The batch size of the training image data loaded per GPU is 1, and each batch contains a video clip containing multiple frames. In each clip, frames are sampled at random intervals (1 to 10 frames).

[0128] 5. Data Augmentation. During training, to improve the robustness of the model, image data is augmented, such as cropping and horizontal flipping. Finally, the RGB data is standardized and output as Tensor data for model processing.

[0129] 6. Input Model Inference. The processed image data is input into the model for inference.

[0130] 7. Output prediction results. The model inference yields the tracking results, including the detection results of all targets in the image: detection confidence and bounding boxes, as well as the target ID.

[0131] 8. Calculate the loss and then backpropagate. During training, calculate the loss between each query and its corresponding ground truth value in the trajectory group, including classification loss, L1 loss, and GIoU loss. Select the query with the smallest loss as the final tracking result. To reduce the impact of classification loss, assign different weights to each loss type: L1 = 5, GIoU = 5, class = 1.

[0132] 9. Save the model. Save the current model after each training cycle.

[0133] 10. End training. Train on the dataset for 130 epochs, reducing the learning rate by a factor of 10 on the 120th epoch. On the 50th, 70th, 90th, and 120th epochs, gradually increase the number of clip frames from the original 2 frames to 3, 4, 5, and 6 frames, respectively. End training after all epochs are completed.

[0134] 11. Obtain the trained model. After training, the final multi-object tracking model is obtained, which can be validated on the validation set.

[0135] After training, the saved multi-object tracking model is obtained and can be used in practical applications. The inference process is as follows: Figure 8 As shown, Figure 8 This is a flowchart illustrating a model reasoning process according to an embodiment of the present invention. The specific reasoning process is as follows:

[0136] 1. Acquiring Image Data. In practical security monitoring scenarios, there are two ways to read data: one is to read previously stored historical video data from the database, and the other is to acquire real-time video from a camera (such as an RGB monocular camera) and return the image data as model input.

[0137] 2. Process image data. Convert the acquired image data into tensors for model input.

[0138] 3. Load the model. Load the trained multi-object tracking model.

[0139] 4. Begin inference. Once the data and model are loaded, begin tracking the target in the scene.

[0140] 5. Input image data sequentially. Input the image data into the model frame by frame in chronological order.

[0141] 6. Model Inference. After the image data is processed by the model, the final prediction result is obtained, including the target ID and bounding box information in each frame.

[0142] 7. Save the prediction results. Save the prediction results as a txt file.

[0143] 8. Visualize Tracking Results. Plot the bounding box information from the prediction results onto the image to mark the target location in the scene, and label its ID in the upper left corner. Provide the visualized tracking results to the operator in real time. For Figure 6 Street view surveillance, visualization results as follows Figure 9 As shown, Figure 9 This is a schematic diagram illustrating a street scene crowd tracking visualization according to an embodiment of the present invention.

[0144] 9. End Inference. For historical stored datasets, the inference process ends when all datasets have been inferred. For real-time monitoring, the operator must manually close the tracking program.

[0145] Furthermore, in another embodiment, multi-target tracking tasks can be applied not only to security monitoring but also to certain specialized tasks, such as tracking individuals in sports or dance performances. Figure 10 As shown, Figure 10 This is a schematic diagram illustrating the application of a multi-target tracking task in a dance scene according to an embodiment of the present invention. In this type of task, the characters appearing in the scene are relatively fixed. Therefore, the difficulty lies in the fact that the appearances of the characters in this scene are mostly similar, and their movement trends are difficult to estimate, which places higher demands on the tracker. The training and inference process for this type of task is the same as in the above embodiment, with the difference being in the setting and method of certain parameters during training. The specific steps and differences are as follows:

[0146] Regarding dataset construction: For dance scenarios, the publicly available dataset DanceTrack can typically be used as the training dataset. However, if tracking in specific scenarios is required, staff will need to construct their own training dataset, following the same method as described in the previous examples.

[0147] Inference based on the input model: This involves inputting the processed image data into the model for inference.

[0148] To terminate training: the model was trained on the dataset for 18 epochs, with the number of frames per clip increased to 3, 4, and 5 at epochs 6, 10, and 14, respectively. For dance scene tracking, the visualization results are as follows... Figure 11 As shown. Figure 11 This is a schematic diagram illustrating the tracking and visualization of a dance scene according to an embodiment of the present invention.

[0149] In one embodiment, to verify the effectiveness of the multi-target tracking method based on Transformer and trajectory query groups proposed in this invention, experiments were conducted on the public multi-target tracking datasets MOT Challenge and DanceTrack, and compared with other techniques to verify the effectiveness of the method. For verification metrics, this embodiment uses CLEAR MOT Metrics and HOTA as verification standards. CLEAR MOT Metrics is widely used and includes MOTA, IDF1, ID-SW, MT, and ML. For HOTA, this project uses HOTA, AssA, and DetA. MOTA is the multi-target tracking accuracy, mainly focusing on the overall error rate of tracking. It measures the overall accuracy of the tracking task by calculating the weighted sum of missed detections, false positives, and ID switching (ID-SW). A higher value indicates better overall performance of the algorithm in detection and tracking. IDF1 is used to measure the ID consistency of the target throughout the sequence. It reflects the stability and continuity of tracking by calculating the ID matching F1 score of the target in the tracking results. A higher IDF1 score indicates that the target is accurately tracked throughout the video, demonstrating good ID consistency. ID Switching Count (ID-SW) represents the number of times the ID of the same target incorrectly switches during tracking. Fewer ID switching counts indicate a more stable tracking algorithm in target association. High-frequency ID-SW means the algorithm struggles to maintain ID consistency over long sequences. MT refers to the number of targets that are mostly tracked correctly, representing the proportion of targets that are correctly tracked for at least 80% of the tracking sequence. A higher MT value indicates better algorithm performance in long-term target tracking. ML refers to the number of targets that are mostly lost, representing the proportion of targets tracked for less than 20% of the entire sequence. A lower ML value indicates more stable tracking performance across different frames and a lower likelihood of target loss. HOTA is a new multi-target tracking evaluation metric that comprehensively evaluates algorithm performance by balancing detection and association capabilities. HOTA measures the quality of target detection in both spatial and temporal dimensions, considering both detection and association performance. A higher HOTA value indicates a better balance between detection and ID association. AssA is HOTA's association accuracy sub-metric, focusing on evaluating the algorithm's ID association ability and measuring whether the same target is correctly kept under the same ID across multiple frames. A higher AssA indicates better ID consistency and association performance. DetA is HOTA's detection accuracy sub-metric, focusing on the accuracy of target detection. A higher DetA indicates that the algorithm can detect and locate targets more accurately, unaffected by association issues.

[0150] As shown in Tables 1 and 2, Ours characterizes the multi-target tracking method based on Transformer and trajectory query groups proposed in this invention. The detection results are compared with the latest multi-target tracking methods based on Transformer on the MOT17 and MOT20 test sets. It can be found that the method of this invention has further improvements in HOTA and IDF1 compared with other techniques, reduces ID switching, and significantly improves MOTA.

[0151] Table 1. Comparison with other techniques on the MOT17 dataset

[0152]

[0153] Table 2 compares the results with other techniques on the MOT20 dataset.

[0154]

[0155] In the DanceTrack dataset, the motion states of the targets are more complex, which places higher demands on the model's tracking capabilities. Our method was tested on this dataset, and the results are shown in Table 3. The results show that our invention (Ours) has significant advantages in detection and tracking performance compared to other models.

[0156] Table 3. Comparison with other technologies on the DanceTrack dataset

[0157]

[0158] This demonstrates that the design of this method has been effectively validated on multiple datasets and different challenge types, indicating that the trajectory query group design of this invention effectively improves the detection capability of the tracking model in complex scenarios, and can still maintain tracking even when the target occlusion state changes frequently, thereby improving the robustness and fault tolerance of the model and making the overall model tracking performance better.

[0159] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0160] Based on the same inventive concept, one embodiment of the present invention provides a multi-target tracking device based on Transformer and trajectory query group. (Reference) Figure 12 , Figure 12 This is a structural block diagram of a multi-target tracking device based on Transformer and trajectory query group, provided in an embodiment of the present invention. Figure 12 As shown, the multi-target tracking device based on Transformer and trajectory query group in this embodiment may include:

[0161] An image input module is used to input the current image frame from the sample video stream into the multi-target tracking model to be trained. The multi-target tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder.

[0162] The image encoding module is used to process the current image frame through the backbone network and the Transformer encoder to obtain the current image encoding features;

[0163] The embedding prediction module is used to input the current image encoding features, detection query features, and trajectory query groups corresponding to the tracked targets in the current image frame into the Transformer decoder when a target is detected in the previous image frame of the current image frame, to obtain one or more prediction embedding results corresponding to multiple targets respectively; each tracked target corresponds to a trajectory query group, and the trajectory query group includes: multiple trajectory query features representing different occlusion levels;

[0164] The embedding output module is used to determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets respectively;

[0165] The tracking and prediction module is used to determine the multi-target tracking result of the sample corresponding to the current image frame based on the output embedding results corresponding to multiple targets respectively.

[0166] The model training module is used to train the multi-target tracking model to be trained based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, so as to obtain the trained multi-target tracking model.

[0167] The tracking and detection module is used to input the video stream to be detected into the trained multi-target tracking model to obtain the multi-target tracking result corresponding to the video stream to be detected.

[0168] Optionally, the device further includes:

[0169] The result prediction module is used to input the current image encoding features and the detection query features into the Transformer decoder when no target is detected in the previous image frame of the current image frame, or when the current image frame is the first image frame, to obtain the prediction embedding result corresponding to the new target in the current image frame.

[0170] The tracking determination module is used to determine the multi-target tracking result of the sample corresponding to the current image frame based on the prediction embedding result corresponding to the new target;

[0171] An initialization module is used to initialize the trajectory query group corresponding to the tracked target in the next image frame of the current image frame, taking the new target as the tracked target, based on the prediction embedding result corresponding to the new target.

[0172] Optionally, the plurality of targets includes: tracked targets and new targets, or, a plurality of tracked targets;

[0173] When the target is the tracked target, each target corresponds to multiple prediction embedding results, and each prediction embedding result is a prediction embedding result corresponding to the trajectory query features of different occlusion levels;

[0174] When the target is the new target, each target corresponds to a prediction embedding result;

[0175] The embedded output module includes:

[0176] The loss calculation module is used to determine the loss between the multiple predicted embedding results corresponding to each target and the label embedding results corresponding to each target.

[0177] The embedding result determination module is used to determine the prediction embedding result with the minimum loss as an output embedding result for each target.

[0178] Optionally, the different occlusion levels include at least three occlusion levels, and the three occlusion levels correspond to different occlusion thresholds;

[0179] During the process of tracking and detecting the video stream to be detected by the trained multi-target tracking model, when the predicted embedding results corresponding to the three occlusion levels of the target are obtained, the predicted embedding result with the highest confidence is determined as the output embedding result corresponding to the target.

[0180] When the confidence level of the output embedding result falls within the occlusion range corresponding to other occlusion levels, the occlusion level representing the target changes.

[0181] Optionally, the trajectory query features include: query content embedding and query location embedding, and the output embedding result includes: output content embedding and output location embedding; the device further includes:

[0182] The query content update module is used to embed the output content of the target in the current image frame, the query content of the target in the trajectory query group in the current image frame, and the long-term memory embedding of the target for each of the multiple targets, and input them to the trajectory query group updater to obtain the updated query content embedding of the target.

[0183] The query position prediction module is used to, for each of multiple targets, at least embed the output position of the target in the current image frame into the position predictor to obtain the predicted query position embedding of the target in the next image frame.

[0184] The query group update module is used to obtain the updated trajectory query group corresponding to the target based on the embedded updated query content corresponding to the target and the embedded predicted query position corresponding to the target in the next image frame.

[0185] The query group determination module is used to use the updated trajectory query group as the trajectory query group corresponding to the tracked target in the next image frame;

[0186] In this context, different query content embedded for the same target share the same query location embedding.

[0187] Optionally, the query content update module includes:

[0188] The first input module is used to concatenate the query content embedding of different occlusion levels of the target in the trajectory query group corresponding to the current image frame with the output content embedding of the target corresponding to the current image frame and input them into the short-term memory aggregator to obtain the query enhancement content embedding of different occlusion levels corresponding to the target.

[0189] The attention module is used to concatenate query enhancement content embeddings of different occlusion levels to obtain concatenated content embeddings. The concatenated content embeddings are used as keys and values, and the query enhancement content embeddings of different occlusion levels are used as queries respectively. Occlusion perception memory attention operation is performed to obtain intermediate content embeddings corresponding to different occlusion levels.

[0190] The feature combination module is used to combine the intermediate content embeddings corresponding to different occlusion levels with the long-term memory embeddings corresponding to the target, so as to obtain the updated content embeddings corresponding to different occlusion levels.

[0191] The content update module is used to embed updated content based on different occlusion levels to obtain the updated query content embedding corresponding to the target.

[0192] Optionally, the query location prediction module includes:

[0193] The first determining module is used to embed the target at the output position corresponding to the current image frame and the target at the historical output position corresponding to the previous image frame into the position predictor to obtain the target at the predicted query position embedding in the next image frame and the target at the historical output position embedding in the current image frame.

[0194] The target is embedded in the historical output position corresponding to the current image frame to provide a continuous temporal context, so as to assist the position predictor in making subsequent prediction query position embedding outputs in the next image frame.

[0195] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the multi-target tracking method based on Transformer and trajectory query group as described in any of the above embodiments of the present invention.

[0196] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the multi-target tracking method based on Transformer and trajectory query group described in any of the above embodiments of the present invention.

[0197] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0198] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0199] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0204] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0205] The foregoing has provided a detailed description of a multi-target tracking method, apparatus, device, and medium based on Transformer and trajectory query group provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / id> < / id>

Claims

1. A multi-target tracking method based on Transformer and trajectory query group, characterized in that, The method includes: The current image frame in the sample video stream is input into the multi-object tracking model to be trained. The multi-object tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder. The current image frame is processed by the backbone network and the Transformer encoder to obtain the current image coding features; If a target is detected in the previous image frame of the current image frame, the current image encoding features, detection query features, and trajectory query groups corresponding to the tracked targets in the current image frame are input into the Transformer decoder to obtain one or more prediction embedding results corresponding to multiple targets respectively; each tracked target corresponds to a trajectory query group, and the trajectory query group includes: multiple trajectory query features representing different occlusion levels; Determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets; Based on the output embedding results corresponding to multiple targets, the sample multi-target tracking result corresponding to the current image frame is determined; Based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, the multi-target tracking model to be trained is trained to obtain the trained multi-target tracking model. The video stream to be detected is input into the trained multi-object tracking model to obtain the multi-object tracking result corresponding to the video stream to be detected. The trajectory query features include: query content embedding and query location embedding; the output embedding result includes: output content embedding and output location embedding; the method further includes: For each of the multiple targets, the output content embedding of the target in the current image frame, the query content embedding of the target in the trajectory query group in the current image frame, and the long-term memory embedding of the target are input into the trajectory query group updater to obtain the updated query content embedding of the target. For each of the multiple targets, the output position of the target corresponding to the current image frame is embedded into the position predictor to obtain the predicted query position embedding of the target in the next image frame; Based on the updated query content embedding corresponding to the target and the predicted query position embedding corresponding to the target in the next image frame, the updated trajectory query group corresponding to the target is obtained; The updated trajectory query group is used as the trajectory query group corresponding to the tracked target in the next image frame; In this context, different query content embedded for the same target share the same query location embedding.

2. The multi-target tracking method based on Transformer and trajectory query group according to claim 1, characterized in that, The method further includes: If no target is detected in the previous image frame of the current image frame, or if the current image frame is the first image frame, the current image encoding features and the detection query features are input into the Transformer decoder to obtain the prediction embedding result corresponding to the new target in the current image frame; Based on the prediction embedding result corresponding to the new target, determine the sample multi-target tracking result corresponding to the current image frame; In the next image frame after the current image frame, the new target is taken as the tracked target, and the trajectory query group corresponding to the tracked target in the next image frame is initialized based on the prediction embedding result corresponding to the new target.

3. The multi-target tracking method based on Transformer and trajectory query group according to claim 1, characterized in that, The multiple targets include: tracked targets and new targets, or multiple tracked targets; When the target is the tracked target, each target corresponds to multiple prediction embedding results, and each prediction embedding result is a prediction embedding result corresponding to the trajectory query features of different occlusion levels; When the target is the new target, each target corresponds to a prediction embedding result; From multiple predicted embeddings corresponding to multiple targets, determine an output embedding for each target, including: Determine the loss between multiple predicted embedding results corresponding to each target and the label embedding results corresponding to each target; The prediction embedding result with the minimum loss is determined as an output embedding result for each target.

4. The multi-target tracking method based on Transformer and trajectory query group according to claim 1, characterized in that, The different occlusion levels include at least three occlusion levels, and the three occlusion levels correspond to different occlusion thresholds; During the process of tracking and detecting the video stream to be detected by the trained multi-target tracking model, when the predicted embedding results corresponding to the three occlusion levels of the target are obtained, the predicted embedding result with the highest confidence is determined as the output embedding result corresponding to the target. When the confidence level of the output embedding result falls within the occlusion range corresponding to other occlusion levels, the occlusion level representing the target changes.

5. The multi-target tracking method based on Transformer and trajectory query group according to claim 1, characterized in that, The target's output content embedding in the current image frame, the target's query content embedding in the trajectory query group corresponding to the current image frame, and the target's long-term memory embedding are input into the trajectory query group updater to obtain the updated query content embedding corresponding to the target, including: The query content embeddings of different occlusion levels in the trajectory query group corresponding to the target in the current image frame are concatenated with the output content embeddings of the target in the current image frame and then input into the short-term memory aggregator to obtain the query enhancement content embeddings of different occlusion levels corresponding to the target. The query enhancement content embeddings of different occlusion levels are concatenated to obtain the concatenated content embedding. The concatenated content embedding is used as the key and value, and the query enhancement content embeddings of different occlusion levels are used as queries respectively. Occlusion perception memory attention operation is performed to obtain the intermediate content embeddings corresponding to different occlusion levels. The intermediate content embeddings corresponding to different occlusion levels are combined with the long-term memory embeddings corresponding to the target to obtain the updated content embeddings corresponding to different occlusion levels. Based on the updated content embedding corresponding to different occlusion levels, the updated query content embedding corresponding to the target is obtained.

6. The multi-target tracking method based on Transformer and trajectory query group according to claim 1, characterized in that, At least the target's output position corresponding to the current image frame is embedded into the position predictor to obtain the target's predicted query position embedding in the next image frame, including: The target is embedded in the output position corresponding to the current image frame, and the target is embedded in the historical output position corresponding to the previous image frame. The input of the target in the position predictor is to obtain the target in the next image frame and the target in the current image frame. The target is embedded in the historical output position corresponding to the current image frame to provide a continuous temporal context, so as to assist the position predictor in making subsequent prediction query position embedding outputs in the next image frame.

7. A multi-target tracking device based on Transformer and trajectory query group, characterized in that, The device includes: An image input module is used to input the current image frame from the sample video stream into the multi-target tracking model to be trained. The multi-target tracking model to be trained includes at least: a backbone network, a Transformer encoder, and a Transformer decoder. The image encoding module is used to process the current image frame through the backbone network and the Transformer encoder to obtain the current image encoding features; The embedding prediction module is used to input the current image encoding features, detection query features, and trajectory query groups corresponding to the tracked targets in the current image frame into the Transformer decoder when a target is detected in the previous image frame of the current image frame, to obtain one or more prediction embedding results corresponding to multiple targets respectively; each tracked target corresponds to a trajectory query group, and the trajectory query group includes: multiple trajectory query features representing different occlusion levels; The embedding output module is used to determine an output embedding result corresponding to each of the multiple targets from one or more prediction embedding results corresponding to the multiple targets respectively; The tracking and prediction module is used to determine the multi-target tracking result of the sample corresponding to the current image frame based on the output embedding results corresponding to multiple targets respectively. The model training module is used to train the multi-target tracking model to be trained based on the sample multi-target tracking results and the multi-target tracking result labels corresponding to each image frame in the sample video stream, so as to obtain the trained multi-target tracking model. The tracking and detection module is used to input the video stream to be detected into the trained multi-target tracking model to obtain the multi-target tracking result corresponding to the video stream to be detected. The trajectory query features include: query content embedding and query location embedding; the output embedding result includes: output content embedding and output location embedding; the device further includes: The query content update module is used to embed the output content of the target in the current image frame, the query content of the target in the trajectory query group in the current image frame, and the long-term memory embedding of the target for each of the multiple targets, and input them to the trajectory query group updater to obtain the updated query content embedding of the target. The query position prediction module is used to, for each of multiple targets, at least embed the output position of the target in the current image frame into the position predictor to obtain the predicted query position embedding of the target in the next image frame. The query group update module is used to obtain the updated trajectory query group corresponding to the target based on the embedded updated query content corresponding to the target and the embedded predicted query position corresponding to the target in the next image frame. The query group determination module is used to use the updated trajectory query group as the trajectory query group corresponding to the tracked target in the next image frame; In this context, different query content embedded for the same target share the same query location embedding.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the multi-target tracking method based on Transformer and trajectory query group as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-target tracking method based on Transformer and trajectory query group as described in any one of claims 1 to 6.

Images