Multi-target tracking method, device and tracking system

Abstract

The invention provides a multi-target tracking method and device and a tracking system, and belongs to the technical field related to computer vision and machine learning, and the method comprises the steps: employing a joint detection and tracking framework for consistent tracking, converting target detection and correlation tasks into a denoising diffusion process, remarkably improving the noise immunity of a model, reducing the ID switching frequency, and improving the tracking efficiency. The robustness of a tracker is greatly improved through the progressive denoising diffusion strategy, in the training stage, paired object frames in two adjacent frames are diffused from a GT frame to random distribution, then the model learns detection and tracking through the inversion process, and in the reasoning process, the robustness of the tracker is greatly improved. According to the model, a randomly generated frame is refined into detection and tracking results through a minimum denoising step, an innovative target association strategy is further introduced in consistency tracking to solve the target shielding problem, and experimental results show that on MOT17 and DanceTrack data sets, consistency tracking exceeds other existing methods in multiple performance indexes.

Description

Technical Field

[0001] The present application relates to the technical fields related to computer vision and machine learning, and in particular to a multi-target tracking method, device and tracking system. Background Art

[0002] Multi-object tracking (MOT) is a key technology in the field of computer vision, which makes it possible to detect, locate and track multiple targets in real time in a video sequence. This is of great significance for various application scenarios, such as video surveillance, traffic management, behavior analysis, etc.

[0003] The prior art includes the following: Tracking based on detection (TBD): Traditional methods first detect targets and then associate detected targets through data association algorithms such as SORT and DeepSORT. These traditional methods rely on the accuracy of initial detection and are prone to failure in complex scenes. Joint Detection and Embedding (JDE): To improve the efficiency of detection and re-identification, the joint detection and embedding method integrates feature extraction with the detector to reduce the need for an independent re-identification module, but there is a technical defect that the detection quality may be sacrificed due to joint optimization; Joint Detection and Tracking (JDT): Joint detection and tracking methods seamlessly combine the detection and tracking stages to improve efficiency and reduce computational overlap. However, JDT methods face challenges in distinguishing closely spaced objects and maintaining consistency in dynamic environments. Existing MOT methods still have shortcomings in noise resistance and ID switching frequency, especially in scenes with occlusion, overlapping targets or fast motion. Traditional diffusion model-based trackers (such as DiffusionTrack) also have the defect of sacrificing certain flexibility and computational efficiency during the iterative denoising process.

[0004] In order to overcome the limitations of the existing technology, this application proposes a novel consistency tracking (ConsistencyTrack) method, which adopts a consistency model generation strategy and significantly improves the model's noise resistance and tracking accuracy by converting detection and association tasks into a denoising diffusion process. Summary of the invention

[0005] The embodiments of the present application provide a multi-target tracking method, device and tracking system, which are intended to solve the problems that multi-target tracking systems often face, such as noise interference and target occlusion, which lead to reduced tracking accuracy and target identity switching errors. An innovative joint detection and tracking (JDT) framework is adopted, and the ability to accurately track multiple targets in complex scenarios is significantly improved through a denoising diffusion process. Compared with the prior art, consistency tracking specially designs an ordinary differential equation (ODE) framework based on probability flow (PF) to achieve efficient conversion from random noise vectors to target detection and tracking results.

[0006] In a first aspect, the present application provides a multi-target tracking method for real-time tracking of multiple targets in a video sequence, the multi-target tracking method comprising: S1. Build a denoising diffusion model based on the joint detection and tracking framework. In the forward diffusion process, extract data points from the real distribution , Gaussian noise is gradually introduced in T steps; in the reverse diffusion process, random noise Through modeling Denoising, removing noise from the target distribution and enhancing the signal, using a denoising diffusion model to correlate detection results for object tracking; S2, the training phase diffuses the paired target boxes in two adjacent frames from the true box to a random distribution, and then the model learning phase performs detection and tracking by reversing the process; S3. In the inference phase, the model refines the randomly generated boxes into detection and tracking results through a minimum denoising step; S4. Introduce innovative target association strategies to solve the target occlusion problem in order to obtain multi-target tracking results.

[0007] Optional, In S1, the denoising diffusion model uses a diffusion process algorithm to minimize the noise interference in the input data, specifically including: Starting with random noise and gradually refining it until the random noise closely matches a sample in the target distribution, the following iterative process is introduced during forward diffusion: ; in, is the current time step In the backward diffusion process, at each backward step t, the conditional probability distribution is represented by a network , using the time step and the previous output As input, we represent: ; in, and , through iterative operations, the noise is gradually reduced in the current state, and finally When the original time step of is less than the preset threshold, make it equal to the predicted box position .

[0008] Optionally, in S1, the denoising diffusion model adopts two methods to implement boundary conditions within the framework of the consistency model, specifically including: let Represents a system with input The first approach directly parameterizes the consistency model, where , ,in, is the range Integer within ; The second approach is to parameterize the consistency model by incorporating residual connections, which can be formalized as: ,in, and is a differentiable function that satisfies and .

[0009] Optionally, in S1, the denoising diffusion model includes a feature extraction background and a data association denoising head, wherein the data association denoising head uses the feature as a condition instead of the original image to gradually refine the paired association box prediction, specifically including: Consistency tracking consists of two components: a feature extraction backbone and a data association denoising head. The feature extraction backbone processes two adjacent input images. To extract deep feature representation, the data association denoising head uses features as conditions instead of the original image, and gradually refines the paired association box predictions from the paired noise boxes. The data samples are composed of paired boundary Composition, of which , neural network for is trained to extract the noise from pairs of frames Medium Forecast , provided that the corresponding two adjacent images , and accordingly generate two corresponding associated confidence scores S; if , then the MOT task degenerates into an object detection problem. The consistent properties allow consistent tracking to solve both object detection and tracking tasks. Set to 1 only during the trace matching process.

[0010] Optionally, in S1, the denoising diffusion model includes a space-time fusion module for realizing the exchange of time information between paired frames in two consecutive frames to promote complete data association, specifically including: The STF module is proposed to achieve the exchange of temporal information between paired boxes of two consecutive frames, facilitating complete data association, given the ROI characteristics of two consecutive time steps. ,in, is the batch size, is the number of regions, is the feature dimension, output query , then proceed with the following transformation: Linear transformation and segmentation: Each query , first transform using linear projection, then split the result into two separate tensors of equal dimension ; Batch matrix multiplication: RoI features from two time steps are concatenated, and then two consecutive batch matrix multiplications are performed to obtain the feature tensor ; Final linear transformation: The feature tensor obtained by batch matrix multiplication is further processed to generate the object query of the current block .

[0011] Optionally, in S2, the paired target boxes in two adjacent frames are diffused from the real boxes to the random distribution in the training phase, and then the model learning phase performs detection and tracking through the reversal process, specifically including: A pair of frames are randomly selected from the video sequence as the input of the model, and the filling strategy is implemented by introducing auxiliary boxes around the GT boxes. First, the GT boxes in the image are supplemented to the total number ; Then, based on the noise addition strategy of the consistency model, random noise is added to the original GT frames of the two frames, and all noisy frames are input into the model for denoising; Finally, the model extracts the association relationship between the instance frames in the two adjacent frames, calculates the loss, and performs back propagation operations. The filled instance is represented as , for the i-th GT box, it is expressed as , applying Gaussian noise to its four parameters , at a randomly selected time step t, the noise box is restricted in scope at the tth time step, and the scaling factor of the noise is determined as follows: ; Then, the original filled samples Introducing noise: ,in, is a randomly generated Gaussian noise. Finally, the range of the noise box is constrained by: ; Among them, represents the scale factor of the noise box, .

[0012] Optionally, in S3, in the inference phase, the model refines the randomly generated boxes into detection and tracking results through a minimum denoising step, specifically including: According to consistency inference, the first input is an image In paired frames , total time step T,t, number of sampling steps ; Output is the final prediction , calculate the time step is the total time step T divided by the number of sampling steps Next, generate a random noise that matches the size of the preset box number , then, from the input pairs of frames The features extracted will be used for subsequent target detection and tracking; for each time step , from 0 to T-1, with Iterate for the step size, at each time step , according to equation Calculating noise parameters , noise parameters Will be used to control the intensity of random noise, using the extracted features and the current random noise , obtain the predicted information through the model, including the predicted box position , Noise Frame , Category Prediction , box coordinates and the correlation score , for the predicted box position and noise box , a box update operation is performed to optimize the position and size of the box. This process is repeated until all time steps have been processed. After all iterative operations are completed, the final prediction result is filtered and optimized through the non-maximum suppression operation to obtain the final prediction nms.

[0013] Optionally, in S4, an innovative target association strategy is introduced to solve the target occlusion problem, specifically including: The target association process adopts the JDT paradigm. First, the paired detection boxes are filtered according to the association scores obtained by the detector. Then, the detection boxes are divided into high confidence and low confidence for separate tracking matching. Traverse the video sequence, for each pair of consecutive frames in the video sequence V , perform the following operations, detection and association, and obtain the detection results of the current paired frame from the consistent trajectory CT ,According to the association score threshold, the detection boxes with high association scores are screened out, and classified according to their confidence levels to perform target association; The first target association uses the intersection-over-union (IoU) similarity metric to match the non-lost tracking trajectory with the high-confidence detection frame. The unmatched trajectory and detection frame are saved in Utrack and Udetection, respectively. The second target association is to match the lost trajectory with the unmatched high-confidence detection frame. The third target association is to match the unmatched trajectory with the low-confidence detection frame. The fourth target association is to match the inactivated trajectory with the unmatched detection frame. Finally, the tracking status is updated. The status of the trajectory is updated according to the matching result. For trajectories that exceed the upper limit of the number of adjacent frames before loss, they are marked as lost, and then the result is returned. The status of the tracked targets in each frame is returned, including activated trajectories, inactivated trajectories, lost trajectories, and removed trajectories.

[0014] In a second aspect, the present application also provides a multi-target tracking device, which is configured to execute the above-mentioned multi-target tracking method.

[0015] In a third aspect, the present application also provides a multi-target tracking system, which is configured to implement the multi-target tracking method described above.

[0016] Beneficial technical effects of this application: The present application provides a multi-target tracking method, device and tracking system, which adopts a joint detection and tracking framework to transform target detection and association tasks into a denoising diffusion process, significantly improving the model's robustness to noise and reducing the frequency of ID switching. This progressive denoising diffusion strategy greatly improves the robustness of the tracker. In the training phase, paired object boxes in two adjacent frames are diffused from GT boxes to random distributions, and then the model learns detection and tracking by reversing this process. In reasoning, the model refines the randomly generated boxes into detection and tracking results through minimum denoising steps. Consistency tracking also introduces an innovative target association strategy to solve the problem of target occlusion. Experimental results show that on the MOT17 and DanceTrack datasets, consistency tracking surpasses other existing methods in multiple performance indicators, especially in terms of reasoning speed and tracking accuracy. In addition, the reasoning mechanism of consistency tracking allows the model to balance accuracy and algorithm efficiency by adjusting the number of sampling steps and the number of proposed boxes in different situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram showing the average accuracy comparison of a multi-target tracking method provided in this application; Figure 2 A flowchart of the consistency tracking algorithm provided for this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0020] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0021] It should be understood that in the present application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0022] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0023] The definitions of symbols in the formulas of the embodiments of the present application are explained as follows: character represents a random time step in the range [0,T]; Indicates the current time step; represents the total time step; Indicates the sampling step time interval; represents the kth frame in the video; Indicates the time interval of the selected frame; Indicates the total number of frames in the video; represents the RoI feature at frame k; Represents a custom output query; Represents a custom output query; Represents an object query; Represents the x / y axis coordinates of the center point of the i-th box; Indicates the width / height of the i-th box; represents the i-th box of the k-th frame ; represents the parameters in the denoiser at the tth time step; represents the model parameters; Represents a designed free-form deep neural network; Represents the split function; Represents batch matrix multiplication functionality; represents a fully connected layer; represents normal distribution; Represents the final answer of consistency tracking; Indicated by Parameterized prediction function; Indicates the calculation coefficient ; represents a positive weighting function; Represents the total loss function during the training phase; represents Focal / L1 / GIoU3d loss function; Three-dimensional generalized intersection on joint networks; Represents the weight of Focal / L1 / GIoU3d loss items; Indicates the maximum and minimum thresholds of noise parameters; Indicates that the noise parameter is between the maximum and minimum thresholds; represents randomly generated Gaussian noise; represents the random noise at the t-th time step in the sampling; Indicates the batch size of samples processed simultaneously; indicates the number of regions analyzed in each sample; Indicates feature size; Represents the generation of random noise in a given dimension; represents image feature extraction based on backbone network; Represents the prediction of the Consistency Track at each time step; Represents non-maximum suppression (NMS) operation; Indicates the NMS operation threshold; Indicates the box update operation threshold; Decoder representing consistency tracking with head network; represents the connection function; Represents the estimation function of the association score of each target; a digital counter representing the number of adjacent untracked frames for each target; represents the number of sampling steps; represents the total number of boxes used in the training phase; Indicates the number of times the previous box is repeated; represents the total number of boxes proposed in reasoning; Indicates the number of currently proposed boxes; Indicates the fill box information at the origin of the time axis; Represents the noise box information at the tth time step; Represents the predicted box information in each time step; Represents the prediction box information at the origin of the time axis; Represents the predicted box coordinates / category / association score; Represents the average precision.

[0024] refer to Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multi-target tracking method for real-time tracking of multiple targets in a video sequence, and the multi-target tracking method includes: S1. Build a denoising diffusion model based on the joint detection and tracking framework. In the forward diffusion process, extract data points from the real distribution , Gaussian noise is gradually introduced in T steps; in the reverse diffusion process, random noise Through modeling Denoising, removes noise and enhances the signal for the target distribution, and uses a denoising diffusion model to associate the detection results for object tracking.

[0025] Specifically, according to the joint detection and tracking (JDT) framework, a denoising diffusion model is constructed. During the forward diffusion process, data points are extracted from the true distribution. , Gaussian noise is gradually introduced in T steps, and during the reverse diffusion process, random noise Through modeling Denoising, removes noise and enhances the signal for the target distribution, and uses a denoising diffusion model to associate the detection results for object tracking.

[0026] S2, the training phase diffuses the paired target boxes in two adjacent frames from the true box to a random distribution, and then the model learning phase performs detection and tracking by reversing the process; S3. In the inference phase, the model refines the randomly generated boxes into detection and tracking results through a minimum denoising step; S4. Introduce innovative target association strategies to solve the target occlusion problem in order to obtain multi-target tracking results.

[0027] In the embodiment of the present application, in S1, the denoising diffusion model adopts a diffusion process algorithm to minimize the noise interference in the input data, specifically including: Starting with random noise and gradually refining it until the random noise closely matches a sample in the target distribution, during forward diffusion, the following iterative process is introduced: ,in, is the current time step In the back-diffusion process, at each backward step t, the conditional probability distribution is approximated by a network , using the time step and the previous output As input, we represent: ; in and , through iterative operations, the noise is gradually reduced in the current state, and finally When the original time step of is less than the preset threshold, make it equal to the predicted box position , that is, finally close to the sample The original time step is close to a real data .

[0028] In the embodiment of the present application, in S1, the denoising diffusion model studies two economical and effective methods to implement boundary conditions within the framework of the consistency model, specifically including: let Represents a system with input The first approach directly parameterizes the consistency model as a free-form deep neural network. ,if ,in is the range Integer within ; The second approach is to parameterize the consistency model by incorporating residual connections, which can be formalized as: ,in and is a differentiable function that satisfies and By using this structure, such as are all differentiable, and the consistency model is becomes differentiable.

[0029] In this application, in S1, the denoising diffusion model includes a feature extraction background and a data association denoising head, wherein the data association denoising head uses features as conditions instead of the original image to gradually refine the paired association box prediction, specifically including: Consistency tracking consists of two main components: feature extraction backbone and data association denoising head. The feature extraction backbone is used to process two adjacent input images. To extract deep feature representations, the data association denoising head uses these deep features as conditions, rather than the original image, to progressively refine the pairwise association box predictions from the pairs of noise boxes. In the setting, the data samples consist of paired boundary boxes. Composition, among which, , neural network for Trained to learn from pairs of noisy boxes Medium Forecast , provided that the corresponding two adjacent images , and accordingly generate two corresponding associated confidence scores S; if , then the MOT task degenerates into an object detection problem. This consistent property allows consistent tracking to solve both object detection and tracking tasks. It should be noted that Set to 1 only during the trace matching process.

[0030] Specifically, we use a pre-trained convolutional neural network as the feature extraction background network to extract features from consecutive frames in the video sequence, and use the feature pyramid network (FPN) to enhance the multi-scale expression ability of the features. The training sample of MOT is a set of input-target ,in is the input image, and They are the bounding box and ID information of the object in the kth frame video, and in the set In the representation, the i-th box k is ,in are the center coordinates of the bounding box, are the width and height of the bounding box, are identification numbers, especially when Missing If there are objects, then , a feature extraction background network is used to extract deep feature representation from two adjacent video frames: the backbone adopts YOLOX, which uses the feature pyramid network (FPN) to extract high-level features from two adjacent frames, and then inputs these features into the diffusion head for the denoising process of conditional data association.

[0031] In this application, in S1, the denoising diffusion model includes a space-time fusion module for realizing the exchange of temporal information between paired frames in two consecutive frames to facilitate complete data association, and further includes: The STF module is proposed to achieve the exchange of temporal information between paired boxes of two consecutive frames, facilitating complete data association, and giving ROI characteristics for two consecutive time steps. ,in is the batch size, is the number of regions, is the feature dimension, output query , then proceed with the following transformation: Linear transformation and segmentation: Each query First, we transform it using linear projection, and then split the result into two separate tensors of equal dimension. ; Batch Matrix Multiplication: RoI features from two time steps are concatenated and then two consecutive batch matrix multiplications (BMM) are performed. ; Final linear transformation: The feature tensor obtained by the batch matrix multiplication (BMM) operation is further processed to generate the object query of the current block ; Index relationship: Index Taken from adjacent time pairs , indicating that the operation takes into account the transformation in the forward and backward directions between consecutive time steps. In addition to the box head and the classification head, an additional association score head is introduced. This head utilizes the fused features of the paired boxes obtained from the spatiotemporal fusion module and inputs them into a linear layer. The output of this head provides a confidence score for data association. In the subsequent post-processing of NMS, it is determined whether the paired box outputs belong to the same target.

[0032] In the present application, in S2, the training phase diffuses the paired target frames in two adjacent frames from the real frames to the random distribution, and then the model learning phase detects and tracks through the reversal process, specifically including: The algorithm randomly selects a pair of frames from a video sequence as the input of the model. In the open source benchmark of MOT, the number of annotated instances between different images usually varies. To address this inconsistency, a filling strategy is implemented by introducing auxiliary boxes around the GT box. First, the GT box in the image is supplemented to the total number ; Then, based on the noise addition strategy of the consistency model, random noise is added to the original GT boxes of the two frames, and all these noisy boxes are input into the model for denoising; Finally, the model extracts the association relationship between the instance boxes in the two adjacent frames, calculates the loss, and performs back propagation operations. These filled instances are represented as , represents the original filling sample, for the i-th GT box, it is expressed as , applying Gaussian noise to its four parameters , at a randomly selected time step t, "box damage phenomenon" usually refers to the process of target detection or tracking, in which the calibrated bounding box loses accuracy or integrity due to target occlusion, blur or rapid motion. This situation may cause the algorithm to fail to correctly identify or track the target, thereby affecting the performance of the system. At the tth moment, the range of the box affected by the noise is strictly constrained. This process is part of simulating noise interference and performing data enhancement, which aims to improve the generalization ability and robustness of the training model. The following describes how noise is added to the original GT box. The range of the noise box at the tth time step is restricted. The scaling factor of the noise is determined as follows: ; Then, the original filled samples Introducing noise: , in, is a randomly generated Gaussian noise. Finally, the range of the noise box is constrained by: ; Among them represents the scale factor of the noise box and is defined as: ; The noise box scaling factor ensures that the noise scaling factor is appropriately adjusted across time steps and that the noise box remains within the specified range of values.

[0033] Specifically, the pre-trained YOLOX detector of the diffusion track is used to perform consistency training on the training sets of the MOT17 and DanceTrack training tracks. For MOT17, the training plan contains 40 rounds of tracking on the combined dataset (including MOT17, CrowdHuman, Cityperson and ETHZ), and 40 rounds of tracking on the training set only on MOT17. For DanceTrack, no additional training data is used, and the model is trained for 60 rounds. In the detection and tracking training stages, data enhancement techniques such as Mosaix and Mixup are also used. Each training sample is directly sampled in each video with a frame interval of The input image is resized to 1440×800. The AdamW optimizer is used with an initial learning rate of 1e-4, which is reduced by a cosine function and a final reduction factor of 0.1. In the first stage, a pre-trained learning rate of 2.5e-5 is used as the warm-up learning rate, and another pre-trained weight is 0.2. The model is trained on an NVIDIA GeForce RTX A100 GPU with FP32 precision. All experiments have a constant seed, and the mini-batch size is set to 3. Each GPU hosts two batches with ,This method is implemented in Python 3.8 using PyTorch 1.10; Define loss functions, including classification loss, bounding box regression loss, and consistency loss, to optimize model performance; Noise ensures that the noise scale factor is properly adjusted in each time step and that the noise box remains within the specified value range; The loss function used to evaluate the predicted bounding box follows the framework established by diffusion tracking, which contains two Loss and loss The former represents the standard L1 loss term, while the latter represents the generalized intersection over union (GIoU) ​​loss. The definition of GIoU is extended to make it compatible with the design of paired boxes. 3D GIoU and 3D IoU are volumetric extensions of the original region-based loss. In addition, the focal loss Used to evaluate the classification of each predicted bounding box, in order to balance the relative influence of each loss component, a positive real-valued weight is assigned to each loss term, so the total loss function at the tth time step is expressed as follows: and in and are the square setbacks consisting of the estimated detection box and the ground truth bounding box of the same object in two adjacent frames, respectively; The 3D GIoU of paired prediction boxes is defined as: ; in Represents the detection box containing the estimate The minimum convex hull of and the bounding box of the ground truth In the framework , and Similar to the above equation, the intersection point A square setback is also formed, including overlapping and , using the self-consistency characteristics of the consistency model, perturbations and examples The associated bounding boxes are computed at consecutive time steps t-1 and t ; In this application, in S3, in the inference stage, the model refines the randomly generated boxes into detection and tracking results through a minimum denoising step, further including: According to consistency inference, the first input is an image In paired frames , total time step T,t, number of sampling steps , the output is the final prediction Calculate time step , which is the total time step T divided by the number of sampling steps Next, generate a random noise that matches the size of the preset box number , these boxes are the regions in the image used to detect and track objects, and then, from the input pairs of frames These features will be used for subsequent target detection and tracking. For each time step t, from 0 to T-1, Iterate for the step size, at each time step t, according to Eq. Calculating noise parameters , noise parameters Will be used to control the intensity of random noise, using the extracted features and the current random noise , obtain the predicted information through the model, including (predicted box position), (noise frame), (category prediction), (box coordinates) and (association score), for the predicted box position and noise box , perform a box update operation to optimize the position and size of the box. This process will be repeated until all time steps are processed. After all iterative operations are completed, the final prediction result is filtered and optimized through the non-maximum suppression operation to obtain the final prediction nms.

[0034] Specifically, in the inference phase, the model receives new video pairs of frames and performs object detection through feature extraction and denoising diffusion modules. First, the previous frame is repeated with Gaussian filling until a predefined number of frames is reached. has arrived, then, according to Add Gaussian noise to the input frame . exist Finally, the tracking result is obtained through a denoising process with a consistency model sampling step.

[0035] Specifically, post-processing techniques such as non-maximum suppression are applied to optimize the final detection and tracking results. At this stage, the model processes the video frames and generates a series of predicted bounding boxes, which may overlap or contain multiple bounding boxes of the same target. In order to improve the accuracy of detection and the stability of tracking, NMS is used to perform the following operations: Select candidate bounding boxes: Select candidate boxes from all bounding boxes generated by the model, which have sufficiently high confidence scores. Calculate the intersection-over-union ratio: Calculate the intersection-over-union ratio of each candidate box with other boxes. Suppress non-maximum boxes: Keep the bounding box with the highest confidence score and suppress (i.e., delete) other overlapping bounding boxes with high IoU values. In this way, NMS ensures that each target is represented by only one most accurate bounding box, thereby optimizing the final detection and tracking results. NMS is a commonly used post-processing technique, which plays an important role in target detection and tracking systems to ensure the quality of the output results. In multi-target tracking systems, NMS helps reduce redundancy and erroneous tracking and improve overall tracking performance.

[0036] In the present application, in S4, according to introducing an innovative target association strategy to solve the target occlusion problem, further comprising: The target association process adopts the JDT paradigm, and the entire target matching process no longer involves additional feature matching. First, the paired detection frames are filtered according to the association scores obtained by the detector, and only the detection frames with higher association scores are retained. This is a specific manifestation of the JDT paradigm. Subsequently, the detection frames are divided into high-confidence and low-confidence for separate tracking and matching to solve potential occlusion problems. A simple Kalman filter is used to re-associate lost targets. Input: video sequence V, single trajectory m, consistent trajectory CT, and various scoring thresholds and parameters, including association scoring threshold, detection scoring threshold, tracking scoring threshold, number of associated boxes Na, and upper limit of adjacent frames before loss, etc. The output is the state of the tracked target in each frame. The processing flow is to traverse the video sequence, and for each pair of consecutive frames in the video sequence V , perform the following operations, detection and association, and obtain the detection results of the current paired frame from the consistent trajectory CT , According to the association score threshold, the detection boxes with high association scores are filtered out, and classified according to their confidence to perform target association. The first target association uses the IoU (intersection over union) similarity metric to match the non-lost tracking trajectory with the high-confidence detection box. The unmatched trajectory and detection box are saved in Utrack and Udetection respectively. The second target association is to match the lost trajectory with the unmatched high-confidence detection box. The third target association is to match the unmatched trajectory with the low-confidence detection box. The fourth target association is to match the inactivated trajectory with the unmatched detection box. Finally, the tracking status is updated. The status of the trajectory is updated according to the matching result, including activation, loss and removal. For the trajectory that exceeds the upper limit of the number of adjacent frames before loss, it is marked as lost, and then the result is returned. The status of the tracked target in each frame is returned, including activated trajectory, inactivated trajectory, lost trajectory and removed trajectory.

[0037] An embodiment of the present application also provides a multi-target tracking device, which is configured to execute any of the methods described above.

[0038] The embodiment of the present application also provides a multi-target tracking system, which implements any of the methods described above. The tracking accuracy and robustness of the model are evaluated on standard datasets, such as MOT17 and DanceTrack datasets, and the tracking results are recorded and analyzed, including the number of successfully tracked targets, the number of identity switches, and the tracking accuracy. The effectiveness of the invention is verified through a series of experiments, including tracking performance on video sequences of different complexities, and comparison with the prior art.

[0039] The pre-trained YOLOX detector of the diffusion track is used to perform consistency training on the training sets of the MOT17 and DanceTrack datasets. For MOT17, the training plan contains 40 tracking on the combined dataset (including MOT17, CrowdHuman, Cityperson and ETHZ), and 40 rounds of training only on MOT17. For the DanceTrack dataset, no additional training data is used. The model is trained for 60 rounds. In the detection and tracking training stage, data enhancement techniques such as Mosaix and Mixup are also used. Each training sample is directly sampled in each video with a frame interval of The input image was resized to 1440×800. The AdamW optimizer was used with an initial learning rate of 1e-4, which was decreased according to the cosine function and a final decrease factor of 0.1. A warm-up learning rate of 2.5e-5 was used in the first stage and a warm-up factor of 0.2 was used in the other stage. The model was trained on one NVIDIA GeForce RTX A100 GPU with FP32 precision. All experiments had a constant seed, and the mini-batch size was set to 3. Each GPU hosted two batches with .

[0040] The core feature of consistent tracking is its self-consistency, which ensures that the mapping effect from any point along the time axis to the origin remains relatively stable. This stability means that once the model is fully trained, it allows flexible adjustment of the number of sampling steps. ,Consistency tracking can improve its detection accuracy by appropriately increasing the sampling time step.,Therefore, in different situations, the model can adjust the sampling step according to specific requirements,,therefore balancing the accuracy and efficiency of the algorithm.,In addition, noise enhancement during training significantly improves the,model's robustness to noise.

[0041] Robustness to Detection Perturbations,To rigorously evaluate the robustness of consistency tracking to noise,in the detection stage, we independently trained and tested its,detection component,ConsistencyDet,on the MS-COCO dataset.,This evaluation is benchmarked against leading detectors,such as DiffusionDet and DETR. Figure 2 As shown, the performance related to the incremental inclusion of the bounding box is significantly improved, achieving stability in , the peak value is On the contrary, DETR It reaches its maximum AP at , and then experiences a sharp decline, dropping significantly to 26.4% at , a decrease of 12.4% from its best AP of 38.8%. Although DiffusionDet also improves by considering more boxes, its performance always lags behind ConsistencyDet. Therefore, ConsistencyDet is not only more robust to noise, but also shows superior transferability and generality in different scenarios involving variable object counts.

[0042] like Figure 2As shown in Figure 1, once the model is trained, it can be used by changing the number of boxes and the sampling time step during inference. Therefore, a consistent tracker can be deployed in multiple scenarios to achieve the desired speed-accuracy trade-off without retraining the network. Accuracy control is achieved by increasing n. Consistent tracking is evaluated with 1000, 2000, and 4000 proposed boxes. The results show that the MOTA value of consistent tracking can be achieved by increasing the number of random noise boxes. In addition, when , which is consistent with the one-step mapping feature of the consistency model; when , the MOTA metric remains in an oscillating state, but its peak value is lower than .

[0043] The present application provides a multi-target tracking method, device and tracking system, which adopts a joint detection and tracking framework to transform target detection and association tasks into a denoising diffusion process, significantly improving the model's robustness to noise and reducing the frequency of ID switching. This progressive denoising diffusion strategy greatly improves the robustness of the tracker. In the training phase, paired object boxes in two adjacent frames are diffused from GT boxes to random distributions, and then the model learns detection and tracking by reversing this process. In reasoning, the model refines the randomly generated boxes into detection and tracking results through minimum denoising steps. Consistency tracking also introduces an innovative target association strategy to solve the problem of target occlusion. Experimental results show that on the MOT17 and DanceTrack datasets, consistency tracking surpasses other existing methods in multiple performance indicators, especially in terms of reasoning speed and tracking accuracy. In addition, the reasoning mechanism of consistency tracking allows the model to balance accuracy and algorithm efficiency by adjusting the number of sampling steps and the number of proposed boxes in different situations.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-target tracking method, characterized in that: Used for real-time tracking of multiple targets in a video sequence, the multi-target tracking method comprises: S1. Build a denoising diffusion model based on the joint detection and tracking framework. In the forward diffusion process, extract data points from the real distribution , Gaussian noise is gradually introduced in T steps; in the reverse diffusion process, random noise Through modeling Denoising, removing noise from the target distribution and enhancing the signal, using a denoising diffusion model to correlate detection results for object tracking; S2, the training phase diffuses the paired target boxes in two adjacent frames from the true box to a random distribution, and then the model learning phase performs detection and tracking by reversing the process; S3. In the inference phase, the model refines the randomly generated boxes into detection and tracking results through a minimum denoising step; S4. Introduce innovative target association strategies to solve the target occlusion problem in order to obtain multi-target tracking results.

2. A multi-target tracking method according to claim 1, characterized in that: In S1, the denoising diffusion model uses a diffusion process algorithm to minimize the noise interference in the input data, specifically including: Starting from random noise and gradually refining it until the random noise matches a sample in the target distribution, the following iterative process is introduced during forward diffusion: ； in, is the current time step In the backward diffusion process, at each backward step t, the conditional probability distribution is represented by a network , using the time step and the previous output As input, we represent: ； in, and , through iterative operations, the noise is gradually reduced in the current state, and finally When the original time step of is less than the preset threshold, make it equal to the predicted box position .

3. A multi-target tracking method according to claim 1, characterized in that: In S1, the denoising diffusion model uses two methods to implement boundary conditions within the framework of the consistency model, specifically including: let Represents a system with input The first approach directly parameterizes the consistency model, where , ,in, is the range Integer within ; The second approach is to parameterize the consistency model by incorporating residual connections, which can be formalized as: ,in, and is a differentiable function that satisfies and .

4. A multi-target tracking method according to claim 1, characterized in that: In S1, the denoising diffusion model includes a feature extraction background and a data association denoising head, wherein the data association denoising head uses features as conditions instead of the original image to gradually refine the paired association box prediction, specifically including: Consistency tracking consists of two components: a feature extraction backbone and a data association denoising head. The feature extraction backbone processes two adjacent input images. To extract deep feature representation, the data association denoising head uses features as conditions instead of the original image, and gradually refines the paired association box predictions from the paired noise boxes. The data samples are composed of paired boundary Composition, of which , neural network for is trained to extract the noise from pairs of frames Medium Forecast , provided that the corresponding two adjacent images , and accordingly generate two corresponding associated confidence scores S; if , then the MOT task degenerates into an object detection problem. The consistent properties allow consistent tracking to solve both object detection and tracking tasks. Set to 1 only during the trace matching process.

5. A multi-target tracking method according to claim 1, characterized in that: In S1, the denoising diffusion model includes a space-time fusion module for realizing the exchange of temporal information between paired frames in two consecutive frames to facilitate complete data association, specifically including: The STF module is proposed to achieve the exchange of temporal information between paired boxes of two consecutive frames, facilitating complete data association, given the ROI characteristics of two consecutive time steps. ,in, is the batch size, is the number of regions, is the feature dimension, output query , then proceed with the following transformation: Linear transformation and segmentation: Each query , first transform using linear projection, then split the result into two separate tensors of equal dimension ; Batch matrix multiplication: RoI features from two time steps are concatenated, and then two consecutive batch matrix multiplications are performed to obtain the feature tensor ; Final linear transformation: The feature tensor obtained by batch matrix multiplication is further processed to generate the object query of the current block .

6. A multi-target tracking method according to claim 1, characterized in that: In S2, the training phase diffuses the paired target boxes in two adjacent frames from the real boxes to the random distribution, and then the model learning phase performs detection and tracking through the reversal process, which specifically includes: A pair of frames are randomly selected from the video sequence as the input of the model, and the filling strategy is implemented by introducing auxiliary boxes around the GT boxes. First, the GT boxes in the image are supplemented to the total number ; Then, based on the noise addition strategy of the consistency model, random noise is added to the original GT frames of the two frames, and all noisy frames are input into the model for denoising; Finally, the model extracts the association relationship between the instance frames in the two adjacent frames, calculates the loss, and performs back propagation operations. The filled instance is represented as , for the i-th GT box, it is expressed as , applying Gaussian noise to its four parameters , at a randomly selected time step t, the noise box is restricted in scope at the tth time step, and the scaling factor of the noise is determined as follows: ； Then, the original filled samples Introducing noise: ,in, is a randomly generated Gaussian noise. Finally, the range of the noise box is constrained by: ; Among them, represents the scale factor of the noise box, .

7. A multi-target tracking method according to claim 1, characterized in that: In S3, in the inference phase, the model refines the randomly generated boxes into detection and tracking results through the minimum denoising step, specifically including: According to consistency inference, the first input is an image In paired frames , total time step T,t, number of sampling steps ; Output is the final prediction , calculate the time step is the total time step T divided by the number of sampling steps Next, generate a random noise that matches the size of the preset box number , then, from the input pairs of frames The features extracted will be used for subsequent target detection and tracking; for each time step , from 0 to T-1, with Iterate for the step size, at each time step , according to equation Calculating noise parameters , noise parameters Will be used to control the intensity of random noise, using the extracted features and the current random noise , obtain the predicted information through the model, including the predicted box position , Noise Frame , Category Prediction , box coordinates and the correlation score , for the predicted box position and noise box , a box update operation is performed to optimize the position and size of the box. This process is repeated until all time steps have been processed. After all iterative operations are completed, the final prediction result is filtered and optimized through the non-maximum suppression operation to obtain the final prediction nms.

8. A multi-target tracking method according to claim 1, characterized in that: In S4, an innovative target association strategy is introduced to solve the target occlusion problem, specifically including: The target association process adopts the JDT paradigm. First, the paired detection boxes are filtered according to the association scores obtained by the detector. Then, the detection boxes are divided into high confidence and low confidence for separate tracking matching. Traverse the video sequence, for each pair of consecutive frames in the video sequence V , perform the following operations, detection and association, and obtain the detection results of the current paired frame from the consistent trajectory CT ,According to the association score threshold, the detection boxes with high association scores are screened out, and classified according to their confidence levels to perform target association; The first target association uses the intersection-over-union (IoU) similarity metric to match the non-lost tracking trajectory with the high-confidence detection frame. The unmatched trajectory and detection frame are saved in Utrack and Udetection, respectively. The second target association is to match the lost trajectory with the unmatched high-confidence detection frame. The third target association is to match the unmatched trajectory with the low-confidence detection frame. The fourth target association is to match the inactivated trajectory with the unmatched detection frame. Finally, the tracking status is updated. The status of the trajectory is updated according to the matching result. For trajectories that exceed the upper limit of the number of adjacent frames before loss, they are marked as lost, and then the result is returned. The status of the tracked targets in each frame is returned, including activated trajectories, inactivated trajectories, lost trajectories, and removed trajectories.

9. A multi-target tracking device, characterized in that: The multi-target tracking device is configured to execute the multi-target tracking method according to any one of claims 1 to 8.

10. A multi-target tracking system, characterized in that: The multi-target tracking system is configured to implement the multi-target tracking method according to any one of claims 1 to 8.

Images