A multi-target tracking method with diffusion model empowered tracking proposal propagation
Generate random and tracked proposals in multi-objective tracking through diffusion model and ProDiffuse encoder, solving the problem of insufficient negative samples, improving tracking performance and robustness, and achieving more accurate multi-objective tracking.
Patent Information
- Application Number
- CN202410024017.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-01-05
AI Technical Summary
With limited negative samples in the dataset, existing multi-objective tracking algorithms are difficult to effectively distinguish the appearance and position differences of tracking objects, resulting in poor tracking performance.
The method of giving tracking proposal propagation is adopted to generate random proposals and tracked target proposals through the denoising diffusion module and the ProDiffuse encoder decoder, and iterative optimization is performed in combination with the DDIM model to improve the bounding box identification and trajectory prediction of tracking objects.
By improving randomly generated noisy proposals and self-conditional proposal propagation, the detection and monitoring capabilities of tracked objects are improved, noise samples are increased to simulate the number of negative samples, and the performance of multi-objective tracking is improved.
Smart Images

Figure CN117893570B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a multi-target tracking method and device with a diffusion model endowed with tracking proposal propagation. Background Art
[0002] The research goal of multi-target tracking algorithms in the field of computer vision is to develop a method that can automatically identify and track multiple targets in a video or image. Its main goals include the following aspects: Target detection and recognition: The primary task of the multi-target tracking algorithm is to accurately detect and identify multiple targets in the input image or video. This includes the acquisition of the target's location, bounding box, appearance features, and category information; Target tracking: The algorithm needs to be able to track the movement of the target in consecutive images or video frames. This involves the consistency and continuity of the target in time. The algorithm needs to predict and update the position of the target in the current frame based on the position information of the target in the previous frame; Robustness and real-time performance: The multi-target tracking algorithm needs to be robust and able to cope with complex situations in various scenes, such as lighting changes, background clutter, target deformation, etc. At the same time, the algorithm also needs to have real-time performance and be able to perform efficient target tracking in real-time video streams.
[0003] The purpose of studying multi-target tracking algorithms is to improve the computer's ability to understand and analyze multiple targets in videos or images, and to provide more accurate and efficient target detection and tracking technologies for various application fields (such as video surveillance, autonomous driving, intelligent transportation, etc.).
[0004] Most of the existing efficient multi-target tracking algorithms are divided into two modes, namely TBD (tracking by detection) mode and JDT (joint detection and tracking) mode. The TBD mode detects the target first and then associates the target information in the time dimension to achieve target tracking. The mainstream TBD mode algorithms include SORT, DeepSORT, ByteTrack, etc., while the JDT mode detects the target, extracts and associates the target identity information in one model at the same time. The more mainstream JDT mode algorithms include JDE, TraDes, MOTR, etc.
[0005] The Tracking by Detection (TBD) paradigm treats multi-object tracking as two independent tasks: object detection and object tracking. First, high-quality detection boxes are generated using an effective detector, and then these boxes are processed by an association model, taking into account appearance and motion cues. The association process exploits pairwise appearance re-identification (Re-ID) similarities for appearance matching and adopts various heuristics such as Intersection over Union (IoU) or Kalman filtering for motion-based association. Although the TBD paradigm has been widely used for multi-object tracking, its two-stage framework hinders end-to-end optimization during training.
[0006] In contrast, the Joint Detection and Tracking (JDT) paradigm aims to address this limitation by achieving detection and association of multiple objects in a single stage. However, JDT-based methods, such as MOTR and TraDeS, face challenges in optimization due to the inherent competition in appearance learning, where detection increases the inter-class variance during the learning process, while ReID increases the intra-class variance, resulting in poor tracking performance.
[0007] End-to-end MOTR (Multiple Object Tracking with Transformers) is a multi-object tracking algorithm based on the Transformer architecture, which has achieved significant breakthroughs in the field of computer vision. The MOTR algorithm combines the tasks of object detection and object tracking, and uses the Transformer model to achieve end-to-end multi-object tracking. By introducing the self-attention mechanism in the Transformer encoder, the MOTR algorithm is able to associate and track objects. Each object is regarded as a separate object in each frame, and is tracked by calculating its similarity with other objects. Through the self-attention mechanism, the algorithm can capture the association information and contextual dependencies between objects, thereby achieving accurate multi-object tracking. MOTR introduces the Transformer encoder to capture the relationship and dynamic characteristics between objects, improving the accuracy of multi-object tracking.
[0008] To alleviate the challenges associated with competitive appearance learning, some methods such as FairMOT have made fine adjustments to the network design, and CSTrack introduced a reciprocal network (REN) with self-relations and cross-relations to encourage better task-related representations in each branch. Although these methods alleviate the competition in appearance learning through network design, this application proposes that multi-target tracking essentially involves a contrastive learning process between tracked objects. In some cases, the tracked objects may be at a close distance, so it is crucial to distinguish their appearance features. In other cases, objects that look similar may be far apart, so it is necessary to distinguish their location information. Therefore, having abundant negative samples in existing datasets is crucial to distinguish the appearance and location differences of the tracked objects. However, this poses a challenge due to the limited availability of negative samples in existing datasets. Summary of the invention
[0009] In order to solve the technical problem of limited availability of negative samples in existing data sets in the prior art, an embodiment of the present invention provides a multi-target tracking method and device with a diffusion model to propagate tracking proposals. The technical solution is as follows:
[0010] On the one hand, a multi-target tracking method with a diffusion model assigned to tracking proposal propagation is provided, the method is implemented by a multi-target tracking device with a diffusion model assigned to tracking proposal propagation, the method comprising:
[0011] S1: Acquire a continuous set of images or videos;
[0012] S2: Inputting each frame of the continuous image set or video into a pre-trained multi-target tracking model in sequence, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder;
[0013] S3: Using the backbone network and the ProDiffuse encoder, image features are extracted for each frame of the image to obtain image features;
[0014] S4: using the denoising diffusion module, passing each frame of the image through a random proposal sequence generation mechanism in turn, generating a proposal query, and obtaining a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query, and a tracked target instance query;
[0015] S5: For each frame of the image, the result of the proposed query and the image features are input into the ProDiffuse decoder, the tracking target of each frame is identified, and the bounding box corresponding to the tracking target is obtained. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame;
[0016] S6: Post-process the result of the proposed query to obtain a trajectory corresponding to the tracking target.
[0017] Preferably, before inputting the continuous image set or each frame of the video into the pre-trained multi-target tracking model in sequence in S01, the method further comprises:
[0018] S01: Obtain a multi-target tracking model to be trained;
[0019] The step S01 of obtaining a multi-target tracking model to be trained includes:
[0020] S011: constructing a denoising diffusion module, wherein the input of the denoising diffusion module is a continuous image set or each frame image of a video;
[0021] S012: Initialize a backbone network based on the ResNet-50 network model, wherein the input of the backbone network is a continuous set of images or each frame of a video;
[0022] S013: construct a ProDiffuse encoder based on the Transformer model, wherein the input of the ProDiffuse encoder is the output of the backbone network;
[0023] S014: constructing a ProDiffuse decoder based on the Transformer model, wherein the input of the ProDiffuse decoder is the output of the ProDiffuse encoder, the output of the denoising diffusion module, and the output of the ProDiffuse decoder of the previous frame;
[0024] S015: Combine the denoising diffusion module, backbone network, ProDiffuse encoder and ProDiffuse decoder into a multi-target tracking model.
[0025] Preferably, before inputting the continuous image set or each frame of the video into the pre-trained multi-target tracking model in sequence in S02, the method further includes:
[0026] S02: training the multi-target tracking model to be trained to obtain a pre-trained multi-target tracking model;
[0027] The step S02 of training the multi-target tracking model to be trained to obtain a pre-trained multi-target tracking model includes:
[0028] S021:,obtaining a continuous set of images or videos, and annotating each frame in the continuous set of images or videos to obtain training data, wherein the annotation includes identifying the tracking target and assigning an ID label, and then annotating a real bounding box of the tracking target;
[0029] S022: Using the backbone network, extract image features from each frame to obtain the first image features;
[0030] S023: inputting the first image feature into a ProDiffuse encoder to obtain a second image feature;
[0031] S024: Obtain a true bounding box, generate a random proposal query based on the true bounding box, and obtain a result of the random proposal query, wherein the random proposal query includes using a denoising diffusion module to sample a random boundary from a Gaussian distribution and add Gaussian noise to obtain a noise bounding box;
[0032] S025: performing a tracked target proposal query based on at least two frames of images, extracting a trajectory corresponding to the tracked target, and obtaining a result of the tracked target proposal query;
[0033] S026: passing the second image feature, the result of the random proposal query and the result of the tracked target proposal query to the ProDiffuse encoder, and outputting the tracked target trajectory and the corresponding bounding box;
[0034] S027: Based on the selected loss function, the multi-target tracking model to be trained is iteratively optimized and trained to obtain a pre-trained multi-target tracking model, wherein the iterative optimization training is to connect the random proposal query with the tracked target proposal query to obtain the tracked target instance query, and the result of the tracked target instance query is input into the ProDiffuse decoder, and the tracked target instance query adopts the DDIM model for iterative optimization, and the loss function is the ensemble average loss.
[0035] Preferably, the S4 utilizes the denoising diffusion module to sequentially pass each frame of the image through a random proposal sequence generation mechanism to generate a proposal query and obtain a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query and a tracked target instance query, including:
[0036] S41: Using the denoising diffusion module, starting from the first frame, a random proposal query is performed to obtain a result of the random proposal query, where the random proposal query generates a proposal sequence based on a Gaussian process;
[0037] S42: using the denoising diffusion module, starting from the second frame, performing a tracked target proposal query to obtain a result of the tracked target proposal query, wherein the tracked target proposal query includes a position reference obtained by the tracked target proposal query of the previous frame, and the position reference is a trajectory corresponding to the tracked target;
[0038] S43: Starting from the second frame, based on the result of the random proposal query and the result of the tracked target proposal query, a tracked target instance query is performed to obtain a result of the tracked target instance query, wherein the result of the tracked target instance query includes instance information, and the instance information is a bounding box corresponding to the tracked target.
[0039] Preferably, the S41 uses the denoising diffusion module to perform a random proposal query starting from the first frame to obtain a result of the random proposal query, wherein the random proposal query generates a proposal sequence based on a Gaussian process, including:
[0040] S411: establishing a diffusion forward process, starting from the first frame, extracting the bounding box corresponding to the tracked target, and filling it to obtain a filled bounding box;
[0041] S412: additional bounding boxes generated based on Gaussian distribution;
[0042] S413: Connecting the filled bounding box with the additional bounding box to obtain a complete bounding box;
[0043] S414: Adding Gaussian noise to all the bounding boxes to obtain a noisy bounding box, wherein the scale of the Gaussian noise is controlled by a function αt, where α follows a monotonically decreasing cosine schedule in different time steps t.
[0044] Preferably, starting from the second frame, S43 performs a tracked target instance query based on the result of the random proposal query and the result of the tracked target proposal query to obtain a tracked target instance query result, wherein the tracked target instance query result includes instance information, and the instance information is a bounding box corresponding to the tracked target, including:
[0045] S431: Positionally encode the result of the random proposal query by using a ProDiffuse encoder to obtain an encoded result of the random proposal query;
[0046] S432: The encoded random proposal query result is updated through the ProDiffuse decoder, and interacts with the image features to generate a trajectory proposal, perform a tracked target proposal query, and obtain the track of the tracked target;
[0047] S433: Initialize the trajectory proposal of the current frame according to the trajectory of the tracked target in the previous frame, continue to track and adjust the trajectory, and then query the tracked target instance based on the image features.
[0048] Preferably, the step S6 of post-processing the result of the proposed query to obtain a trajectory corresponding to the tracking target includes:
[0049] S61: Integrate the DDIM model into the multi-target tracking model;
[0050] S62: Based on the result of the proposed query, the tracking target is optimized by using the DDIM iterative algorithm;
[0051] S63: fusing the motion information and the image features to obtain a motion feature representation;
[0052] S64: Using the motion feature representation as the input of the DDIM iterative algorithm to obtain a refined trajectory prediction;
[0053] S65: Generate a self-conditional proposal query based on the result of the proposal query;
[0054] S66: Fusing the self-conditional proposal query with other tracked target proposal queries;
[0055] S67: The fused proposal query is used as the input of the subsequent ProDiffuse decoder to enhance the position information of the tracked target.
[0056] On the other hand, a multi-target tracking device with a diffusion model assigned to track proposal propagation is provided, and the device is applied to a multi-target tracking method with a diffusion model assigned to track proposal propagation, and the device comprises:
[0057] Image set module: used to obtain continuous image sets or videos;
[0058] A multi-target tracking model module: used to sequentially input each frame of the continuous image set or video into a pre-trained multi-target tracking model, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder;
[0059] Feature extraction module: used to extract image features for each frame of image using the backbone network and ProDiffuse encoder to obtain image features;
[0060] Proposal query module: used to use the denoising diffusion module to pass each frame of the image through the random proposal sequence generation mechanism in turn to generate a proposal query and obtain a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query and a tracked target instance query;
[0061] Decoder module: for each frame of the image, input the result of the proposed query and the image features into the ProDiffuse decoder, identify the tracked target in each frame, and obtain the bounding box corresponding to the tracked target. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame;
[0062] Post-processing module: used to post-process the result of the proposed query to obtain the trajectory corresponding to the tracking target.
[0063] On the other hand, a multi-target tracking device with diffusion model assigned to tracking proposal propagation is provided, and the multi-target tracking device with diffusion model assigned to tracking proposal propagation includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, a method as described in any one of the multi-target tracking methods with diffusion model assigned to tracking proposal propagation is implemented.
[0064] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any of the methods described in the above-mentioned diffusion model-based multi-target tracking method for tracking proposal propagation.
[0065] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0066] First, the embodiment of the present invention constructs a denoising diffusion training process to achieve actual tracking objects by improving randomly generated noisy proposals, and propagates proposals during tracking by introducing learnable proposal offsets. Specifically, a random proposal is first generated using a Gaussian function, and the region of interest and its corresponding bounding box are extracted using a denoising diffusion process. In the subsequent tracking stage, the previously identified target is merged with the additional proposal generated using the Gaussian function. By using self-conditional proposal propagation and denoising diffusion processes, the region of interest of the current frame is obtained. In this process, proposals randomly generated by Gaussian functions can detect newly appearing objects, while existing tracking targets can be continuously monitored and tracked.
[0067] Therefore, the technical solution provided by the embodiment of the present invention first proves that multi-target tracking can be solved in a generative way; then the technical solution provided by the embodiment of the present invention designs a self-conditional propagation scheme, and the randomly generated proposals can be refined into actual object proposals and flexible proposal offsets in initialization and subsequent iterations, respectively, to achieve proposal propagation; finally, in view of the small number of negative samples in the data set, the technical solution provided by the embodiment of the present invention uses a diffusion model to simulate the number of generated negative samples; using the diffusion model to increase the noise samples, that is, the number of negative samples, helps to improve the performance of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0069] Figure 1 It is a flow chart of a multi-target tracking method with a diffusion model endowed with tracking proposal propagation provided by an embodiment of the present invention;
[0070] Figure 2 is a structural schematic diagram of a multi-target tracking model provided by an embodiment of the present invention;
[0071] Figure 3 It is a block diagram of a multi-target tracking device with a diffusion model endowed with tracking proposal propagation provided by an embodiment of the present invention;
[0072] Figure 4 It is a structural schematic diagram of a multi-target tracking device that uses a diffusion model to propagate tracking proposals provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0073] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0074] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0075] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0076] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.
[0077] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0078] The embodiment of the present invention provides a multi-target tracking method with a diffusion model assigned to the tracking proposal propagation, which can be implemented by a multi-target tracking device with a diffusion model assigned to the tracking proposal propagation, and the multi-target tracking device with a diffusion model assigned to the tracking proposal propagation can be a terminal or a server. Figure 1 The flowchart of the multi-target tracking method with the diffusion model endowed with tracking proposal propagation is shown. The processing flow of the method may include the following steps:
[0079] S1: Acquire a continuous set of images or videos;
[0080] S2: Inputting each frame of the continuous image set or video into a pre-trained multi-target tracking model in sequence, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder;
[0081] Preferably, before S2, it includes:
[0082] S01: Obtain a multi-target tracking model to be trained, including:
[0083] S011, constructing a denoising diffusion module, wherein the input of the denoising diffusion module is a continuous image set or each frame image of a video;
[0084] S012: Initialize a backbone network based on the ResNet-50 network model, wherein the input of the backbone network is a continuous set of images or each frame of a video;
[0085] S013: construct a ProDiffuse encoder based on the Transformer model, wherein the input of the ProDiffuse encoder is the output of the backbone network;
[0086] S014: constructing a ProDiffuse decoder based on the Transformer model, wherein the input of the ProDiffuse decoder is the output of the ProDiffuse encoder, the output of the denoising diffusion module, and the output of the ProDiffuse decoder of the previous frame;
[0087] S015: Combine the denoising diffusion module, backbone network, ProDiffuse encoder and ProDiffuse decoder into a multi-target tracking model.
[0088] Preferably, before S2, it also includes:
[0089] S02: Training the multi-target tracking model to be trained to obtain a pre-trained multi-target tracking model, specifically including:
[0090] S021: Acquire a continuous set of images or videos, and annotate each frame in the continuous set of images or videos to obtain training data, wherein the annotation includes identifying a tracking target and assigning an ID label, and then annotating a real bounding box of the tracking target;
[0091] S022: Using the backbone network, extract image features from each frame to obtain the first image features;
[0092] S023: inputting the first image feature into a ProDiffuse encoder to obtain a second image feature;
[0093] S024: Obtain a true bounding box, generate a random proposal query based on the true bounding box, and obtain a result of the random proposal query, wherein the random proposal query includes using a denoising diffusion module to sample a random boundary from a Gaussian distribution and add Gaussian noise to obtain a noise bounding box;
[0094] S025: performing a tracked target proposal query based on at least two frames of images, extracting a trajectory corresponding to the tracked target, and obtaining a result of the tracked target proposal query;
[0095] S026: passing the second image feature, the result of the random proposal query and the result of the tracked target proposal query to the ProDiffuse encoder, and outputting the tracked target trajectory and the corresponding bounding box;
[0096] S027: Based on the selected loss function, the multi-target tracking model to be trained is iteratively optimized and trained to obtain a pre-trained multi-target tracking model, wherein the iterative optimization training is to connect the random proposal query with the tracked target proposal query to obtain the tracked target instance query, and the result of the tracked target instance query is input into the ProDiffuse decoder, and the tracked target instance query adopts the DDIM model for iterative optimization, and the loss function is the ensemble average loss.
[0097] In some embodiments, in order to improve computational efficiency, during inference, the present application divides the multi-target tracking model into two separate components: ProDiffuse encoder and ProDiffuse decoder. Figure 2 As shown in Figure 1, the ProDiffuse encoder is used to extract image features, which are considered as conditions for sampling. The ProDiffuse encoder is executed only once during single image inference, while the ProDiffuse decoder is responsible for performing iterative calculations. The partition strategy model of the diffusion track significantly reduces the computational burden and improves the speed of the algorithm without affecting its overall performance. Figure 2 Schematic diagram of the multi-object tracking model is illustrated. For the first frame, there is no track query and the application only feeds the random proposal query to the ProDiffuse decoder. The decoder output along with the proposal query is used to generate the proposal query for the next iteration step. In subsequent frames, the predicted box from the previous frame is concatenated to the random proposal and input to the ProDiffuse decoder for bounding box prediction.
[0098] This application explores a new paradigm for multi-target tracking in which diffusion models empower tracking proposal propagation. To speed up training, this application uses ResNet-50 as the backbone network. The backbone network is responsible for extracting image features from each frame and then passing it to the ProDiffuse encoder for further processing. The output of the ProDiffuse encoder is used as a condition for the denoising diffusion process, where the results of the next step are sampled. The ProDiffuse decoder takes into account the complexity of the tracking task. This application believes that uncertainty reasoning is crucial to improving the accuracy and robustness of the tracking model. To this end, this application introduces random proposals as additional input to the ProDiffuse decoder, unlike previous methods that use fixed-length learnable queries. The addition of random proposals helps to extract more accurate and robust tracking information. Importantly, since random proposals only pass information about positioning and size, they do not limit the number of proposals. In addition, the variability of the number of inputs enables the model to process more proposals than during training, thereby improving tracking performance during model reasoning.
[0099] It should be further explained that the training target of the diffusion process can be noise or the final image. The ultimate goal of this application is to obtain the bounding box and ID label of the tracked object. Therefore, the optimization target of the model is the final tracking bounding box. Although the diffusion process is performed on a single image, the tracking task requires time information. Therefore, the Collective Average Loss in MOTR is finally used to optimize the model.
[0100] Preferably, SCPP is Self-Conditional Proposal Propagation. During the reasoning process, the particularity of the tracking task allows the tracking result box of the previous frame to be used as an additional input for the next frame, called a tracking proposal, to reduce the number of random proposals while maintaining the stability of the number of proposals. Even if the tracking proposal contains noise, such as the target motion that changes over time, the tracking task can still extract valid information from it.
[0101] Preferably, DDIM (DENOISING DIFFUSION IMPLICIT MODELS) is an effective scheme for reducing the sampling step in the generative model. Due to iterative optimization training, the model can iteratively optimize its own results during the inference process.
[0102] Preferably, since the model has learned the ability to extract targets from noise during training, in the inference process, not only random Gaussian distribution proposals are used, but also additional track proposals containing certain prior information, which are derived from the output results of the previous frame. Due to the anti-interference ability obtained from the noise box training, the model can still extract useful information from these track proposals containing interference information, thereby improving the tracking accuracy of the model.
[0103] It should be noted that during the training method, random bounding boxes are used as input to the ProDiffuse decoder, and the ProDiffuse decoder can effectively handle any number of random bounding boxes, regardless of whether the number of these bounding boxes is different from the number used during training. In addition, through empirical analysis, the number of bounding boxes tends to increase in complex scenes relative to simple scenes. This mechanism provides applications with the flexibility to choose an appropriate number of bounding boxes to trade off between accuracy and computational efficiency.
[0104] S3: Using the backbone network and the ProDiffuse encoder, image features are extracted for each frame of the image to obtain image features;
[0105] S4: using the denoising diffusion module, passing each frame of the image through a random proposal sequence generation mechanism in turn, generating a proposal query, and obtaining a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query, and a tracked target instance query;
[0106] Preferably, S4 includes:
[0107] S41: Using the denoising diffusion module, starting from the first frame, a random proposal query is performed to obtain a result of the random proposal query, where the random proposal query generates a proposal sequence based on a Gaussian process;
[0108] S42: using the denoising diffusion module, starting from the second frame, performing a tracked target proposal query to obtain a result of the tracked target proposal query, wherein the tracked target proposal query includes a position reference obtained by the tracked target proposal query of the previous frame, and the position reference is a trajectory corresponding to the tracked target;
[0109] S43: Starting from the second frame, based on the result of the random proposal query and the result of the tracked target proposal query, a tracked target instance query is performed to obtain a result of the tracked target instance query, wherein the result of the tracked target instance query includes instance information, and the instance information is a bounding box corresponding to the tracked target.
[0110] Preferably, S43 includes:
[0111] S431: establishing a diffusion forward process, starting from the first frame, extracting the bounding box corresponding to the tracked target, and filling it to obtain a filled bounding box;
[0112] S432: additional bounding boxes generated based on Gaussian distribution;
[0113] S433: Connect the filled bounding box and the additional bounding box to obtain a complete bounding box.
[0114] S434: Adding Gaussian noise to all the bounding boxes to obtain a noisy bounding box, wherein the scale of the Gaussian noise is controlled by a function αt, where α follows a monotonically decreasing cosine schedule in different time steps t.
[0115] Preferably, S44 includes:
[0116] S441: Positionally encode the result of the random proposal query by using a ProDiffuse encoder to obtain an encoded result of the random proposal query;
[0117] S442: Update the encoded random proposal query result through the ProDiffuse decoder, interact with the image features, generate a trajectory proposal, perform a tracked target proposal query, and obtain the track of the tracked target;
[0118] S443: Initialize the trajectory proposal of the current frame according to the trajectory of the tracked target in the previous frame, continue to track and adjust the trajectory, and then query the tracked target instance based on the image features.
[0119] S5: For each frame of the image, the result of the proposed query and the image features are input into the ProDiffuse decoder, the tracking target of each frame is identified, and the bounding box corresponding to the tracking target is obtained. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame;
[0120] It should be noted that the comparison of the method of the embodiment of the present application on the challenging DanceTrack test set shows that the method of the embodiment of the present application shows significant performance on the DanceTrack dataset, significantly improving the detection and association capabilities.
[0121] Among all the joint detection and tracking (JDT) methods, the method of the embodiment of the present application stands out with its outstanding performance, achieving impressive scores of HOTA 60.1, MOTA 85.5 and IDF1 61.7. These results demonstrate the effectiveness of the method of the embodiment of the present application in dealing with the unique challenges posed by dance scenes, where diverse movements and appearances require complex tracking solutions.
[0122] In addition, compared with the tracking before detection (TBD) method, the method of the embodiment of the present application shows excellent competitiveness, surpassing most TBD methods, including ByteTrack, with an excellent DetA metric score of 6.1, and achieving excellent results in the AssA metric. Compared with OCSORT with YOLOX as the detector, although slightly inferior in the DetA metric, the method of the embodiment of the present application has a clear advantage of 8.8 in the AssA metric, and has a commendable lead of 5.0 in the HOTA metric.
[0123] The results fully demonstrate the ability of the method of the embodiment of the present application to handle the complexity of multi-target tracking with diffusion model-enabled tracking proposal propagation, especially in complex scenes. By taking advantage of the benefits of noise boxes and adopting complex tracking strategies, the method of the embodiment of the present application helps to establish a new benchmark, which may open up new possibilities for future progress in the field of multi-target tracking with diffusion model-enabled tracking proposal propagation.
[0124] S6: Post-process the result of the proposed query to obtain a trajectory corresponding to the tracking target.
[0125] Preferably, S6 includes:
[0126] S61: Integrate the DDIM model into the multi-target tracking model;
[0127] S62: Based on the result of the proposed query, the tracking target is optimized by using the DDIM iterative algorithm;
[0128] S63: fusing the motion information and the image features to obtain a motion feature representation;
[0129] S64: Using the motion feature representation as the input of the DDIM iterative algorithm to obtain a refined trajectory prediction;
[0130] S65: Generate a self-conditional proposal query based on the result of the proposal query;
[0131] S66: Fusing the self-conditional proposal query with other tracked target proposal queries;
[0132] S67: The fused proposal query is used as the input of the subsequent ProDiffuse decoder to enhance the position information of the tracked target.
[0133] In some embodiments, however, it is worth noting that during the inference process, the proposal query and the trajectory query are concatenated and input into the network for inference. To ensure consistency between training and inference, in the input of this connection, the trajectory query does not use the DDIM sampling method. DDIM is also a method for improving target motion modeling, which can improve the accuracy of target detection and tracking to improve the prediction of trajectories and the detection of newly appeared targets. Therefore, during the iterative optimization process, the proposal query and the trajectory query are first separated. DDIM is only applied to the proposal query one by one. Then, the optimized proposal query is concatenated with the trajectory query and input into the conditional decoder to obtain the result of iterative optimization. , Optimizing the tracking process using DDIM can improve the accuracy and robustness of multi-target tracking given by the diffusion model to the tracking proposal propagation, especially when dealing with fast-moving targets and complex scenes.
[0134] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.
[0135] Figure 3 1 is a block diagram of a multi-target tracking device with a diffusion model assigned to track proposal propagation according to an exemplary embodiment, wherein the device is used in a multi-target tracking method with a diffusion model assigned to track proposal propagation. Figure 3,The device includes an image set module, a multi-target tracking model module, a feature extraction module, a proposal query module, a decoder module and a post-processing module.
[0136] Image set module: used to obtain continuous image sets or videos;
[0137] A multi-target tracking model module: used to sequentially input each frame of the continuous image set or video into a pre-trained multi-target tracking model, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder;
[0138] Feature extraction module: used to extract image features for each frame of image using the backbone network and ProDiffuse encoder to obtain image features;
[0139] Proposal query module: used to use the denoising diffusion module to pass each frame of the image through the random proposal sequence generation mechanism in turn to generate a proposal query and obtain a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query and a tracked target instance query;
[0140] Decoder module: for each frame of the image, input the result of the proposed query and the image features into the ProDiffuse decoder, identify the tracked target in each frame, and obtain the bounding box corresponding to the tracked target. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame;
[0141] Post-processing module: used to post-process the result of the proposed query to obtain the trajectory corresponding to the tracking target.
[0142] An embodiment of the present invention provides a multi-target tracking device with diffusion model assigned to tracking proposal propagation, and the multi-target tracking device with diffusion model assigned to tracking proposal propagation includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, a method as described in any one of the multi-target tracking methods with diffusion model assigned to tracking proposal propagation is implemented.
[0143] An embodiment of the present invention provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any of the methods described in the above-mentioned diffusion model-assigned tracking proposal propagation multi-target tracking method.
[0144] The technical solution provided by the embodiment of the present invention generates the number of negative samples by simulation and increases the number of noise samples, that is, the number of negative samples, which helps to improve the multi-target tracking performance of the diffusion model endowed with tracking proposal propagation.
[0145] Figure 4 is a structural diagram of a multi-target tracking device that uses a diffusion model to propagate tracking proposals, as provided in an embodiment of the present invention. Figure 4 As shown, the multi-target tracking device that uses the diffusion model to propagate the tracking proposal may include the above Figure 3 The multi-target tracking apparatus 410 with a diffusion model assigned to it and a tracking proposal propagation may include a processor 2001 .
[0146] Optionally, the multi-target tracking device 410 that uses a diffusion model to impart tracking proposal propagation may further include a memory 2002 and a transceiver 2003 .
[0147] The processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0148] Combine the following Figure 4 The components of the multi-target tracking device 410 that uses a diffusion model to propagate tracking proposals are specifically introduced as follows:
[0149] The processor 2001 is the control center of the multi-target tracking device 410 that the diffusion model assigns to the propagation of the tracking proposal, and can be a processor or a general term for multiple processing elements. For example, the processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).
[0150] Optionally, the processor 2001 may execute various functions of the multi-target tracking device 410 that imparts tracking proposal propagation to the diffusion model by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0151] In a specific implementation, as an embodiment, the processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.
[0152] In a specific implementation, as an embodiment, the multi-target tracking device 410 that imparts the tracking proposal propagation with a diffusion model may also include multiple processors, such as Figure 4 2001 and processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0153] The memory 2002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0154] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the processor 2001 or exist independently, and is endowed with the interface circuit ( Figure 4 2001) is coupled to the processor 2001, which is not specifically limited in the embodiment of the present invention.
[0155] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0156] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0157] Optionally, the transceiver 2003 may be integrated with the processor 2001 or may exist independently and impart tracking proposal propagation to the interface circuit of the multi-target tracking device 410 through the diffusion model ( Figure 4 2001) is coupled to the processor 2001, which is not specifically limited in the embodiment of the present invention.
[0158] It should be noted that Figure 4 The diffusion model shown in the figure gives the structure of the multi-target tracking device 410 for propagating tracking proposals and does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0159] In addition, the technical effects of the multi-target tracking device 410 with diffusion model assigned to tracking proposal propagation can refer to the technical effects of the multi-target tracking method with diffusion model assigned to tracking proposal propagation described in the above method embodiment, which will not be repeated here.
[0160] It should be understood that the processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0161] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0162] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0163] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0164] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0165] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0166] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0168] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0169] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0170] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0171] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A multi-target tracking method with a diffusion model endowed with tracking proposal propagation, characterized in that: include: S1: Acquire a continuous set of images or videos; S2: Inputting each frame of the continuous image set or video into a pre-trained multi-target tracking model in sequence, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder; S3: Using the backbone network and the ProDiffuse encoder, image features are extracted for each frame of the image to obtain image features; S4: using the denoising diffusion module, passing each frame of the image through the random proposal sequence generation mechanism in turn, generating a proposal query, and obtaining a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query, and a tracked target instance query; including: S41: Using the denoising diffusion module, starting from the first frame, a random proposal query is performed to obtain a result of the random proposal query, wherein the random proposal query generates a proposal sequence based on a Gaussian process; comprising: S411: establishing a diffusion forward process, starting from the first frame, extracting the bounding box corresponding to the tracked target, and filling it to obtain a filled bounding box; S412: additional bounding boxes generated based on Gaussian distribution; S413: Connecting the filled bounding box with the additional bounding box to obtain a complete bounding box; S414: Add Gaussian noise to all the bounding boxes to obtain a noise bounding box. The scale of the Gaussian noise is determined by the function control, At different time steps follows a monotonically decreasing cosine schedule; S42: using the denoising diffusion module, starting from the second frame, performing a tracked target proposal query to obtain a result of the tracked target proposal query, wherein the tracked target proposal query includes a position reference obtained by the tracked target proposal query of the previous frame, and the position reference is a trajectory corresponding to the tracked target; S43: Starting from the second frame, based on the result of the random proposal query and the result of the tracked target proposal query, a tracked target instance query is performed to obtain a tracked target instance query result, wherein the tracked target instance query result includes instance information, and the instance information is a bounding box corresponding to the tracked target; including: S431: Positionally encode the result of the random proposal query by using a ProDiffuse encoder to obtain an encoded result of the random proposal query; S432: The encoded random proposal query result is updated through the ProDiffuse decoder, and interacts with the image features to generate a trajectory proposal, perform a tracked target proposal query, and obtain the track of the tracked target; S433: Initializing the trajectory proposal of the current frame according to the trajectory of the tracked target in the previous frame, continuously tracking and adjusting the trajectory, and then querying the tracked target instance based on the image features; S5: For each frame of the image, the result of the proposed query and the image features are input into the ProDiffuse decoder, the tracking target of each frame is identified, and the bounding box corresponding to the tracking target is obtained. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame; S6: Post-process the result of the proposed query to obtain a trajectory corresponding to the tracking target.
2. The multi-target tracking method with diffusion model-based tracking proposal propagation according to claim 1, characterized in that: Before inputting the continuous image set or each frame of the video into the pre-trained multi-target tracking model in sequence in S2, the method further includes: S01: Obtain a multi-target tracking model to be trained; The step S01 of obtaining a multi-target tracking model to be trained includes: S011: constructing a denoising diffusion module, wherein the input of the denoising diffusion module is a continuous image set or each frame image of a video; S012: Initialize a backbone network based on the ResNet-50 network model, wherein the input of the backbone network is a continuous set of images or each frame of a video; S013: construct a ProDiffuse encoder based on the Transformer model, wherein the input of the ProDiffuse encoder is the output of the backbone network; S014: constructing a ProDiffuse decoder based on the Transformer model, wherein the input of the ProDiffuse decoder is the output of the ProDiffuse encoder, the output of the denoising diffusion module, and the output of the ProDiffuse decoder of the previous frame; S015: Combine the denoising diffusion module, backbone network, ProDiffuse encoder and ProDiffuse decoder into a multi-target tracking model.
3. The multi-target tracking method with diffusion model-based tracking proposal propagation according to claim 2, characterized in that: Before inputting the continuous image set or each frame of the video into the pre-trained multi-target tracking model in sequence in S2, the method further includes: S02: training the multi-target tracking model to be trained to obtain a pre-trained multi-target tracking model; The step S02 of training the multi-target tracking model to be trained to obtain a pre-trained multi-target tracking model includes: S021: Acquire a continuous set of images or videos, and annotate each frame in the continuous set of images or videos to obtain training data, wherein the annotation includes identifying a tracking target and assigning an ID label, and then annotating a real bounding box of the tracking target; S022: Using the backbone network, extract image features from each frame to obtain the first image features; S023: inputting the first image feature into a ProDiffuse encoder to obtain a second image feature; S024: Obtain a true bounding box, generate a random proposal query based on the true bounding box, and obtain a result of the random proposal query, wherein the random proposal query includes using a denoising diffusion module to sample a random boundary from a Gaussian distribution and add Gaussian noise to obtain a noise bounding box; S025: performing a tracked target proposal query based on at least two frames of images, extracting a trajectory corresponding to the tracked target, and obtaining a result of the tracked target proposal query; S026: passing the second image feature, the result of the random proposal query and the result of the tracked target proposal query to the ProDiffuse encoder, and outputting the tracked target trajectory and the corresponding bounding box; S027: Based on the selected loss function, the multi-target tracking model to be trained is iteratively optimized and trained to obtain a pre-trained multi-target tracking model, wherein the iterative optimization training is to connect the random proposal query with the tracked target proposal query to obtain the tracked target instance query, and the result of the tracked target instance query is input into the ProDiffuse decoder, and the tracked target instance query adopts the DDIM model for iterative optimization, and the loss function is the ensemble average loss.
4. The multi-target tracking method with diffusion model-based tracking proposal propagation according to claim 3, characterized in that: The step S6 of post-processing the result of the proposed query to obtain a trajectory corresponding to the tracking target includes: S61: Integrate the DDIM model into the multi-target tracking model; S62: Based on the result of the proposed query, the tracking target is optimized by using the DDIM iterative algorithm; S63: fusing the motion information and the image features to obtain a motion feature representation; S64: Using the motion feature representation as the input of the DDIM iterative algorithm to obtain a refined trajectory prediction; S65: Generate a self-conditional proposal query based on the result of the proposal query; S66: Fusing the self-conditional proposal query with other tracked target proposal queries; S67: The fused proposal query is used as the input of the subsequent ProDiffuse decoder to enhance the position information of the tracked target.
5. A multi-target tracking device with a diffusion model endowed with tracking proposal propagation, characterized in that: The device is applicable to the method according to any one of claims 1 to 4, and comprises: Image set module: used to obtain continuous image sets or videos; A multi-target tracking model module: used to sequentially input each frame of the continuous image set or video into a pre-trained multi-target tracking model, wherein the multi-target tracking model comprises a denoising diffusion module, a backbone network, a ProDiffuse encoder and a ProDiffuse decoder, wherein the backbone network is a ResNet-50 network model, the ProDiffuse encoder is a transformer-based encoder, and the ProDiffuse decoder is a transformer-based decoder; Feature extraction module: used to extract image features for each frame of image using the backbone network and ProDiffuse encoder to obtain image features; Proposal query module: used to use the denoising diffusion module to pass each frame of the image through the random proposal sequence generation mechanism in turn to generate a proposal query and obtain a result of the proposal query, wherein the proposal query includes a random proposal query, a tracked target proposal query and a tracked target instance query; Decoder module: for each frame of the image, input the result of the proposed query and the image features into the ProDiffuse decoder, identify the tracked target in each frame, and obtain the bounding box corresponding to the tracked target. The result of the proposed query includes the result of the random proposal query in the first frame, and includes the result of the tracked target instance query starting from the second frame; Post-processing module: used to post-process the result of the proposed query to obtain the trajectory corresponding to the tracking target.
6. An electronic device, characterized in that: The electronic device comprises: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program codes; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of claims 1 to 4 above.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method of any one of claims 1 to 4.
Citation Information
Patent Citations
Online multi-target tracking method based on multifunctional aggregation and tracking simulation training
CN112651995A
Multi-target tracking method, system and device and readable storage medium
CN116485838A
Video multi-target tracking method based on multi-scale channel feature aggregation
CN117173217A
Space-time fusion multi-target tracking method, device, equipment and medium
CN117314965A
Method for tracking object by using convolutional neural network including tracking network and computing device using the same
US10269125B1
