A method of sequential tracking based on diffusion modeling

Abstract

The application discloses a kind of serialization tracking methods based on diffusion modeling.First, the main part of feature extraction network is constructed as encoder module.Second, the decoder module based on diffusion model is built and divided into noise adding stage and de-noising stage.A suitable loss function and optimization method are defined to ensure the convergence in the network training process.The tracking network is fully trained to accurately identify and track the target.The training and inference process are unified in time sequence, multiple templates and a search area are input in the training stage, so that multiple frame updating is realized in the inference stage.In the inference stage, the diffusion model directly samples from random Gaussian noise to generate the sequence of tracking frame.This method performs well in similar interference and target deformation scenarios, not only with high tracking accuracy, but also with real-time sequence generation speed, i.e., the method can perform real-time tracking at a speed of 51 frames per second, meeting the real-time requirements in practical applications.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning, and specifically relates to a serialization tracking method based on diffusion modeling. Background Technology

[0002] In the field of computer vision, especially in multimedia applications, single-object tracking plays a crucial role in various scenarios, including autonomous driving, ecological management, and medical treatment. Single-object tracking primarily refers to the continuous localization of a single entity contained within a rectangular bounding box, given the entity's initial state.

[0003] Early single-object tracking techniques can be broadly categorized into two types: interaction-based single-stream tracking networks (such as SimTrack, Mixformer, grm, stark, and ostrack) and interaction-based two-stream tracking networks (such as TranST, siamban, sparsett, keepTrack, and tomp). These methods rely on correlation measurement frameworks at the image feature level. Despite their effectiveness, the custom-designed head networks in these algorithms struggle to extract accurate location information from image tasks.

[0004] To address the limitations of previous tracking methods, pioneering works such as SeqTrack and ARTrack introduced an autoregressive framework. This framework transforms the loss optimization in the head network into a discrete sequence generation problem. In this way, the tracking task is defined as a coordinate sequence generation task, establishing a connection between the image and the coordinate sequence.

[0005] However, autoregressive frameworks for tracking tasks have two significant limitations. First, as demonstrated in the SeqTrack study, the performance of the tracking model is heavily dependent on the size of the shared vocabulary. Increasing the vocabulary size often leads to saturation or even degrades tracking performance. Second, generating coordinate sequences recursively requires manual intervention to determine the generation order. For example, the sequence can be generated from the top-left corner to the bottom-right corner ([xmin, ymin, xmax, ymax]) or from the center coordinate point to the boundary points ([x, y, w, h]). Different coordinate sequence generation directions significantly impact tracking performance. These limitations hinder recursive methods from further improving tracking performance.

[0006] In other words, current recursive methods in tracking tasks suffer from lossy mapping of coordinate sequences due to finite vocabulary size. This mapping is inherently limited by the discretization of coordinates. To overcome the information loss and sequence constraints caused by autoregression, a lossless method capable of predicting each continuous coordinate value in parallel is needed. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a serialization tracking method based on diffusion modeling. Its core is the proposal of a tracker based on a diffusion model, which aims to solve the inherent defects and limitations in the implementation of previous autoregressive trackers.

[0008] Specifically, the diffusion model addresses the limitations of autoregressive frameworks in visual tracking by shifting distribution estimation from finite discrete variables to infinite continuous probabilities. This reduces the loss of semantic information in feature mapping. Furthermore, the diffusion model's tracker supports parallel sequence generation, breaking the sequence prediction limitations of methods like SeqTrack. Following current autoregressive methods, the tracker in this invention also employs the same encoder and decoder architecture to predict coordinate sequences. For the encoder, this invention provides an optional ladder module instead of simply using a vanilla VisionTransformer to extract visual features. This is a parameter-efficient transfer learning method involving only multilayer perceptrons to enhance feature representation and achieve domain adaptation of the target tracking dataset. In the decoder stage, the diffusion model's noise addition and denoising processes are used to generate the tracking box sequence.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a serialization tracking method based on diffusion modeling, comprising the following steps:

[0010] Step 1: Build a ViT-based backbone network to initially extract features and achieve image encoder pre-training; the backbone network includes template branches and search region branches;

[0011] Step 2: Build a decoder module based on the diffusion model to realize the process of decoding the tracking box sequence from noise;

[0012] Step 3: Optimize parameters and loss function to ensure the stability and convergence of the training process;

[0013] Step 4: First, the 0th frame image with a specific initial bounding box is input into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) and the width w and height h. The backbone network of the template branch outputs the initial template feature map. Then, the subsequent frames in the video, except for the 0th frame, are input into the search template branch in sequence, and a tracking box sequence is randomly sampled in Gaussian noise. This tracking box is input into the decoder module to iteratively generate the final tracking box where the target is located.

[0014] This invention offers the following advantages: Experiments on general datasets demonstrate that the proposed diffusion-modeled serialization tracker exhibits optimal tracking performance in various complex scenarios, such as complete occlusion and morphological changes. By modeling sequences based on a diffusion model, the dependence of traditional serialization tracking on vocabulary size is effectively avoided, enhancing the tracker's accuracy and robustness across various tasks. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a serialization tracking method based on diffusion modeling according to the present invention;

[0016] Figure 2 A qualitative schematic diagram illustrating the serialization modeling based on diffusion modeling proposed in this invention.

[0017] Figure 3 This is a quantitative schematic diagram illustrating the serialization modeling based on diffusion modeling proposed in this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0019] like Figure 1 The diagram shown is a flowchart of a serialization tracking method based on diffusion modeling according to the present invention, which specifically includes:

[0020] Step 1: Build a ViT-based backbone network to initially extract features and achieve pre-training of the image encoder;

[0021] Specifically, step 1 includes:

[0022] Step 1.1: Select ViT (Virtual Attention) as the backbone network for feature extraction. This backbone network is a single-stream network. The inputs of the template branch and the search region branch are concatenated and fed into the ViT backbone network. The ViT backbone network first divides the input image into fixed-size blocks (usually 16x16 pixels) and treats each block as a word. Then, each block is transformed into a high-dimensional vector through a linear embedding layer.

[0023] Step 1.2: The obtained block embedding sequence is then fed into a series of attention coding layers. Each attention coding layer contains a multi-head self-attention module and a feedforward neural network (FFN), comprising a total of 12 different stages. Each stage maintains a consistent dimensionality and consists of a multi-head self-attention module, a feedforward neural network, and skip connections. Figure 1 The output of each attention encoding layer is fed into a ladder layer for domain adaptation based on the tracking dataset. The ladder layer consists of only three FFN layers, where the input of each layer and the output of the previous layer pass through an FFN layer to obtain shared features F. P .

[0024] Step 1.3: For the pre-trained network, retain its pre-trained model parameters. Simultaneously, the MAE pre-training strategy is used in the ViT backbone network learning to further enhance the network's learning ability. Specifically, MAE uses an unlabeled dataset for pre-training. In this process, the model divides the input image into blocks and then randomly masks a portion of these blocks. Next, the model encodes the unmasked blocks using an encoder to generate feature representations. Subsequently, MAE trains a model with strong feature representation capabilities by randomly masking a portion of the image and then having the model reconstruct these masked pixels.

[0025] Step 2: Build a decoder module based on the diffusion model, namely the DM-decoder, to realize the process of decoding the tracking box sequence from noise;

[0026] The process of building the decoder module based on the diffusion model is as follows:

[0027] Step 2.1: Establish a noise-adding module for the diffusion model. The pre-given tracking box sequence S0 is noise-added. This noise-adding process simulates gradually adding noise from the real data distribution until the data becomes completely randomized. This is typically achieved through a noise-adding function that progressively transforms the original data into noisy data. This process can be performed in multiple steps, adding a certain amount of noise at each step until the data becomes sufficiently "fuzzy" or "random." Its mathematical representation is as follows:

[0028] ;

[0029] in, It is a coordinate sequence with Gaussian noise added to the timestamp t, and The input is the original coordinate sequence without Gaussian noise, where I is the identity matrix. It is Gaussian noise. This indicates a Gaussian distribution. is a hyperparameter determined by the continuous product of noise variances βs. For the noise addition process, at each timetamp, the noise variance schemes β1, β2, ..., βt with independent and identically distributed (iid) Gaussian noise are added to . Above, to form In the configuration of this invention, the bounding box ([x,y,w,h]) is used as ;

[0030] Step 2.2: Establish a noise denoising module for the diffusion model. This is the reverse of the noise addition process. It aims to gradually recover the original data from the noisy data. It iteratively recovers the original boxes from the noisy data using a neural network, learning the joint distribution of the tracking boxes and the original input image. This allows for the iterative generation of the desired target tracking boxes during inference. Figure 1 As shown, after adding noise, the sequence passes through an instance adaptation layer, and the noisy region is selected as the feature vector in the feature map. Subsequently, multi-head self-attention modules (MSA and MCA) and a feedforward neural network (FFN) are used to fuse the feature vector of the noisy region with the shared feature F from the backbone network. P t is a randomly generated vector, which is mapped and combined with the output of the multi-head self-attention module. Finally, the parameter values ​​of the denoised target box are output through a feedforward neural network (FFN). The L1 loss function serves as a metric between the denoised value and the true value. The original coordinate sequence, input without Gaussian noise, becomes the noise-adding module of the diffusion model in step 2.1. Subsequently The noise denoising module of the diffusion model after step 2.2 becomes LadderEnc and DMDec are the ViT-based backbone network and DM decoder, respectively. The following... and L was used as a loss function in the calculation. D This is the L1 distance loss; the specific formula is:

[0031] , ,

[0032] in, Input position codes for template branches. Input the position code for the branch of the search region.

[0033] Step 3: Optimize parameters and loss function to ensure the stability and convergence of the training process;

[0034] The training optimization process is designed as follows:

[0035] Step 3.1: For the loss function optimization design of the tracking network, L1 distance loss is used in the diffusion denoising process to measure the consistency between the denoising sequence and the original sequence. At the same time, cross-entropy classification loss and intersection-union (IU) localization loss are added to further enhance the localization ability. The relevant formulas are as follows:

[0036] ;

[0037] in, These are the actual labels, and n is the number of categories. It is the probability of learning.

[0038] ;

[0039] in, It is the bounding box predicted by the model. It is a real bounding box.

[0040] ;

[0041] Where N is the number of samples, It is a predicted value. It is the actual value.

[0042] During the optimization of network training, AdamW was selected as the optimizer to enhance the robustness of loss function learning and accelerate convergence.

[0043] Step 3.2: The four datasets COCO, TrackingNet, LaSOT, and GOT10k are used as the training basis. In the training label generation stage, this invention focuses on the generation results of sequences. Unlike traditional trackers that only consider the positive and negative samples of pixel feature vectors inside and outside the bounding box, this sequential tracking is closer to the generation of natural language, thereby reducing interference in the image.

[0044] Step 3.3: To ensure network convergence and training efficiency, the initial learning rate (1e-4), decay strategy (step descent strategy), number of iterations (200), and update threshold (0.7) were finely adjusted based on the size and complexity of the dataset. This step not only improved the network's learning efficiency but also ensured stable and reliable tracking performance.

[0045] Step 4: Reasoning process given the initial box;

[0046] The tracking and reasoning process in the initial state is as follows:

[0047] Step 4.1: First, the 0th frame image with a specific initial bounding box is input into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) and the width and height. The backbone network of the template branch outputs the initial template feature map.

[0048] Step 4.2: In the subsequent target tracking task, all frames in the video except frame 0 are sequentially input into the search region branch. A tracking box sequence is randomly sampled from Gaussian noise and input into the decoder network. The final tracking box containing the target is iteratively generated, employing the DDIM inference strategy. Specifically, the DDIM (Denoising Diffusion Implicit Models) inference strategy is primarily based on its unique denoising process. The core principle of DDIM is to overcome the limitations of Markov chains, allowing for skip-step denoising during sampling instead of step-by-step denoising, thus accelerating the sampling speed. During DDIM inference, the model first constructs a dynamic graph composed of a series of subgraphs, each corresponding to a task. Subgraphs are connected through shared nodes, which can be shared by different subgraphs. This design allows different tasks to share model parameters, thereby reducing model complexity.

[0049] In this embodiment, results based on a dataset of general tasks demonstrate the advanced performance of the tracker of the present invention. It is worth noting that... Figure 2 This section demonstrates the tracking capabilities of the method of this invention in the face of significant challenges such as target disappearance, interference from similar objects, and cluttered backgrounds. The algorithm of this invention is compared with state-of-the-art Seqtrack and Ostrack algorithms, where the bounding box represents the ground truth. In the first line, the tracker of this invention demonstrates its ability to accurately locate the target even when interfered with by objects similar to the target and in a cluttered background. The second line highlights the tracker's ability to quickly locate the target after it re-enters the field of view. The last line shows that in scenes filled with objects similar to the target, the tracker of this invention can accurately identify and track the target, thus demonstrating its excellent target discrimination ability. Meanwhile, as... Figure 3 As shown, it quantitatively describes its overall tracking performance (all) and tracking performance under different attributes during long-term tracking, such as background cluster and aspect ratio change. It can be observed that the method of the present invention is always in a leading position in tracking performance.

[0050] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any implementation method that utilizes a diffusion model for general task tracking falls within the scope of protection of the present invention.

Claims

1. A serialization tracking method based on diffusion modeling, characterized in that, The method includes the following steps: Step 1: Build a ViT-based backbone network to initially extract features and achieve image encoder pre-training; the backbone network includes template branches and search region branches; Step 2: Build a decoder module based on the diffusion model to realize the process of decoding the tracking box sequence from noise; Step 3: Optimize parameters and loss function to ensure the stability and convergence of the training process; Step 4: First, the 0th frame image with a specific initial bounding box is input into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) and the width w and height h. The backbone network of the template branch outputs the initial template feature map. Then, the subsequent frames in the video, except for the 0th frame, are input into the search region branch in sequence, and a tracking box sequence is randomly sampled in Gaussian noise. This tracking box is input into the decoder module to iteratively generate the final tracking box where the target is located. Step 2 includes: Step 2.1: Establishing a noise-adding module for the diffusion model, which adds noise to the pre-given tracking box sequence S0 through a noise-adding function. This function gradually transforms the original data into noisy data through multiple steps, each step adding a certain amount of noise. Its mathematical representation is as follows: ； in, It is a coordinate sequence with Gaussian noise added to the timestamp t, and The input is the original coordinate sequence without Gaussian noise, where I is the identity matrix. It is Gaussian noise. Indicates a Gaussian distribution. β is a hyperparameter determined by the continuous product of noise variances βs. For the noise addition process, at each time point, noise variance schemes β1, β2, ..., βt with independent and identically distributed Gaussian noise are added to βs. Above, to form Wherein, the bounding box [x,y,w,h] is used as ; Step 2.2: Establish a noise denoising module for the diffusion model. After adding noise, an instance adaptation layer is passed through the noise region in the feature map as the feature vector. Subsequently, the multi-head self-attention module MSA and MCA, along with the feedforward neural network FFN, are used to fuse the feature vector of the noise region with the shared feature F from the backbone network. P t is a randomly generated vector, which is mapped and combined with the output of the multi-head self-attention module. Finally, the parameter values ​​of the denoising target box are output through the feedforward neural network FFN, where the L1 loss function is used as a measure between the denoising value and the true value. Step 3 includes: Step 3.1: For the L1 loss function optimization design, L1 distance loss is used in the diffusion denoising process to measure the consistency between the denoising sequence and the original sequence. At the same time, cross-entropy classification loss and intersection-union ratio (IU) localization loss are added to further enhance the localization ability. The relevant formulas are as follows: ； in, These are the actual labels, and n is the number of categories. It is the probability of being learned; ； in, It is the bounding box predicted by the model. It is a real bounding box; ； Where N is the number of samples, It is a predicted value. It is the actual value; During the optimization of network training, AdamW was selected as the optimizer to enhance the robustness of loss function learning and accelerate convergence. Step 3.2: The four datasets COCO, TrackingNet, LaSOT, and GOT10k are used as the training base; Step 3.3: Adjust the initial value of the learning rate, decay strategy, descent strategy, number of iterations and update threshold to improve the learning efficiency of the network.

2. The serialization tracking method based on diffusion modeling according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Select ViT with pure attention as the backbone network for feature extraction. The inputs of the template branch and the search region branch are concatenated and input into the ViT backbone network. The ViT backbone network performs block segmentation, dividing the input image into blocks of fixed size and treating each block as a word. Then, each block is transformed into a high-dimensional vector through a linear embedding layer to obtain the block embedding sequence. Step 1.2: The obtained block embedding sequence is then fed into a series of attention coding layers. Each attention coding layer contains a multi-head self-attention module and a feedforward neural network (FFN). The output of each attention coding layer is fed into a ladder layer for domain adaptation based on the tracking dataset. The ladder layer consists of only three FFN layers, where the input of each layer and the output of the previous layer pass through an FFN layer to obtain shared features F. P ; Step 1.3: For the pre-trained backbone network, retain its pre-trained network model parameters.

Images