Serialization tracking method based on diffusion modeling

Abstract

The invention discloses a serialization tracking method based on diffusion modeling. Firstly, a trunk of a feature extraction network is constructed as an encoder module. Secondly, building a decoder module based on a diffusion model, wherein the decoder module is divided into a noise adding stage and a noise removing stage; and defining a proper loss function and an optimization method so as to ensure convergence in a network training process. A tracking network is fully trained, so that the tracking network can accurately identify and track a target. The training process and the reasoning process are unified in time sequence, multiple templates and one search area are input in the training stage, and therefore multi-frame updating is achieved in the reasoning stage. In the reasoning stage, the diffusion model directly samples from random Gaussian noise, so that a sequence of a tracking frame is generated. The method is excellent in performance under similar interference and target deformation scenes, not only is high in tracking precision, but also has real-time sequence generation speed, that is, the method can carry out real-time tracking at the speed of 51 frames per second, and the real-time requirement in practical application is met.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a serialization tracking method based on diffusion modeling. Background Art

[0002] In the field of computer vision, especially in multimedia applications, single object tracking plays an important role in many scenarios, including autonomous driving, ecological management, and medical treatment. Single object tracking mainly refers to the continuous positioning of a single entity contained in a rectangular bounding box, where the initial state of the entity is given.

[0003] Early single target tracking techniques can be roughly divided into two categories: interactive single-stream tracking networks (such as SimTrack, Mixformer, grm, stark, ostrack) and interactive dual-stream tracking networks (such as TranST, siamban, sparsett, keepTrack, tomp). These methods rely on the correlation measurement framework at the image feature level. Although they are effective, the customized head networks in these algorithms have difficulty extracting accurate location information from image tasks.

[0004] To address the limitations of previous generation tracking methods, pioneering works such as SeqTrack and ARTrack introduced an autoregressive framework. This framework transforms the loss optimization in the Head network into a discrete sequence generation problem. In this way, the tracking task is defined as a coordinate sequence generation task, establishing a connection between images and coordinate sequences.

[0005] However, the autoregressive framework for tracking tasks has two significant limitations. First, as shown in the SeqTrack study, the performance of the tracking model is heavily dependent on the size of the shared vocabulary. Increasing the vocabulary size usually leads to saturation or even degradation of tracking performance. Second, generating coordinate sequences in a recursive manner requires manual intervention to determine the generation order. For example, the sequence can be generated from the upper left corner point to the lower right corner point ([xmin, ymin, xmax, ymax]) or from the center coordinate point to the boundary point ([x, y, w, h]). Different coordinate sequence generation directions can have a significant impact on tracking performance. These limitations prevent recursive methods from further improving tracking performance.

[0006] In other words, current recursive methods in tracking tasks result in a lossy mapping of coordinate sequences due to the limited vocabulary size. This mapping is inherently limited due to the discretization of coordinates. To overcome the information loss and sequence constraint issues introduced by autoregression, a lossless method is needed that can predict the continuous values ​​of each coordinate in parallel. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a serialized tracking method based on diffusion modeling. The core of the method is to propose a tracker based on a diffusion model. The method aims to solve the inherent defects and limitations of previous autoregressive trackers in implementation.

[0008] Specifically, the diffusion model can solve the limitations of the autoregressive framework in visual tracking by transferring distribution estimates from finite discrete variables to infinite continuous probabilities. This reduces the loss of semantic information in feature maps. In addition, the diffusion model tracker supports parallel sequence generation, breaking the limitations of sequence prediction in methods such as SeqTrack. According to the current autoregressive method, the tracker of the present invention also uses the same encoder and decoder architecture to predict the coordinate sequence. For the encoder, the present invention provides an optional ladder module instead of simply using the vanilla VisionTransformer to extract visual features. This is a parameter-efficient transfer learning method that only involves multi-layer perceptrons to enhance feature representation and achieve domain adaptation of target tracking datasets. In the decoder stage, the denoising and de-noising process of the diffusion model is used to achieve the generation of tracking box sequences.

[0009] To achieve the above object, the present invention adopts the following technical solution, a serialization tracking method based on diffusion modeling, comprising the following steps: Step 1: Build a backbone network based on ViT to preliminarily extract features and implement image encoder pre-training; the backbone network includes a template branch and a search area branch; Step 2: Build a decoder module based on the diffusion model to realize the process of decoding the tracking box sequence from the noise; Step 3: Optimize parameters and loss functions to ensure the stability and convergence of the training process; Step 4: First, input the 0th frame image with a specific initial bounding box into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) as well as the width w and height h. The backbone network of the template branch outputs the initial template feature map. Subsequently, the subsequent frames in the video except the 0th frame are input into the search template branch in sequence, and a tracking box sequence is randomly sampled in Gaussian noise. This tracking box is input into the decoder module, and the final tracking box of the target is iteratively generated.

[0010] The present invention has the following beneficial effects: Experiments on general data sets show that the proposed serialized tracker based on diffusion modeling has the best tracking performance in various complex scenes such as complete occlusion and morphological changes. By modeling the sequence based on the diffusion model, the dependence of traditional serialized tracking on the number of vocabularies is effectively avoided, and the accuracy and robustness of the tracker in various tasks are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a serialization tracking method based on diffusion modeling of the present invention; Figure 2 A qualitative schematic diagram of the serialization modeling based on diffusion modeling proposed for the implementation of the present invention; Figure 3 A quantitative schematic diagram of the serialization modeling based on diffusion modeling proposed for the implementation of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned purpose, the present invention adopts the following technical scheme.

[0013] like Figure 1 FIG. 1 is a flow chart of a serialization tracking method based on diffusion modeling of the present invention, which specifically includes: Step 1: Build a backbone network based on ViT to initially extract features and implement image encoder pre-training; Specifically, step 1 includes: Step 1.1: Select pure attention ViT as the backbone network for feature extraction. The backbone network is a single-stream network. The inputs of the template branch and the search area branch are concatenated and input into the ViT backbone network. The ViT backbone network first divides the input image into blocks of fixed size (usually 16x16 pixel blocks) and treats each block as a word. Then, each block is converted into a high-dimensional vector through a linear embedding layer.

[0014] Step 1.2: The obtained block embedding sequence is then fed into a series of attention encoding layers. Each attention encoding layer contains a multi-head self-attention module (Multi-Head Self-Attention) and a feed-forward neural network (FFN), including a total of 12 different stages. The dimensions of each stage are consistent, and they are composed of multi-head self-attention modules, feed-forward neural networks, and jump connections. Figure 1 The output of each layer of attention encoding is connected to the ladder layer for domain adaptation based on the tracking dataset. The ladder layer consists of only three layers of FFN, where the input of each layer and the output of the previous layer pass through an FFN layer to obtain the shared feature F P .

[0015] Step 1.3: For the pre-trained network, retain its pre-trained network model parameters, and the MAE pre-training strategy is used in the backbone network learning of ViT to further improve the learning ability of the network. Specifically, MAE uses an unlabeled dataset for pre-training. In this process, the model divides the input image into blocks and then randomly masks some of the blocks. Next, the model encodes the unmasked blocks through the encoder to generate feature representations. Then MAE randomly masks some blocks in the image and then lets the model reconstruct these masked pixels, thereby training a model with strong feature representation capabilities.

[0016] Step 2: Build a diffusion model-based decoder module, namely DM-decoder, to realize the process of decoding the tracking box sequence from the noise; The construction process of the decoder module based on the diffusion model is as follows: Step 2.1: Establish a noise adding module for the diffusion model, and add noise to the pre-given tracking box sequence S0. The noise adding process can simulate the process of gradually adding noise from the real data distribution until the data becomes completely randomized. It is usually implemented by a noise adding function, which gradually converts the original data into noisy data. This process can be carried out in multiple steps, and a certain amount of noise is added in each step until the data becomes sufficiently "fuzzy" or "random". Its mathematical representation is as follows: ; in, is the coordinate sequence with Gaussian noise added to timestamp t, and is the original coordinate sequence input without adding Gaussian noise, I is the identity matrix, is Gaussian noise. Represents a Gaussian distribution. is a hyperparameter that is determined by the continuous product of the noise variance βs. For the noise addition process, at each timestamp, the noise variance schedule β1,β2,…,βt with independent and identically distributed (iid) Gaussian noise is added to on, to form In the present setting, the bounding box ([x, y, w, h]) is used as ; Step 2.2: Establish a noise de-noising module for the diffusion model. This is the opposite operation to the noise adding process. It aims to gradually recover the original data from the noise data. It restores the noise-processed frame to the original frame through the neural network iteration by iteration. It can learn the joint distribution of the tracking frame and the original input image, so as to achieve the iterative generation of the tracking frame of the required target during the inference process. Figure 1As shown in Figure 1, after the sequence is denoised, it passes through an instance adaptation layer and selects the noise region in the feature map as the feature vector. Then, the multi-head self-attention module (MSA and MCA) and the feed-forward neural network (FFN) are used to fuse the noise region feature vector with the shared feature vector from the backbone network. P , t is a randomly generated vector, which is mapped and combined with the output of the multi-head self-attention module, and finally outputs the parameter value of the de-noised target box through the feed-forward neural network FFN. The L1 loss function is used as a measure between the de-noised value and the true value. The original coordinate sequence without Gaussian noise is input into the denoising module of the diffusion model in step 2.1 and becomes , then The noise de-noising module of the diffusion model after step 2.2 becomes , where LadderEnc and DMDec are the backbone network and DM decoder based on ViT. and is used to calculate the loss function, L D That is the L1 distance loss; the specific formula is: , , in, Enter the position code for the template branch, Enter the location code for the search area branch.

[0017] Step 3: Optimize parameters and loss functions to ensure the stability and convergence of the training process; The design of the training optimization process is as follows: Step 3.1: For the optimization design of the loss function of the tracking network, the L1 distance loss is used to measure the consistency between the sequence after denoising and the sequence before denoising during the diffusion denoising process. At the same time, the cross entropy classification loss and the intersection-over-union positioning loss are added to further enhance the positioning ability. The relevant formula is as follows: ; in, is the true label, n is the number of categories, is the learned probability.

[0018] ; in, is the bounding box predicted by the model, is the true bounding box.

[0019] ; Where N is the number of samples, is the predicted value, is the true value.

[0020] In the process of optimizing network training, AdamW was selected as the optimizer to enhance the robustness of loss function learning and accelerate convergence.

[0021] Step 3.2: COCO, TrackingNet, LaSOT and GOT10k are used as the training basis. In the generation stage of training labels, the present invention pays special attention to the generation results of the sequence. Unlike the traditional tracker that only considers the positive and negative samples of the pixel feature vector inside and outside the bounding-box, this serialized tracking is closer to the generation of natural language, thereby reducing interference in the image.

[0022] Step 3.3: In order to ensure the convergence effect and training efficiency of the network, the initial value of the learning rate 1e-4, the decay strategy (step descent strategy), the number of iterations (200) and the update threshold (0.7) were finely adjusted according to the size and complexity of the data set. This step not only improves the learning efficiency of the network, but also ensures the stability and reliability of the tracking performance.

[0023] Step 4: Reasoning process given the initial box; The tracking reasoning process in the initial state is specifically as follows: Step 4.1: First, input the 0th frame image with a specific initial bounding box into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) and width (width) and height (height). The backbone network of the template branch outputs the initial template feature map; Step 4.2: In the subsequent target tracking task, the subsequent frames in the video except the 0th frame are sequentially input into the search area branch, and a tracking frame sequence is randomly sampled in Gaussian noise, and this tracking frame is input into the decoder network, and the tracking frame where the final target is located is iteratively generated. The reasoning strategy of DDIM is adopted. Specifically, the reasoning strategy of DDIM (DenoisingDiffusion Implicit Models) is mainly based on its unique denoising process. The core principle of DDIM is to get rid of the limitation of Markov chain, so that it is not necessary to denoise step by step during sampling, but to skip denoising, thereby speeding up the sampling speed. In the reasoning process of DDIM, the model first constructs a dynamic graph, which consists of a series of subgraphs, each of which corresponds to a task. The subgraphs are connected through shared nodes, which can be shared and used by different subgraphs. This design allows the parameters of the model to be shared between different tasks, thereby reducing the complexity of the model.

[0024] In this embodiment, the results based on the dataset of general tasks show the advanced performance of the tracker of the present invention. It is worth noting that Figure 2 The example tracking capabilities of the method of the present invention in the face of major challenges such as target disappearance, interference from similar objects, and background clutter are demonstrated. The algorithm of the present invention is compared with the latest Seqtrack and Ostrack, where the Bounding Box represents the true value. In the first row, the tracker of the present invention demonstrates the ability to accurately locate the target in the presence of interference from objects similar to the target and background clutter. The second row emphasizes the ability of the tracker of the present invention to quickly locate the target after the target re-enters the field of view. The last row shows that in a scene full of objects similar to the target, the tracker of the present invention can accurately identify and track the target, thereby demonstrating its excellent target discrimination ability. At the same time, as Figure 3 As shown, it quantitatively describes its overall tracking performance (all) and tracking performance of different attributes under long-term tracking, such as background interference (Background Cluster) and perspective change (Aspect Ratio Change). It can be observed that the method of the present invention is always in a leading position in tracking performance.

[0025] The above is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. As long as the implementation method of general task tracking by using the diffusion model belongs to the protection scope of the present invention.

Claims

1. A serialization tracking method based on diffusion modeling, characterized in that: The method comprises the following steps: Step 1: Build a backbone network based on ViT to preliminarily extract features and implement image encoder pre-training; the backbone network includes a template branch and a search area branch; Step 2: Build a decoder module based on the diffusion model to realize the process of decoding the tracking box sequence from the noise; Step 3: Optimize parameters and loss functions to ensure the stability and convergence of the training process; Step 4: First, input the 0th frame image with a specific initial bounding box into the template branch. This bounding box is defined by a set of parameters, including the center position coordinates (x, y) as well as the width w and height h. The backbone network of the template branch outputs the initial template feature map. Subsequently, the subsequent frames in the video except the 0th frame are input into the search area branch in sequence, and a tracking box sequence is randomly sampled in Gaussian noise. This tracking box is input into the decoder module, and the final tracking box of the target is iteratively generated.

2. According to the method of claim 1, the serialization tracking method based on diffusion modeling is characterized in that: Step 1 specifically includes: Step 1.1: Select pure attention ViT as the backbone network for feature extraction. The inputs of the template branch and the search area branch are concatenated and input into the ViT backbone network. The ViT backbone network performs a block operation, dividing the input image into blocks of fixed size and treating each block as a word. Then, each block is converted into a high-dimensional vector through a linear embedding layer to obtain a block embedding sequence. Step 1.2: The obtained block embedding sequence is then fed into a series of attention encoding layers. Each attention encoding layer contains a multi-head self-attention module and a feed-forward neural network FFN. The output of each attention encoding layer is connected to the ladder layer for domain adaptation based on the tracking dataset. The ladder layer consists of only 3 layers of FFN, where the input of each layer and the output of the previous layer pass through an FFN layer to obtain the shared feature F P ; Step 1.3: For the pre-trained backbone network, retain its pre-trained network model parameters.

3. A serialization tracking method based on diffusion modeling according to claim 2, characterized in that: Step 2 includes: Step 2.1: Establish a noise adding module for the diffusion model, and add noise to the pre-given tracking box sequence S0. This is achieved through a noise adding function, which gradually converts the original data into noisy data. This is done through multiple steps, and a certain amount of noise is added in each step. Its mathematical expression is as follows: ； in, is the coordinate sequence with Gaussian noise added to timestamp t, and is the original coordinate sequence input without adding Gaussian noise, I is the identity matrix, is Gaussian noise, represents a Gaussian distribution, is a hyperparameter, which is determined by the continuous product of the noise variance βs. For the noise addition process, at each timestamp, the noise variance schedule β1,β2,…,βt with independent and identically distributed Gaussian noise is added to on, to form , where the bounding box [x,y,w,h] is used as ; Step 2.2: Establish a noise de-noising module for the diffusion model. After adding noise, the noise area is selected as the feature vector in the feature map after passing through an instance adaptation layer. Then the multi-head self-attention module MSA and MCA and the feed-forward neural network FFN are used to fuse the noise area feature vector with the shared feature F from the backbone network. P , t is a randomly generated vector, which is combined with the output result of the multi-head self-attention module after mapping, and finally the parameter value of the denoised target box is output through the feedforward neural network FFN, where the L1 loss function is used as a measure between the denoised value and the true value.

4. The serialization tracking method based on diffusion modeling according to claim 1 is characterized in that: Step 3 includes: Step 3.1: For the optimization design of L1 loss function, L1 distance loss is used to measure the consistency between the sequence after denoising and the sequence before denoising during the diffusion denoising process. At the same time, cross entropy classification loss and intersection-over-union positioning loss are added to further enhance the positioning ability. The relevant formula is as follows: ； in, is the true label, n is the number of categories, is the learned probability; ； in, is the bounding box predicted by the model, is the true bounding box; ； Where N is the number of samples, is the predicted value, is the true value; In the process of optimizing network training, AdamW was selected as the optimizer to enhance the robustness of loss function learning and accelerate convergence; Step 3.2: Four datasets, COCO, TrackingNet, LaSOT, and GOT10k, are used as the training basis; Step 3.3: Adjust the initial value of the learning rate, attenuation strategy, descent strategy, number of iterations and update threshold to improve the learning efficiency of the network.

Images