A target tracking method based on local response maximum likelihood modeling

Abstract

The application discloses a target tracking method based on local response maximum likelihood modeling. First, a convolutional neural network backbone is selected, second, a module based on local response maximum likelihood modeling is built, including a local maximum likelihood value extraction module, a local maximum likelihood value correction module and a local maximum likelihood value selection module model, and a suitable optimization method and a loss function are selected. In the 0th frame, only the initial template frame input is used as a reference, the template branch of the network is input to obtain the corresponding feature map, the subsequent frame is input to the search area branch to obtain the corresponding feature map, and the correlation measurement is completed with the feature map of the 0th frame, so that the position of the maximum response value is regarded as the position of the target. The method achieves excellent performance on five public tracking datasets, balances the tracking speed and accuracy, and the inference speed can reach 81 frames per second, meeting the real-time requirements of actual scenes and exceeding similar accuracy algorithms.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning, and specifically relates to a target tracking method based on local response maximum likelihood modeling. Background Technology

[0002] Target tracking is a fundamental task in computer vision, with wide applications in human-computer interaction, visual surveillance, and autonomous driving. Its main objective is to estimate and identify the possible states of a target across a series of frames using only information from the initial frame. Most current algorithms employ Transformer-based Siamese models, where template matching has proven effective empirically, and all adhere to the assumption that the target's response map follows a Gaussian distribution. However, traditional Transformer trackers fundamentally fail to meet this requirement, primarily due to feature mismatches arising from challenges such as target deformation, partial occlusion, and scale variations. This problem becomes even more pronounced when employing self-attention mechanisms, as their global modeling capabilities incorporate information from noisy regions.

[0003] Transformer-based Siamese network models have garnered significant attention for their outstanding performance in tracking tasks. They leverage self-attention and cross-attention mechanisms to handle and expand dependencies between feature maps, facilitating more efficient information exchange. Trackers like TransT, STARK, STMTrack, DTT, AiATrack, and Mixformer employ Transformers to improve tracking accuracy, aiming to approximate a perfect Gaussian distribution between bi-branch matches during training. However, achieving an accurate Gaussian approximation of the tracking response distribution remains an ongoing challenge. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a target tracking method based on local response maximum likelihood modeling. The core of this method is the proposed modeling based on local response maximum likelihood. This approach aims to solve the problem that previous trackers could not achieve approximate global Gaussian fitting.

[0005] In fact, the approximation of the Gaussian distribution in current trackers is often affected by factors such as target deformation and partial occlusion during inference. Because the training process focuses solely on predicting an ideal Gaussian distribution, a mismatch arises between the training and inference processes, making this problem inherent in existing trackers. This issue primarily stems from the failure to consider the differences between the training and inference phases when approximating a global Gaussianized response map. Therefore, it's intuitive to consider modeling this imbalance during training by using lossy modeling of the Gaussian relationship, rather than simply fitting a perfect Gaussianized response map during training.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a target tracking method based on local response maximum likelihood modeling, comprising the following steps:

[0007] Step 1: Select a suitable lightweight backbone network as the feature extractor, and select appropriate pre-training parameters; the backbone network is based on the Siamese network architecture, and the input of the backbone network is divided into template branches and search region branches;

[0008] Step 2: Construct the attention module; the attention module based on the branch connection of the search region includes a local maximum likelihood extraction module, a local maximum likelihood correction module, and a local maximum likelihood selection module. The output feature map of the template branch is input to the local maximum likelihood extraction module, the local maximum likelihood correction module, and the local maximum likelihood selection module to obtain the feature map; the attention module based on the branch connection of the search region obtains the feature map through the attention encoding module and the attention decoding module, and the two feature maps are concatenated as input;

[0009] Step 3: Build the tracking head module. Input the above-stitched feature map. The tracking head module is divided into convolutional branches and fully connected branches. Each branch has a classification head and a regression head. Define the input mode for classification and regression problems.

[0010] Step 4: Design the mathematical expression for the loss function, define the optimization method and optimization parameters, and ensure stable gradient descent is achieved;

[0011] Step 5: Define the tracking inference process, define the invariance of the 0th frame image and the variability of subsequent frames other than the 0th frame, and achieve stable tracking on various general tracking datasets.

[0012] The beneficial effects of this invention are:

[0013] Experiments on GOT-10k, LaSOT, OTB100, UAV123, and TrackingNet datasets demonstrate that the proposed tracker using local response maximum likelihood modeling achieves state-of-the-art tracking performance in both short-term and long-term tracking scenarios. Furthermore, it exhibits superior performance in challenging scenarios such as rapid transformations and target deformation. This invention balances the consistency between training and inference in tracking through local response modeling and employs maximum likelihood modeling to reduce Transformer sparsity while enhancing the target modeling capability of local responses, thereby improving the tracker's accuracy and robustness in complex scenarios.

[0014] This invention constructs an attention module based on local maximum likelihood modeling. This module includes a local maximum likelihood extraction module, a local maximum likelihood correction module, and a local maximum likelihood selection module. The goal of these three modules is to achieve lossy Gaussian approximation, thereby replacing the original global lossless Gaussian matching and reducing the difficulty and instability of tracking model training. Simultaneously, appropriate optimization methods and loss functions are selected to ensure stable gradient descent, thus reducing the classification and regression losses to near zero. This method achieves excellent performance on five publicly available tracking datasets, balancing tracking speed and accuracy. Its inference speed reaches 81 frames per second, not only meeting the real-time requirements of practical scenarios but also far exceeding the accuracy of similar algorithms. Attached Figure Description

[0015] Figure 1 This is a flowchart of a target tracking method based on local response maximum likelihood modeling according to the present invention;

[0016] Figure 2 This is a schematic diagram of the local maximum likelihood extraction module;

[0017] Figure 3 This is a general data qualitative diagram illustrating the modeling method proposed in this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0019] like Figure 1 The diagram shown is a flowchart of a target tracking method based on local response maximum likelihood modeling according to the present invention. Figure 1 As shown, the tracking network used in this method consists of a backbone network module, an attention module, and a tracking head module. The input image passes through these three modules sequentially, specifically including:

[0020] Step 1: Build the backbone network module as the feature extractor. The backbone network is based on the Siamese network architecture. The input of the backbone network module is divided into template branch and search region branch.

[0021] Step 2: Construct the attention module; the attention module based on branch connections in the search region includes a local maximum likelihood extraction module, a local maximum likelihood correction module, and a local maximum likelihood selection module. The output feature map of the template branch is input to the local maximum likelihood extraction module, the local maximum likelihood correction module, and the local maximum likelihood selection module to obtain the feature map; the attention module based on branch connections in the search region obtains the feature map through an attention encoding module and an attention decoding module, and the two feature maps are concatenated as input;

[0022] Step 3: Build the tracking head module. Input the above-concatenated feature map. The tracking head module is divided into convolutional branches and fully connected branches. Each branch has a classification head and a regression head.

[0023] Step 4: Design the mathematical expression for the loss function, define the optimization method and optimization parameters, and ensure stable gradient descent is achieved;

[0024] Step 5: Define the tracking inference process, define the invariance of the 0th frame image and the variability of subsequent frames other than the 0th frame, and achieve stable tracking on various general tracking datasets.

[0025] Furthermore, step 1 includes: selecting a suitable lightweight backbone network as the feature extractor, and simultaneously selecting appropriate pre-training parameters; the lightweight backbone network is designed as follows:

[0026] Step 1.1: The lightweight pure convolutional network ConvNext is selected as the core network framework for feature extraction. This network framework selects the first four stages of ConvNext (defined by the native ConvNext and in...) Figure 1 The last three stages are given in the text, and each stage uses the same basic module. Specifically, the structure of each module includes a 7×7 depthwise convolution and two 1×1 pointwise convolutions. The number of modules in the different stages are 3, 3, 9, and 3, respectively, and the feature dimensions of the initial inputs to the template branch and the search region branch of the backbone network are respectively... Each stage includes a downsampling module, consisting of a layer regularization and a 2D convolution module. After each downsampling module, the features are processed by a 7×7 channel-wise convolution and two 1×1 pointwise convolutions, followed by a regularization module and an activation function module. The final output feature dimension is... .

[0027] Step 1.2: Select the parameters for the network pre-training. Among tiny, small, and large, the tiny version with the smallest number of parameters was selected.

[0028] Step 1.3: Load the pre-trained parameters of the tiny version as the initial parameters for tracking training, and select a step-decreasing learning rate strategy to treat the training process as a process of fine-tuning the neural network.

[0029] Furthermore, step 2 includes: constructing an attention module, which comprises a local maximum likelihood extraction module, a local maximum likelihood correction module, and a local maximum likelihood selection module. These three modules are all based on the search region branch. For the template branch, features are input into the attention encoding module, which maintains the same formula as the attention module:

[0030] ;

[0031] Where Q, K, and V are the input queries, key-value pairs and vectors, respectively. The scaling weights are used, and softmax is the normalization function.

[0032] The construction process of the local maximum likelihood extraction module, the local maximum likelihood correction module, and the local maximum likelihood selection module is as follows:

[0033] Step 2.1: Construct a Local Maximum Likelihood (LML) extraction module. This module divides the input feature map into different clustering regions (foreground and background). A clustering module based on similarity comparison is also established, dividing the input feature map into different clustering regions. Attention is calculated for each region, and the feature vector with the maximum response is selected based on the attention similarity map. Specifically:

[0034] Specifically, such as Figure 2 The clustering and attention operations in the code involve calculating attention for each cluster region and selecting the feature vector with the largest response based on the attention similarity map. Specifically, in each iterative clustering stage, the input vector is fairly divided into two different clusters. In the k-th hierarchical level, each subset is divided k times. The specific measurement process is as follows:

[0035] ;

[0036] ;

[0037] in, y1 is the input vector, y2 is the cluster center, and y2 is the average value of each cluster, which makes similar vectors in the feature domain classified into the same cluster set. This is the definition of similarity, which is defined by the cosine function. , It is the dimension accumulated from 1 to d. It is the input feature vector.

[0038] Considering that the global response is difficult to fully fit the target information, the global response is treated as a regional aggregation variable of local response points. The maximum likelihood portion of the local response is selected to modify the features of the local response, and the local discrete features are re-measured. Under the premise that different cluster centers are defined as different individual clusters, the attention response metric is as follows:

[0039] ;

[0040] in, and It contains the query and key for each individual cluster. Using dimensionality values ​​and Softmax as the normalization function, two methods for correcting local responses are designed: index-guided and value-guided. For index-guided, values ​​falling within the local region are set to 1, and values ​​outside the local region are set to 0, forming a binary mask. For value-guided, values ​​within the local region are set to the learning parameters for local attention, while values ​​outside the local region remain set to 0, forming a soft mask. These masks are applied to the maximum likelihood approximation as learnable parameter values ​​to strengthen specific attention regions. The final result is unfolded into a complete attention map and passed to... Figure 2 The right half, from Figure 2 Clustering is divided into multiple local attention and value / index guidance, and it can be seen that the local maximum likelihood value is extracted. Finally, the local maximum likelihood value extraction module is realized through the clustering module and attention mechanism.

[0041] Step 2.2: As Figure 2 As shown, a local maximum likelihood correction module and a local maximum likelihood selection module are constructed. The global response is used as a regional aggregation variable for local response points. The maximum likelihood portion of the local response is selected to correct the features of the local response, thus re-measuring the local discrete features. The specific implementation is as follows:

[0042] Considering the regional limitations of the global response feature vector on the local response feature vector—that is, responses falling within a local region are more important than those falling outside—a process is designed to select the maximum response within the global region based on this assumption. The TopK function (extracting the K largest values ​​from each row of the matrix and their corresponding positions) is used as a selector for global sparse information. These maximum values ​​are selected based on global attention. The indices generated by the TopK function are passed to subsequent index-guided attention. For index-guided attention, values ​​within each cluster are set to 1, while values ​​outside the cluster are set to 0. The indices passed by the TopK function and the index-guided attention are combined to apply to the target features. Specifically, the TopK function generates an index mask, which can be applied to local regions as a selection condition; that is, the values ​​corresponding to the indices are selected. The selected features are then fused with the features proposed by the TopK function to strengthen the local response features. This involves using the TopK function's index-optimized values / index attention, followed by fusion. Finally, the features are input into the attention decoding module.

[0043] Step 3: Build the tracking head module and define the input patterns for classification and regression problems;

[0044] The process of building the tracking head module is as follows:

[0045] Step 3.1: To address the fuzzy matching problem between the fixed-parameter bounding boxes and targets encountered in traditional methods, a dual-head tracking module is designed, consisting of convolutional and fully connected branches. Each branch learns both classification and regression losses, which are then fused together according to given parameters. Based on the feature maps of the classification and regression heads, for each location (x, y), a corresponding mapping location in the input image can be found. These mapping locations combine to form image patches. Subsequently, these specific image patches are directly classified, and the target bounding box is regressed at the corresponding location. This method treats each location as an independent training sample for direct processing.

[0046] Step 4: Design the mathematical expression for the loss function, define the optimization method and optimization parameters, and ensure stable gradient descent is achieved;

[0047] The process of constructing the loss function and optimization method is as follows:

[0048] Step 4.1: To design an effective loss function on the tracking network, a joint optimization method was adopted, combining classification loss and regression IOU loss. Specifically, cross-entropy classification loss, IOU localization loss, and L1 distance metric loss were used.

[0049] ;

[0050] in These are the actual labels, and n is the number of categories. It is the probability of learning.

[0051] ;

[0052] in It is the bounding box predicted by the model. It is a real bounding box.

[0053] ;

[0054] Where N is the number of samples. It is a predicted value. It is the actual value.

[0055] The combined application of these loss functions ensures that the network can accurately identify and locate the target position during training. To improve the learning stability and convergence of the loss functions, the AdamW optimizer is used. This approach ensures that the network can robustly learn the target tracking task during training.

[0056] Step 4.2: During training, the designed tracking network selected six datasets—COCO, TrackingNet, LaSOT, GOT10k, ILSVRC DET, and ILSVRC VID—as training datasets to improve the model's adaptability. When generating training labels, a simplified method was used: all pixels inside the ground truth bounding boxes were selected as positive samples, while all pixels outside the ground truth bounding boxes were considered negative samples. Positive and negative samples were randomly selected for training through sampling. This positive-negative sample partitioning allows for more accurate training of the tracking network. Simultaneously, this training strategy helps improve the model's robustness and generalization ability, enabling it to perform well in various real-world scenarios.

[0057] Step 4.3: To ensure the convergence capability of the network, an initial value for the learning rate and a decay strategy were designed. After adjusting the relevant parameters multiple times, 20 rounds were selected as the number of training batches, and a stepwise learning strategy was adopted. In the 10th training batch, the learning rate was adjusted from 1e-4 to 1e-5 to ensure that the network can reach a stable convergence state. These adjustments and optimizations aim to improve the efficiency and stability of training, thereby ensuring that the tracking model can achieve the best performance during training.

[0058] Step 5: Define the tracking inference process, define the invariance of the 0th frame image and the variability of subsequent frames other than the 0th frame, achieve stable tracking on various general tracking datasets, and reduce the impact of challenging problems such as scale transformation and similar interference.

[0059] The process of building the tracking reasoning process is as follows:

[0060] Step 5.1: In the tracking task, the image of frame 0, along with the parameters of the initial bounding box and the target location, is input into the template branch. After processing by the backbone network, attention module, and classification and regression heads, stable features of the image of frame 0 can be obtained. These features contain basic information about the target, which helps the network accurately identify and track the target.

[0061] Step 5.2: Input the images after frame 0 into the search region branch in sequence. Each frame image passes through the backbone network, attention module, classification and regression head to obtain the feature map that changes in each frame.

[0062] Step 5.3: Determine the location of the target by measuring the similarity between the template branch and the search branch. The location selected by the Top-K function is the location of the target.

[0063] In this example, results based on datasets such as TrackingNet, LaSOT, and GOT10k demonstrate the advanced performance of the tracker of this invention. Figure 3 As shown, the local response maximum likelihood modeling proposed in this invention demonstrates excellent tracking performance in challenging scenarios involving similar interference, partial occlusion, background interference, and viewpoint changes. PMTrack, based on this invention, achieved state-of-the-art performance that year on the short-term OTB dataset with 91.2% accuracy and 70.6% success rate. Simultaneously, on the long-term TrackingNet dataset, PMTrack achieved an AUC of 81.8% compared to state-of-the-art trackers, and showed performance improvements of 0.1% and 0.4% compared to popular algorithms SparseTT and TransT, respectively.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any method that enhances local response through global modeling for target tracking tasks and then implements single-target tracking based on attention falls within the scope of protection of the present invention.

Claims

1. A target tracking method based on local response maximum likelihood modeling, characterized in that, Includes the following steps: Step 1: Select a suitable lightweight backbone network as the feature extractor, and select appropriate pre-training parameters; the backbone network is based on the Siamese network architecture, and the input of the backbone network is divided into template branches and search region branches; Step 2: Build the attention module; The attention module based on search region branch connections includes a local maximum likelihood extraction module, a local maximum likelihood correction module, and a local maximum likelihood selection module. The output feature map of the template branch is input to the local maximum likelihood extraction module, the local maximum likelihood correction module, and the local maximum likelihood selection module to obtain a feature map. The attention module based on search region branch connections obtains a feature map through an attention encoding module and an attention decoding module, and the two feature maps are concatenated as input. Step 3: Build the tracking head module. Input the above-stitched feature map. The tracking head module is divided into convolutional branches and fully connected branches. Each branch has a classification head and a regression head. Define the input mode for classification and regression problems. Step 4: Design the mathematical expression for the loss function, define the optimization method and optimization parameters, and ensure stable gradient descent. Step 5: Define the tracking inference process, define the invariance of the 0th frame image and the variability of subsequent frames other than the 0th frame, and achieve stable tracking on various general tracking datasets; Step 2 includes: Step 2.1: Establish a local maximum likelihood extraction module and a clustering module based on similarity comparison. Divide the input feature map into different clustering regions, perform attention calculation for each region, and select the feature vector with the maximum response based on the attention similarity map. Step 2.2: Establish a local maximum likelihood correction module and a local maximum likelihood selection module. The global response is used as a regional aggregation variable of the local response points. The maximum likelihood part of the local response is selected to correct the features of the local response, and the local discrete features are re-measured. The maximum response is selected in the local region and finally input into the attention decoder.

2. The target tracking method based on local response maximum likelihood modeling according to claim 1, characterized in that, Step 1 includes: Step 1.1: ConvNext, a pure convolutional network, is selected as the backbone network. ConvNext consists of four stages, each using the same basic modules. Each basic module includes channel-wise convolution and pointwise convolution. The feature dimensions initially input to the template branch and the search region branch of the backbone network are respectively... Each stage includes a downsampling module, consisting of a regularization layer and a 2D convolution module. After each downsampling module, the features undergo channel-wise convolution and point-wise convolution, a regularization module, an activation function module, a template branch, and a search region branch. The final output feature dimension is... ; Step 1.2: Select the parameters for network pre-training, choosing the version with the minimum number of parameters; Step 1.3: Load the pre-trained parameters of this version as the initial parameters for tracking training, and select a step-decreasing learning rate strategy to treat the training process as a process of fine-tuning the neural network.

3. The target tracking method based on local response maximum likelihood modeling according to claim 1, characterized in that, Step 3 includes: Step 3: Design a dual-head tracking head module, which is divided into a convolutional branch and a fully connected branch. Each branch learns classification loss and regression loss together and fuses the losses according to the given parameters. On the feature maps of the classification head and regression head, for each position (x, y), find the corresponding mapping position in the input image. These mapping positions are combined to form an image patch. Then, the image patch is directly classified and the target bounding box is regressed at the corresponding position.

4. The target tracking method based on local response maximum likelihood modeling according to claim 3, characterized in that, Step 4 includes: The process of constructing the loss function and optimization method is as follows: Step 4.1: Employ a joint optimization method, combining classification loss and regression IOU loss. Specifically, use cross-entropy classification loss, IOU localization loss, and L1 distance metric loss: ； in These are the actual labels, and n is the number of categories. It is the probability of being learned; ； in It is the bounding box predicted by the model. It is a real bounding box; ； Where N is the number of samples. It is a predicted value. It is the actual value; Step 4.2: During the training process, six datasets were selected as training datasets: COCO, TrackingNet, LaSOT, GOT10k, ILSVRC DET, and ILSVRC VID. Step 4.3: Design the initial value of the learning rate and the decay strategy. The network was trained by adjusting the relevant parameters multiple times.

5. The target tracking method based on local response maximum likelihood modeling according to claim 4, characterized in that: Step 5 includes: Step 5.1: In the tracking task, the parameters of the 0th frame image, the initial bounding box, and the target location are input into the template branch. After processing by the backbone network and the attention module, the feature map of the 0th frame image is obtained. Step 5.2: Input the images after frame 0 into the search region branch in sequence. Each frame image passes through the backbone network and attention module of the search region branch to obtain the feature map that changes in each frame. Step 5.3: Determine the location of the target by measuring the similarity between the template branch and the search branch. The location selected by Top-K is the location of the target. Step 5.4: Based on the predicted position coordinates, select the image patch to be input in the image of frame t+1 where the target may appear, repeat the feature extraction process of the search region branch, keep the template branch features unchanged, and repeat until tracking is completed.

Images