A target tracking method based on a local and global perspective compatible Transformer structure

Abstract

A target tracking method based on a local and global perspective compatible Transformer structure.The Transformer structure of the present application adopts a network structure of ViT-Base, takes a local search region and a global search region cropped according to a tracking position of a previous frame as inputs, and shares weights of a feature extraction and feature fusion network of two search branches, so as to realize local and global perspective target tracking under a unified model.A feature encoding network based on the Transformer is used to obtain a feature vector of a tracking frame, and when a regression quality prediction score of a local tracking result is low, a tracking frame with the minimum feature vector distance to a given target of the first frame is selected from multiple results of global and local tracking as the final tracking result of the current frame; when the regression quality prediction score of the local tracking result is high, the local tracking result is used as the final tracking result of the current frame.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning and target tracking, and relates to a classification algorithm ViT-Base based on a Transformer architecture, in particular to a target tracking method based on a Transformer structure and compatible with local and global perspectives. BACKGROUND

[0002] Given the state of a target to be tracked in the initial frame of a video (including position coordinates and length-width data), a target tracking algorithm will track the specified target in subsequent continuous video frames. The target is not limited by species and can be any selected object. In view of the situation that the target may not be visible in the local field for a period of time due to factors such as occlusion, out-of-frame, rapid movement, etc. in a long period of video, the algorithm should also have the ability to re-detect the target in the global region. In most cases, continuous target tracking in the local region and target re-detection in the global region require multiple modules to cooperate or rely on a sliding window mechanism to complete, such as the LTMU and SPLT algorithms. At the same time, for online updating of the state decision, current algorithms mostly provide decision clues in combination with fixed time intervals and confidence thresholds, such as the STARK and MixFormer algorithms, which have a higher risk of updating for long videos. SUMMARY

[0003] The application aims to provide a target tracking method based on a Transformer structure and compatible with local and global perspectives, so that one model has both continuous tracking capability in a short tracking scene and target re-detection capability in a long tracking scene. When the confidence of the local tracking result is low, the tracking box with the smallest distance to the feature vector of the given target in the first frame can be selected from multiple tracking results of global and local tracking as the final tracking result of the current frame. At the same time, dynamic template updating can provide rich changes in target appearance in the time dimension, improve the accuracy of tracking, and improve the comprehensive performance of tracking.

[0004] The technical solution of the application is as follows:

[0005] A target tracking method based on a Transformer structure and compatible with local and global perspectives, comprising the following steps:

[0006] Step 1: Given the bounding box b of the target to be tracked in the initial frame, wherein b = {x, y, w, h}, obtain the local tracking candidate result box and multiple global tracking candidate result boxes of the current video frame through a local-global tracker based on a Transformer structure, and the regression quality prediction scores of these tracking candidate result boxes; init

[0007] The local-global tracker structure based on a Transformer structure is specifically as follows:​

[0008] The structure of the local-global tracker is composed of a joint feature extraction and feature fusion network, a regression box prediction network, and a regression quality prediction network; the joint feature extraction and feature fusion network is composed of a ViT-Base structure based on a Transformer; a first frame target position is expanded by a certain proportion in all directions, and a template region containing the target to be tracked as the center is obtained by cropping; the position of the target to be tracked in the previous frame is expanded in the current frame by a certain proportion to obtain a target search region of the current frame; the target search region and the template region are divided into image blocks of the same size with 16 as the interval, and the image blocks of the target search region and the template region are encoded, and the encoded image blocks are sequentially spliced and sent into a Transformer encoding layer with a learnable object query, to complete feature extraction and fusion of the target search region features and the template region features; then, the target search region features with the target distinguishing ability are sent into the regression box prediction network to obtain a predicted regression box; the object query feature is input into the regression quality prediction network; before the regression box prediction network, the object query feature is weighted to obtain the one-to-one correspondence between the target search region features and the predicted regression box; and thus, the target tracking results in the local and global search regions and the predicted scores are obtained.

[0009] The step 1 specifically includes the following processing steps:

[0010] (1) Preprocessing: the local search region needs to crop and scale the input image based on the last frame tracking result, so that the picture size is the same as the network input size; the global search region needs to scale and pad the input image, so that the picture size is the same as the network input size;

[0011] (2) Network forward inference to obtain a predicted tracking candidate result box and a corresponding regression quality prediction score;

[0012] Step 2: when the regression quality prediction score of the local tracking candidate result box is higher than a set threshold, the local tracking result is used as the final tracking result of the current frame; when the regression quality prediction score of the local tracking candidate result box is lower than the set threshold, the tracking candidate result box with the minimum distance to the feature vector of the initial frame target b init given in the first frame is selected from the global tracking candidate result box and the local tracking candidate result box as the final tracking result of the current frame; the feature vector is extracted by a feature encoding network from the tracking candidate result box, and each tracking candidate result box corresponds to a 256-dimensional feature vector; wherein, the regression quality prediction score of the tracking candidate result box is lower than a set threshold τ filterThe tracking candidate result frame is defined as a low-quality regression frame, and is not involved in the screening of the final tracking result;

[0013] Step 3: According to the regression quality prediction score of the current and historical frames, it is decided whether the tracking state of the current frame is suitable for updating the dynamic template;

[0014] The local tracking result of the t-th frame is denoted as The regression quality prediction score thereof is denoted as The initialization target frame of the first frame is b init , the template clipping function is denoted as p(·), and the model forward inference is denoted as θ(·); when the following three conditions are met, the frame state is used for dynamic template updating; otherwise, it is not updated.

[0015]

[0016]

[0017]

[0018] Wherein, τ high is a prediction score threshold, τ back is an IoU threshold.

[0019] Step 4: Steps 1-3 are performed for each frame until the end of the video tracking.

[0020] Advantages of the present application:

[0021] (1) The local-global tracker can unify the continuous target tracking ability of the focused local area and the global target redetection ability in the case of target loss or invisibility into one model, and obtain certain benefits from the integrated training strategy.

[0022] (2) The regression quality prediction branch provides a relatively robust score, which can activate the full-image detection branch after the target is lost or invisible, and the feature vector provided by the feature encoding network can participate in the secondary scoring of the candidate tracking frame, and finally lock the candidate frame with the smallest feature vector distance to the given target of the initial frame as the redetection result. The designed dynamic template updating strategy better captures the appearance changes of the target in the time dimension and other information, and helps improve the overall tracking performance. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a structural diagram of a local-global tracker based on a Transformer structure.

[0024] Figure 2 is a structural diagram of a feature encoding model based on a Transformer structure. DETAILED DESCRIPTION

[0025] The specific embodiments of the present application are further illustrated below in conjunction with the accompanying drawings and technical solutions.

[0026] Figure 1 The structure diagram of the local-global tracker based on the Transformer structure. The encoding layer of the image block is composed of convolution; the Transformer encoding layer is composed of 12 layers of Block modules, each Block containing an Attention module structure and an MLP module structure. The final local-global tracker will obtain the regression prediction box of the current frame local region tracking and its corresponding regression quality prediction score. When the regression quality prediction score of the local tracking result is higher than the set threshold, the local tracking result is used as the final tracking result of the current frame; when the regression quality prediction score of the local tracking result is lower than the set threshold, the tracking box with the smallest distance to the feature vector of the given target of the first frame is selected from the multiple results of global and local tracking as the final tracking result of the current frame. Figure 2 The structure diagram of the feature encoding model based on the Transformer structure. Its basic structure is the same as that of Figure 1 The learned to-be-queried target feature will comprehensively express the features of the candidate box region, and finally obtain the feature vector of the candidate box region. The candidate box with the closest distance to the initial frame target region feature will be identified as the final tracking result of the current frame. Finally, according to the regression quality prediction score and other conditions of the current and historical frames, the algorithm will decide whether to use the current frame tracking state as the update of the dynamic template.

[0027] The training set of the local-global tracker is composed of the training set parts of COCO2017, LaSOT, TrackingNet and GOT-10k, and the data augmentation adopts horizontal flip and brightness jitter. The optimizer adopts ADAMW, the initial learning rate is set to 0.0001, a total of 110 epochs are trained, the learning rate is decayed by 0.1 times in the last 30 epochs, each epoch is composed of 60000 groups of data, and each group of data contains two template regions and one local and global search region. In the training, the regression box assignment under the global region is completed by the Hungarian algorithm. The regression quality prediction branch takes the IoU of the corresponding prediction box and the true value as the prediction target. The training set of the feature encoding model is composed of the tracking results of the local-global tracker on the LaSOT training set part, the results with a true value greater than 0.7 are positive samples, and the results with a true value less than 0.1 are negative samples, and the loss function adopts the combination of classification loss and ternary loss. The optimizer adopts SGD, the initial learning rate is set to 0.0001, a total of 1000 epochs are trained, and the learning rate is decayed by 0.1 times after 500 epochs.

[0028] In the inference process, the input size of the template region is 192x192, the input size of the local search region is 384x384, and the area size of the region before scaling is 4 times and 16 times the area of the tracking target, respectively. The input size of the global search region is 640x640. Two templates are used, one for dynamic template updating and the other remaining unchanged after initialization in the first frame. The threshold for regression quality prediction score is τ high = 0.7, the verification threshold is τ back = 0.7, and the screening threshold is τ filter = 0.5. The global tracking prediction candidate box is set to 5. In the inference process of the feature encoding model, the image block cropped by the tracking candidate box will be scaled to 128x128.

Claims

1.A target tracking method based on a local and global view compatible Transformer structure, characterized in that, The steps are as follows: Step 1: Given the bounding box b of the target to be tracked in the initial frame init = {x, y, w, h}, obtain the local tracking candidate result box and multiple global tracking candidate result boxes of the current video frame by the local-global tracker based on the Transformer structure, and the regression quality prediction scores of these tracking candidate result boxes; The structure of the local-global tracker based on the Transformer structure is as follows: The structure of the local-global tracker is composed of a joint feature extraction and feature fusion network, a regression box prediction network and a regression quality prediction network; the joint feature extraction and feature fusion network is composed of a ViT-Base structure based on the Transformer; a template region containing the target to be tracked as the center is obtained by expanding the position of the target to be tracked in the first frame by a certain proportion and cropping; the search region of the current frame is obtained by expanding the position of the target to be tracked in the previous frame in the current frame by a certain proportion; the target search region and the template region are respectively divided into a plurality of image blocks of the same size with 16 as the interval, and the image blocks of the target search region and the template region are encoded, then the encoded image blocks are sequentially spliced and sent into the Transformer coding layer with the learnable query target feature, so as to complete the feature extraction and the fusion of the target search region feature and the template region feature; then the target search region feature with the target distinguishing ability is sent into the regression box prediction network to obtain the predicted regression box; the query target feature is sent into the regression quality prediction network as input; before the regression box prediction network, the query target feature will weight the target search region feature to obtain the one-to-one correspondence between the target search region feature and the predicted regression box; so as to obtain the target tracking result in the local and global search regions and the predicted score thereof; The step 1 specifically comprises the following processing steps: (1) preprocessing: the local search region needs to crop and scale the input image based on the tracking result of the previous frame, so that the picture size is the same as the network input size; the global search region needs to scale and pad the input image, so that the picture size is the same as the network input size; (2) the network forward inference obtains the predicted tracking candidate result box and the corresponding regression quality prediction score; Step 2: when the regression quality prediction score of the local tracking candidate result box is higher than a set threshold, the local tracking result is adopted as the final tracking result of the current frame; when the regression quality prediction score of the local tracking candidate result box is lower than a set threshold, the tracking candidate result box with the minimum feature vector distance from the initial frame target b init of the first frame is selected from the multiple results of the global tracking candidate result box and the local tracking candidate result box as the final tracking result of the current frame; the feature vector is extracted by a feature encoding network from the tracking candidate result box, and each tracking candidate result box corresponds to a 256-dimensional feature vector; wherein the tracking candidate result box with a regression quality prediction score lower than a set threshold τ filter is defined as a low-quality regression box and does not participate in the selection of the final tracking result. Step 3: according to the regression quality prediction scores of the current and historical frames, it is decided whether the tracking state of the current frame is suitable for updating the dynamic template; Let the local tracking result of the t-th frame be denoted as The regression quality prediction score of the t-th frame is denoted as The initialization target frame of the first frame is denoted as b init , the template cropping function is denoted as p(·), and the model forward inference is denoted as θ(·); when the following three conditions are met, the frame state is used for dynamic template updating; otherwise, it is not updated; where τ high is a prediction score threshold, τ back is an IoU threshold; Step 4: steps 1-3 are executed for each frame until the video ends.

Images