A single-target tracking method based on aggregated token-aware features

Abstract

The present invention provides a single target tracking method based on aggregated token perception features, and specifically relates to the field of vision and image processing technology. A single target tracking method based on aggregated token perception features, the method comprising the following steps: Step 1: Under single target tracking based on aggregated token perception features, an overall model consisting of a ViT‑B backbone network, an aggregated token perception feature module, a multi-scale feature fusion module and a positioning head is constructed; Step 2: Initialization: Given a data set X containing N video sequences, the first frame image and the initial bounding box position of the target are selected, and the target detection model of the pre-trained ViT‑B backbone network or the manually annotated bounding box is used to initialize the tracking model, the feature extraction and enhancement process, the template key information is obtained through the aggregated token perception feature module, the feature representation of the current frame image and the target area is accurately extracted using a deep neural network, and the cropped target area is input into the feature extraction network.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of visual and image processing, and particularly relates to a single target tracking method based on aggregated token perception features. BACKGROUND

[0002] Single target tracking is an important task in the field of computer vision, which is widely used in video surveillance, automatic driving, intelligent transportation and other fields. Researching an efficient single target tracking network has important theoretical and practical significance for promoting social production and daily life. The performance of tracking is affected by many factors, such as light, occlusion, size change and complex background interference, etc. The traditional method uses convolutional neural network to extract and match features, but there is a problem that global and local features cannot be well fused, resulting in subtle features being weakened. Vision Transformer gradually becomes a mainstream hotspot in the single target tracking task due to its advantages in capturing long-distance dependence and self-attention mechanism. However, such model usually contains a large number of parameters and computing overhead, and simple pruning of it will cause performance loss, which poses a challenge to the single target tracking task with high real-time requirement. Therefore, the research on lightweight single target tracking network has become an important direction.

[0003] At present, the lightweight single target tracking network still has some problems in performance, mainly including the following two points:

[0004] (1) Balance between precision and speed in lightweight network: In the lightweight single target tracking network, there are some deficiencies in the balance between precision and speed. This is because in the process of designing lightweight model, it is necessary to balance the model running speed and tracking accuracy. Some lightweight models may sacrifice certain accuracy in pursuit of faster speed, resulting in decreased precision in tracking target.

[0005] (2) Robustness in complex scenes: In the lightweight single target tracking network, the robustness is obviously defective in the face of complex scenes, such as dramatic changes in light, complex and changeable background, etc. This deficiency may lead to unstable performance or even failure of the tracking network in complex environments.

[0006] In summary, the present application provides a single target tracking method based on aggregated token perception features. SUMMARY

[0007] In order to solve the problems of the prior art, the present application provides a single target tracking method based on aggregated token perception features.

[0008] In order to achieve the above-mentioned purpose, the present application realizes the following technical solutions:

[0009] A single target tracking method based on aggregated token perception features, characterized in that the method comprises the following steps:

[0010] Step 1: Based on the single target tracking of the aggregated token perception features, the overall model composed of the ViT-B backbone network, the aggregated token perception feature module, the multi-scale feature fusion module and the positioning head is constructed.

[0011] Step 2: Initialization: Given the data set X containing N video sequences, the first frame image and the initial bounding box position of the target are selected, and the target detection model of the pre-trained ViT-B backbone network or the manually labeled bounding box is used to initialize the tracking model.

[0012] Step 3: Feature extraction and enhancement process, the template key information is obtained through the aggregated token perception feature module, the feature representation of the current frame image and the target region is accurately extracted by using the deep neural network, the cropped target region is input into the feature extraction network, the network will pass through multiple attention layers in turn, and the texture features from the bottom layer to the semantic features from the high layer are extracted, and finally a fixed length feature vector is output as the feature representation of the target.

[0013] Step 4: Target positioning and tracking: input the current frame features and initial target features into the tracking network, repeat the above feature extraction step, continuously process the video sequence tracking target in the subsequent frame image through the trained model, and finally output the video sequence with target position labeling.

[0014] Step 5: Model training and performance evaluation: the training of the model needs training data set and test data set, and multiple evaluation indexes are needed to evaluate the performance of the model.

[0015] In step 3, the aggregated token perception feature module can effectively aggregate the original representation information of the image for perception features, and the aggregation process of the image can be summarized as follows:

[0016] Row avg =avg(Z 11 , Z 12 , Z 13 …Z 1n )

[0017] Column_avg=avg(Z 11 , Z 21 , Z 31 …Z n1 )

[0018] Z′=Z-Row_avg+Z-Column_avg

[0019] Where Z ijrepresents the value of the i-th row and j-th column of the original image two-dimensional feature map, Row_avg represents the average value of each row of the two-dimensional feature map, column_avg represents the average value of each column of the two-dimensional feature map, avg(·) represents taking the average value, and z' represents the value finally input into the VIT after the above operation, that is,

[0020]

[0021] Through the above operation of subtracting the average value of the template feature map, the information in the feature map can be aggregated and weighted, so that the model pays more attention to the target region.

[0022] In step 1, the multi-scale feature fusion module is used to make up for the lack of feature extraction and capture the shortcomings of small targets or image subtle changes. The multi-scale feature fusion module has four different layers, each layer containing different numbers and sizes of convolutions to obtain more rich receptive fields and deeper semantic information. The output features are input into the multi-scale feature fusion module, and the outputs of each layer are fused through different layers and different size convolutions.

[0023] The above process can be expressed as:

[0024] D0=Z output

[0025]

[0026] where Z output represents the output of the template image after VIT, D i represents the output of the i-th layer in the module, and cat represents the concatenation operation.

[0027] The template feature after the above operation has deeper semantic information. In order to preserve this information, we finally perform residual connection between the results obtained by each layer and the original feature D0, and perform element-wise addition operation to obtain the final result D output :

[0028] D output =D0+D1+D2+D3+D4

[0029] The entire network model maintains the same feature extraction mode for the template image Z when performing attention calculation;

[0030] When performing attention calculation on the search image X, we continue the asynchronous feature extraction mode, and perform separate feature extraction on the search image X for the first 5 layers, as follows:

[0031]

[0032] Where Q, K and V are query matrix, key matrix and value matrix respectively. Subscript Z and X represent the matrix items belonging to template and search area respectively. Z' represents the output of template image Z after the aggregation token perception module.

[0033] On the subsequent several layers, the output D of the template image features after the multi-scale feature fusion module output The attention calculation is performed after splicing, and the publicity is as follows:

[0034]

[0035] Where Q, K and V are query matrix, key matrix and value matrix respectively. Subscript X and D represent the matrix items belonging to search area and the output of template image after the multi-scale feature fusion module.

[0036] In step 3, corner point positioning and center head positioning in target tracking are two common prediction methods for determining the position and pose of the target in the image or video. Since the corner point positioning has the problem that the bounding box is much larger than the target, resulting in excessive redundancy and error information, we use the center head method for prediction here, that is, three convolution branches are used to represent center classification, offset regression and size regression respectively, and then the corresponding regression coordinates are used to calculate the bounding box to determine the target position as the final prediction result.

[0037] For the loss function part, weighted focal loss is used for classification, and l1 loss and ordinary GIoU loss are combined for training. The final total loss function is represented as follows:

[0038]

[0039] Where λ G = 2 and is the trade-off weight.

[0040] In step 5, the training data sets: GOT-10K, a dataset containing 10000 video clips, covering more than 560 types of moving objects and 87 types of motion patterns, there is no overlap between training and testing categories, TrackingNet: a large-scale target tracking dataset, the training set data includes 30132 sequences, each sequence has an average of 471.4 frames, LaSOT: a large-scale, long-sequence target tracking dataset, containing 1140 training sequences, 280 test sequences, 1400 sequences, a total of more than 3.5 million frames, with an average video length of 2500 frames, all sequences are long-term, COCO: dataset has 328,000 pictures, containing 2.5 million labeled instances, by using advanced class detection, instance labeling, instance segmentation, etc. User interface creation;

[0041] For the baseline test of GOT-10k dataset, the one-time protocol is followed, only the GOT-10k split training set is used as the training set of the model, and the model is trained for 100 epochs. For other benchmarks, the TrackingNet, Lasot, GOT-10K and COCO training split are used to train the model for 300 epochs.

[0042] The optimizer is AdamW optimizer, the weight decay is 1e-4, the initial learning rate of the encoder and the decoder is 4e-5 and 4e-4 respectively, and the batch size is set to 32.

[0043] It is implemented in Python 3.11 using PyTorch 2.2 on a 48G NVIDIA A40 graphics card, and each epoch takes about 5 minutes to train.

[0044] The area under the curve (AUC) of the evaluation index is used to measure the overall performance of the tracking algorithm at different overlap thresholds, and the calculation formula is as follows:

[0045]

[0046] Where SR is the success rate at different overlap thresholds, and a is the overlap threshold (IoU). Higher AUC values indicate better performance of the algorithm under various conditions.

[0047] The accuracy (P) measures the average deviation of the predicted position from the true target position, especially the deviation of the center position. The calculation formula is as follows:

[0048]

[0049] Where N is the total number of frames, C i is the center position of the i-th frame prediction box, is the center position of the i-th frame real box, is the Euclidean distance between the two center points, I is the indicator function, which takes 1 when the distance is less than or equal to the threshold δ, otherwise 0. High accuracy indicates the accuracy of the predicted center position of the tracked target.

[0050] The normalized accuracy (P Norm ) is used to measure the tracking accuracy at different scales, and the standardization process eliminates the influence of scale differences between different videos;

[0051]

[0052] Where diag(G i ) is the diagonal length of the i-th frame real box;

[0053] Average overlap (AO) measures the average overlap of the predicted and ground truth bounding boxes, reflecting the matching degree of the tracking result and the real target;

[0054]

[0055] where B i is the i-th frame predicted bounding box, G i is the i-th frame ground truth bounding box, and IoU(B i , G i ) is the intersection over union of the predicted and ground truth bounding boxes.

[0056] SR 0.5 and SR 0.75 represent the success rates at IOU thresholds of 0.5 and 0.75, respectively, for evaluating the performance of the tracking algorithm at different strictness levels.

[0057]

[0058] where alpha is the IOU threshold (0.5 or 0.75), and I is the indicator function, which takes 1 when the IOU is greater than or equal to the threshold, and 0 otherwise.

[0059] FPS measures the processing speed of the tracking algorithm, which is particularly important in real-time applications.

[0060]

[0061] A high FPS value indicates that the algorithm can process more frames in real time.

[0062] Compared with the prior art, the present application has the following advantages:

[0063] The present application proposes a single target tracking method based on aggregated token perception features, which highlights the information in the image by aggregating tokens without increasing the network burden, and obtains the aggregated token instead of the original template token by calculating the key information of the template image, thereby enhancing the key information of the template, and a multi-scale feature fusion module that fuses deep and shallow semantic information, makes up for the loss caused by pruning, and enriches the image information.

[0064] A large number of experiments on three datasets have proved the effectiveness and robustness of the single target tracking method based on aggregated token perception features, and its advantages in various scenarios. The method can still achieve fast and accurate tracking under conditions such as target occlusion and target appearance change. The single target tracking method provided by the present application brings an efficient and lightweight solution to the target tracking field through its innovative aggregated token perception feature method and network structure, and has a wide application prospect, especially in modern applications that require real-time processing of large amounts of data. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 For the network framework of the present application;

[0066] Figure 2 For the multi-scale feature fusion module framework of the present application. DETAILED DESCRIPTION

[0067] The present application will be further described in conjunction with the drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and are not used to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or modifications to the present application, and these equivalent forms also fall within the scope defined by the present application.

[0068] A single target tracking method based on aggregated token perception features, the method comprising the following steps:

[0069] Step 1: Based on the single target tracking of aggregated token perception features, the overall model composed of ViT-B backbone network, aggregated token perception feature module, multi-scale feature fusion module and positioning head is constructed.

[0070] Step 2: Initialization: Given the data set X containing N video sequences, the first frame image and the initial bounding box position of the target are selected, and the target detection model of the pre-trained ViT-B backbone network or the manually labeled bounding box is used to initialize the tracking model.

[0071] Step 3: Feature extraction and enhancement process, the template key information is obtained through the aggregated token perception feature module, the feature representation of the current frame image and the target region is accurately extracted by using the deep neural network, the cropped target region is input into the feature extraction network, the network will pass through multiple attention layers in turn, extract the texture features from the bottom layer to the semantic features of the high layer, and finally output a fixed length feature vector as the feature representation of the target.

[0072] Step 4: Target positioning and tracking: input the current frame features and initial target features into the tracking network, repeat the above feature extraction step, continuously process the video sequence tracking target in the subsequent frame image through the trained model, and finally output the video sequence with target position label.

[0073] Step 5: Model training and performance evaluation: the training of the model needs training data set and test data set, and multiple evaluation indexes are needed to evaluate the performance of the model.

[0074] In step 3, the aggregated token perception feature module can effectively aggregate the original representation information of the image for perception features, and the aggregation process of the image can be summarized as follows:

[0075] Rowavg = avg(Z 11 , Z 12 , Z 13 ... Z 1n )

[0076] Column_avg = avg(Z 11 , Z 21 , Z 31 ... Z n1 )

[0077] Z' = Z - Row_avg + Column_avg

[0078] wherein Z ij represents the value of the i-th row and j-th column of the original image two-dimensional feature map, Row_avg represents the average value of each row of the two-dimensional feature map, Column_avg represents the average value of each column of the two-dimensional feature map, avg(·) represents taking the average value, and Z' represents the value finally input into the VIT after the above operation, i.e.

[0079]

[0080] Through the above operation of subtracting the average value of the template feature map, the information in the feature map can be aggregated and weighted, so that the model pays more attention to the target area. The splicing of the token perception feature module and the Cls_Token carrying global features in the VIT can make the model better understand the graphics and better process images under different conditions.

[0081] In step 1, the multi-scale feature fusion module is used to compensate for the lack of feature extraction and capture small targets or subtle changes in images. The multi-scale feature fusion module has four different layers, each layer containing different numbers and sizes of convolutions to obtain more rich receptive fields and deeper semantic information. The output features are transmitted into the multi-scale feature fusion module, and the outputs of each layer are fused through different layers and different size convolutions.

[0082] The above process can be expressed as:

[0083] D0 = Z output

[0084]

[0085] wherein Z output represents the output of the template image after VIT, D i represents the output of the i-th layer in the module, and cat represents the splicing operation.

[0086] The template features after the above operation have deeper semantic information. In order to preserve this information, finally we make residual connection between the results obtained by each layer and the original feature D0, and perform element-wise addition operation to obtain the final result D output :

[0087] D output = D0+D1+D2+D3+D4

[0088] The entire network model keeps the feature extraction mode of the template image Z unchanged when performing attention calculation.

[0089] When performing attention calculation on the search image X, we continue the asynchronous feature extraction mode, and perform separate feature extraction on the search image X for the first 5 layers, as follows:

[0090]

[0091] Where Q, K and V are query matrix, key matrix and value matrix respectively. Subscripts Z and X represent matrix items belonging to the template and search region respectively. Z' represents the output of the template image Z after the aggregation token perception module.

[0092] In the subsequent several layers, the output D output of the template image feature after the multi-scale feature fusion module is spliced and then the attention calculation is performed, as follows:

[0093]

[0094] Where Q, K and V are query matrix, key matrix and value matrix respectively. Subscripts X and D represent matrix items belonging to the search region and the output of the template image after the multi-scale feature fusion module.

[0095] On the basis of the above two modules, in order to ensure the performance and practicability of the network, the two network branches have the characteristic of shared weights in the process of template image and search image feature extraction.

[0096] In step 3, corner point positioning and center head positioning in target tracking are two common prediction methods for determining the position and pose of the target in the image or video. Since corner point positioning has the problem of boundary box much larger than the target, resulting in too much redundant and error information, we use the center head method for prediction here, that is, three convolution branches are used to represent center classification, offset regression and size regression respectively, and then the corresponding regression coordinates are used to calculate the bounding box to determine the target position as the final prediction result.

[0097] For the loss function part, the weighted focal loss is used for classification, and the l1 loss and the ordinary GIoU loss are combined for training. The final total loss function is represented as follows:

[0098]

[0099] where λ G = 2 and is the trade-off weight.

[0100] In step 5, the training dataset: GOT-10K, a dataset containing 10000 video clips, covering more than 560 classes of moving objects and 87 kinds of motion patterns, there is no overlap between the training and testing categories, TrackingNet: a large-scale target tracking dataset, the training set data includes 30132 sequences, each sequence has an average of 471.4 frames, LaSOT: a large-scale, long-sequence target tracking dataset, containing 1140 training sequences, 280 test sequences, 1400 sequences, a total of more than 3.5 million frames, with an average video length of 2500 frames, all sequences are long-term, COCO: a dataset with 328,000 images, containing 2.5 million labeled instances, created by using advanced class detection, instance labeling, instance segmentation, etc. User interface.

[0101] When training the model, for the GOT-10k dataset benchmark, follow the one-time protocol, only use the GOT-10k segmented training set as the training set of the model, and train the model for 100 epochs, for other benchmarks, use TrackingNet, Lasot, GOT-10K and COCO training segmentation to train the model for 300 epochs.

[0102] The optimizer is AdamW optimizer, the weight decay is 1e-4, the initial learning rate of the encoder and the decoder is 4e-5 and 4e-4 respectively, and the batch size is set to 32.

[0103] It is implemented in Python 3.11 using PyTorch 2.2 on a 48G NVIDIA A40 graphics card, and each epoch takes about 5 minutes to train.

[0104] The area under the curve (AUC) of the evaluation index is used to measure the overall performance of the tracking algorithm under different overlap thresholds, and the calculation formula is as follows:

[0105]

[0106] where SR is the success rate under different overlap thresholds, and α is the overlap threshold (IoU). Higher AUC values indicate that the algorithm performs better under various conditions.

[0107] The precision (P) measures the average deviation of the predicted position from the true target position, especially the deviation of the center position. The calculation formula is as follows:

[0108]

[0109] where N is the total number of frames, C i is the center position of the i-th frame of the predicted frame, is the center position of the i-th frame of the true frame, is the Euclidean distance between the two center points, and I is an indicator function that takes 1 when the distance is less than or equal to the threshold value δ, and 0 otherwise. High precision indicates the accuracy of the predicted tracking target center position.

[0110] The normalized precision (P Norm ) is used to measure the tracking accuracy at different scales, and the standardization process eliminates the influence of scale differences between different videos.

[0111]

[0112] where diag(G i ) is the diagonal length of the i-th frame of the true frame;

[0113] The average overlap (AO) measures the average overlap of the predicted frame and the true frame, reflecting the matching degree of the tracking result and the true target.

[0114]

[0115] where B i is the i-th frame of the predicted bounding box, G i is the i-th frame of the true bounding box, and IoU(B i , G i ) is the intersection over union of the predicted frame and the true frame.

[0116] SR 0.5 and SR 0.75 represent the success rate at IOU thresholds of 0.5 and 0.75, respectively, for evaluating the performance of the tracking algorithm at different strictness levels.

[0117]

[0118] where α is the IoU threshold (0.5 or 0.75), and I is an indicator function that takes 1 when the IoU is greater than or equal to the threshold value, and 0 otherwise.

[0119] FPS measures the processing speed of the tracking algorithm, which is particularly important in real-time applications.

[0120]

[0121] A high FPS value indicates that the algorithm can process more frames in real-time.

[0122] Params measures the number of parameters in the model, mainly including weight and bias parameters. The number of parameters reflects the storage requirements and training complexity of the model.

[0123] MACs measures the amount of computation required during a single forward propagation, especially the number of multiply-accumulate operations. It reflects the computational complexity and running efficiency of the model, and is an important indicator for evaluating the inference speed of the model on different hardware devices.

[0124] Some classic and latest real-time and non-real-time algorithms in the single target tracking field are compared on the Lasot, TrackingNet and Got10k datasets, and the results are shown in Table 1.

[0125] Table 1, comparison of the proposed method and other methods on the Lasot, TrackingNet and GOT-10k datasets.

[0126]

[0127] On the Nvidia Tesla T4-16GB GPU, the most advanced non-real-time tracker is tested and evaluated for lightweight model from the computational complexity (MACs), floating-point operations (FLOPs), parameter quantity (Params) and frames per second (FPS), and the detailed results are shown in Table 2.

[0128] Table 2, comparison of the model and existing non-real-time tracker in terms of computational complexity (MACs), parameter quantity (Params) and frames per second (FPS).

[0129]

Claims

1. A single target tracking method based on aggregated token perception features, characterized in that: The method comprises the following steps: Step 1: Under single target tracking based on aggregated token-aware features, an overall model consisting of the ViT-B backbone network, the aggregated token-aware feature module, the multi-scale feature fusion module, and the localization head is constructed. The multi-scale feature fusion module has four different layers, each containing a different number and size of convolutions. The output features are passed to the multi-scale feature fusion module, and the output of each layer is fused through different layers and different sizes of convolutions; Step 2: Initialization: Given a dataset X containing N video sequences, select the first frame image and the initial bounding box position of the target, and initialize the tracking model using the object detection model of the pre-trained ViT-B backbone network or manually annotated bounding boxes; Step 3: Feature extraction and enhancement process: The template key information is obtained through the aggregated token perception feature module. The deep neural network is used to accurately extract the feature representation of the current frame image and the target area. The cropped target area is input into the feature extraction network. The network sequentially passes through multiple attention layers, extracting from low-level texture features to high-level semantic features, and finally outputs a fixed-length feature vector as the feature representation of the target. Step 4: Target localization and tracking: Input the current frame features and the initial target features into the tracking network, repeat the feature extraction steps, and continue to track the target in the subsequent frame images through the trained model processing video sequence, and finally output a video sequence with the target location annotated; Step 5: Model training and performance evaluation, In step 3, the aggregated token perception feature module aggregates the original representation information of the image for perception features, wherein the process of aggregating the image is as follows: Row avg =Aug(Z 11 ,With 12 ,With 13 ...With 1n ) Column_avg=avg(Z 11 ,From 21 ,From 31 …From n1 ) Z′=Z-Row_avg+Z-Column_avg where Z 1n Represents the value of the 1st row and nth column of the 2D feature map of the original image, where Z n1 Represents the value of the first column of the nth row of the two-dimensional feature map of the original image, Row_avg represents the average value of each row of the two-dimensional feature map, Column_avg represents the average value of each column of the two-dimensional feature map, avg(·) represents the average value, and Z′ represents the value finally input into the ViT-B backbone network.

2. A single target tracking method based on aggregated token perception features according to claim 1, characterized in that: In step 1, the multi-scale feature fusion module has four different layers, each of which contains convolutions of different numbers and sizes. The output features are passed to the multi-scale feature fusion module, and the output of each layer is fused through convolutions of different layers and sizes. D0=Z output where Z output Denotes the output of the template image after passing through the ViT-B backbone network, D i represents the output of the i-th layer in the module, and cat represents the concatenation operation; In step 1, the result obtained in each layer is residually connected with the original feature D0, and the element-by-element addition operation is performed to obtain the final result D output : D output ND0+D1+D2+D3+D4 In the subsequent layers, the template image features are passed through the multi-scale feature fusion module and then output D output Perform splicing and then calculate attention.

3. The single target tracking method based on aggregated token perception features according to claim 1, characterized in that: In step 3, prediction is performed using a center head approach, i.e., three convolution branches are used to represent center classification, offset regression, and size regression, respectively. Then, the corresponding regression coordinates are used to calculate the bounding box, and the target position is determined as the final prediction result. For the loss function part, weighted focal loss is used for classification, and l1 loss and ordinary GIoU loss are combined for training.

Images