A Single-Stream Lightweight Object Tracking Method Based on Knowledge Distillation

Abstract

The present invention belongs to the fields of machine learning, computer vision, and object tracking, and provides a single-stream lightweight object tracking method based on knowledge distillation. The present invention realizes a lightweight single-stream object tracking network model based on a pure Transformer backbone network, and uses the OSTrack algorithm as the teacher model to distill the lightweight tracking model. The present invention proposes a similarity matrix between the template and the feature region, which can efficiently represent the interaction information between the template and the search region in the single-stream tracking algorithm. Using this information as the supervision signal in knowledge distillation can improve the performance of the lightweight single-stream object tracking model. The present invention improves the performance of the lightweight object tracking network and provides a more powerful object tracking model for edge computing scenarios.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, computer vision, and object tracking, and relates to knowledge distillation, single-stream object tracking algorithms, as well as single-object tracking algorithms OSTrack and image classification algorithms DeiT; specifically, it is a single-stream lightweight object tracking method based on knowledge distillation. Background Art

[0002] The object tracking task aims to continuously and accurately locate the position, scale, and other information of an object of interest in the initial frame in a video sequence in subsequent frames. Object tracking is one of the basic tasks in computer vision and is the application foundation for many computer vision-related problems such as action recognition, event detection, behavior understanding, and video object detection. Moreover, visual tracking has extremely broad practical application prospects in many fields such as national defense, medical care, transportation, entertainment, and photography.

[0003] Object tracking methods crop according to the target information input by the user in the initial frame as a template, and crop the search area according to the target position and size of the previous frame in subsequent frames, thus transforming the tracking task into a matching task between the template and the search area. With the development of deep learning technology, object tracking has gradually transitioned from two-stream object tracking methods to single-stream object tracking methods. Two-stream object tracking methods (such as SiamFC, SiamRPN++, TransT) usually use a backbone network with shared parameters for feature extraction and then use a cross-correlation module to achieve the matching between the template and the search area. This type of method separates the feature extraction and feature interaction modules, and the tracking effect of the model is relatively poor. Single-stream tracking methods (such as OSTrack, SeqTrack, ARTrack, etc.) use the Transformer structure to perform feature extraction and feature interaction simultaneously. Although the tracking performance has been improved, the computational complexity and the number of parameters of the Transformer structure itself are relatively large, making it difficult to deploy these algorithms on edge devices.

[0004] It should be noted that all the above-mentioned algorithms are designed around GPU computing resources, and a large amount of computing resources are often required as support during actual inference applications. However, in devices such as autonomous driving, drones, and intelligent tracking pan-tilts, the computing power is greatly limited. The HiT algorithm specifically designs a single-stream tracking framework with a hybrid architecture of convolution and Transformer for edge computing devices. This method enhances the performance of the tracking method by fusing visual features at different levels. Unfortunately, the number of parameters of this method is relatively large, and the model structure is relatively complex, facing great difficulties in deployment and optimization.

[0005] Therefore, how to design a high-performance tracking algorithm adapted to edge computing devices and achieve a balance between speed and accuracy is one of the challenges that need to be urgently solved in current tracking tasks. Summary of the Invention

[0006] The present invention aims to provide a single-stream lightweight object tracking method based on knowledge distillation. By designing an interactive distillation module, the performance of existing lightweight tracking algorithms based on Transformer is improved. The algorithm proposed by the present invention can achieve a performance similar to that of other models when the computational complexity is reduced to about 46% of other models and the number of parameters is reduced to about 15% of other models, greatly reducing the computing power requirements of lightweight tracking models.

[0007] Technical solution of the present invention:

[0008] A single-stream lightweight object tracking method based on knowledge distillation, the steps are as follows:

[0009] Step 1: Establish a single-stream object tracking model;

[0010] The single-stream object tracking model mainly consists of an image patch embedding module, a Transformer encoder module, and a tracking head;

[0011] The image patch embedding module consists of a convolutional layer with a convolutional kernel of and a stride of , and deformation and splicing operations; the input of the image patch embedding module is an image pair, including two images of a template and a search region; the template is cropped based on the target position in the initial frame of the video stream, and the search region is cropped in the current frame of the video stream based on the target position of the previous frame; the template and the search region are first downsampled 16 times through a convolutional layer with shared parameters, and then both are transformed into token sequences through deformation and positional encoding is added; here, a classification token and a distillation token are also introduced; then, the token sequences generated by the template and the search region, the classification token, and the distillation token are spliced into a new token sequence, which is used as the output of the image patch embedding module;

[0012] The Transformer encoder module mainly consists of multiple stacked multi-head attention modules and multi-layer perceptrons. Its input is the new token sequence output by the image patch embedding module, and the output is the token sequence feature that has completed feature extraction and feature interaction; after completing feature extraction and feature interaction, the token sequence feature is re-split into a template feature and a search region feature, and the classification token and the distillation token are only used to enhance feature extraction;

[0013] The tracking head mainly consists of a classification branch and a regression branch. The input of both branches is the search region feature. The classification branch outputs the scores of each feature point in the downsampled search region to obtain the approximate position of the center point of the target. The regression branch calibrates the position of the center point of the target and outputs the size of the target. Combining the outputs of the classification head and the regression head gives the accurate position of the target.

[0014] Step 2: Load the initial weights. Use the weights of the DeiT image classification model implemented based on knowledge distillation as the initial weights of the backbone network, as follows: The patch embedding module and the Transformer encoder module in the single-stream object tracking model require the weights of the DeiT image classification model for initialization. Among them, the structure of the patch embedding module in the single-stream object tracking model is different from that of the patch embedding module in the DeiT image classification model. The following processing is required during the weight loading process: Since the input format of the DeiT image classification model is an image, and the input in the single-stream object tracking model is two images, the template and the search region, the corresponding position encoding formats are different. To adapt the formats, first separate the classification token and the distillation token of the position encoding in the weights. Then, deform the remaining sequence-form position encoding in the position encoding into a two-dimensional image format, and then adapt the two-dimensional image format to the different image sizes of the template and the search region through bilinear interpolation to achieve the loading of the initial weights of the position encoding part. The Transformer encoder in the single-stream object tracking model has the same structure as the Transformer encoder in the DeiT image classification model, and the weights can be directly loaded.

[0015] Step 3: Build the computational framework for knowledge distillation, as follows: The knowledge distillation part includes the information expression method of distillation, the configuration of the teacher model, and the loss function.

[0016] In the information expression method of distillation, introduce the similarity matrix between the template feature and the search region feature, and calculate the cosine similarity between each pair of tokens in the template feature and the search region feature. The formula is:

[0017]

[0018] where, and are the tokens in the template feature and the search region feature respectively. The similarity matrix is used to describe the relationship between the template feature and the search region feature.

[0019] The teacher model selects the OSTrack model for single-stream object tracking. Since the output features of all tokens are required, the model without candidate token elimination is adopted. The student model is the single-stream object tracking model in step 1. During the training process, the teacher model inputs the same image data as the student model, adopts the evaluation mode, does not perform gradient calculation and parameter update, and only retains the similarity matrix calculated by the template feature and the search area feature after feature extraction of the teacher model. The similarity matrix output by the student model uses the similarity matrix of the teacher model as the supervision signal.

[0020] In the part of the loss function, the classification head of the student model adopts the Focal loss function, the regression head adopts the GIOU loss function and the L1 loss function, and the loss function for similarity matrix distillation adopts the L1 loss function. The total loss function is:

[0021]

[0022] Among them, 、 、 、 represent the weights of the Focal loss function, the L1 loss function, the GIOU loss function, and the loss function for similarity matrix distillation, respectively.

[0023] Advantages of the present invention:

[0024] (1) The Token similarity distillation algorithm proposed by the present invention can effectively utilize the features of the single-stream object tracking teacher model and can achieve high-performance tracking on the edge computing platform.

[0025] (2) The lightweight object tracking algorithm proposed by the present invention better balances speed and accuracy and significantly improves the running speed of the object tracking algorithm in the edge computing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the framework diagram of the model structure and distillation of the present invention.

[0027] Figure 2 It is the schematic diagram of similarity matrix calculation. DETAILED DESCRIPTION OF THE INVENTION

[0028] The following further illustrates the specific implementation manners of the present invention in combination with the drawings and technical solutions.

[0029] First, a single-stream object tracking model is built. The structure of the single-stream object tracking model is as Figure 1As shown in the figure, the ViT-Tiny model mentioned in DeiT is used as the backbone network of the target tracking model, that is, a Transformer encoder with 12 layers and 192 channels is used for feature extraction and feature fusion. The head uses a classification plus regression form to obtain the target position. The classification head uses 5 convolutional layers, and finally, the probability that each downsampled position is the target is obtained through sigmoid calculation; the regression head uses two groups of 5 convolutional layers, one group for regressing the offset of the target center point, and the other group for regressing the size of the target.

[0030] The process of image patch embedding can be expressed as:

[0031]

[0032]

[0033] In the formula, is the search area, is the template, is the convolutional layer in image patch embedding, represents the operation of flattening the image patch, and are the token sequences converted from the search area and the template. The input of the Transformer encoder block can be expressed as:

[0034]

[0035] In the formula, is the feature input to the Transformer encoder, represents the concatenation operation, represents the classification token, represents the distillation token, and respectively represent the position encodings corresponding to the template and the search area.

[0036] The process of the Transformer encoder block can be expressed as:

[0037]

[0038] In the formula, represents the feature input to the nth layer, represents the attention module, represents the multi-layer perceptron module, represents the feature output from the nth layer.

[0039] During the process of loading the pre-trained weights, first split the position encoding:

[0040]

[0041] In the formula, are the position encodings corresponding to the image, classification token, and distilled token respectively, represents the splitting operation, is the position encoding in the pre-trained weights. Then, the position encoding of the image part is processed as follows:

[0042]

[0043] In the formula, is the position encoding corresponding to the search area, represents the flattening operation, represents bilinear interpolation, represents deforming to a two-dimensional shape, and the position encoding corresponding to the template is obtained in the same way.

[0044] The distilled similarity matrix is calculated as Figure 2 shown. First, the template and the search area image are used to extract features by the image encoder, then the cosine similarity between the features of the template and the search area is calculated to obtain the similarity matrix. At the same time, the similarity matrix is also calculated in the single-stream object tracking teacher model, and knowledge transfer is performed through this similarity matrix. The formula for calculating the similarity is:

[0045]

[0046] In the formula, and are the tokens in the template feature and the tokens in the search area feature respectively.

[0047] During the training process, the classification head uses the Focal loss function to balance positive and negative samples. After the regression head calculates the target box, the L1 loss function and the GIOU loss function are used to jointly optimize it, while the distilled calculation of the similarity matrix is optimized through the L1 Loss. The calculation process of the distilled loss function is as follows:

[0048]

[0049] In the formula, and represent the subscripts of in the teacher model similarity matrix and the student model similarity matrix respectively, and represent the shape of the similarity matrix, represents the loss function corresponding to the similarity matrix. The total loss function is expressed as

[0050]

[0051] Among them, , , , represent the weights of the Focal loss function, L1 loss function, GIOU loss function, and the loss function of similarity matrix distillation, respectively.

[0052] Table 1 shows the comparison between the lightweight tracking algorithm proposed in this invention and other efficient trackers. It can be seen that the object tracking algorithm proposed in this invention achieves a tracking performance close to that of HiT while doubling the speed compared to the HiT algorithm, reaching 66.5% AO on the GOT10K dataset, exceeding HiT by 1.5% AO. The result on the TrackingNet dataset is only 0.1% AUC lower, and the result on the LaSOT dataset is only 0.5% AUC lower. Moreover, it is faster and has higher tracking accuracy than other high-performance object tracking algorithms, achieving a better balance between speed and accuracy.

[0053] Table 1 Performance Comparison

[0054]

[0055] In addition, as shown in Table 2, the number of parameters and the theoretical computational amount of this invention have also been greatly compressed. Compared with the current better-performing HiT model, the number of parameters has been reduced by 6 times, and the computational amount has been reduced by 2 times, realizing a faster object tracker.

[0056] Table 2 Comparison of the Number of Parameters and Computational Amount

[0057]

Claims

1. A single-stream lightweight object tracking method based on knowledge distillation, characterized in that, The steps are as follows: Step 1: Establish a single-stream object tracking model; The single-stream object tracking model mainly consists of an image patch embedding module, a Transformer encoder module, and a tracking head; The image patch embedding module consists of a convolutional layer with a kernel size of 16×16 and a stride of 16, and deformation and splicing operations; the input of the image patch embedding module is an image pair, including two images of a template and a search region; the template is cropped based on the target position in the initial frame of the video stream, and the search region is cropped in the current frame of the video stream based on the target position of the previous frame; the template and the search region are first downsampled 16 times through a convolutional layer with shared parameters, and then both are transformed into token sequences through deformation and positional encoding is added; a classification token and a distillation token are also introduced here; then the token sequences generated by the template and the search region, the classification token, and the distillation token are spliced into a new token sequence as the output of the image patch embedding module; The Transformer encoder module mainly consists of multiple stacked multi-head attention modules and multi-layer perceptrons. Its input is the new token sequence output by the image patch embedding module, and the output is the token sequence feature that has completed feature extraction and feature interaction; after completing feature extraction and feature interaction, the token sequence feature is re-split into a template feature and a search region feature, and the classification token and the distillation token are only used to enhance feature extraction; The tracking head consists of a classification branch and a regression branch, and the input of both branches is the search region feature; The classification branch outputs the scores of each feature point of the downsampled search region to obtain the position of the center point of the target; the regression branch calibrates the position of the center point of the target and outputs the size of the target; combining the outputs of the classification head and the regression head gives the accurate position of the target; Step 2: Load the initial weights. Use the weights of the DeiT image classification model implemented based on knowledge distillation as the initial weights of the backbone network, specifically as follows: The image patch embedding module and the Transformer encoder module in the single-stream object tracking model require the weights of the DeiT image classification model for initialization. Among them, the structure of the image patch embedding module in the single-stream object tracking model is different from that of the image patch embedding module of the DeiT image classification model. The following processing needs to be carried out during the weight loading process: To adapt the format, first separate the classification token and the distillation token of the positional encoding in the weights, deform the remaining positional encoding in the form of a sequence in the positional encoding into a two-dimensional image format first, and then adapt the two-dimensional image format to the different image sizes of the template and the search region through bilinear interpolation to achieve the loading of the initial weights of the positional encoding part; the structure of the Transformer encoder in the single-stream object tracking model is the same as that of the Transformer encoder of the DeiT image classification model, and the weights can be directly loaded; Step 3: Build a computational framework for knowledge distillation, specifically as follows: The knowledge distillation part includes the expression method of distilled information, the configuration of the teacher model, and the loss function; In the information expression method of distillation, a similarity matrix of template features and search region features is introduced, and the cosine similarity is calculated pairwise for each token in the template features and search region features. The formula is as follows: where feature1 and feature2 are the tokens in the template features and search region features respectively, and the similarity matrix is used to describe the relationship between the template features and search region features; The teacher model selects the OSTrack model for single-stream object tracking. Since the output features of all tokens are required, the model without candidate token elimination is adopted. The student model is the single-stream object tracking model in step 1. During the training process, the teacher model inputs the same image data as the student model, adopts the evaluation mode, does not perform gradient calculation and parameter update, and only retains the similarity matrix calculated by the template features and search region features after feature extraction of the teacher model. The similarity matrix output by the student model uses the similarity matrix of the teacher model as the supervision signal; In the part of the loss function, the classification head of the student model uses the Focal loss function, the regression head uses the GIOU loss function and the L1 loss function, and the loss function of similarity matrix distillation uses the L1 loss function. The total loss function is as follows: loss = α Focal l Focal + α L1 l L1 + α GIOU l GIOU + α similarity l similarity Among them, α Focal , α L1 , α GIOU , α similarity represent the weights of the Focal loss function, the L1 loss function, the GIOU loss function, and the loss function of similarity matrix distillation, respectively.

Images