Multi-target tracking method based on lightweight network
By using lightweight networks and scoring networks to filter features in multi-objective tracking, combining deformable attention and Hungarian losses, the problem of high computational complexity of Transformer model is solved, and efficient multi-objective tracking is achieved, suitable for edge device deployment.
Patent Information
- Application Number
- CN202411123537.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
AI Technical Summary
The existing Transformer-based multi-objective tracking method has a long training time and is difficult to deploy to edge devices due to the large calculation complexity and large amount of parameters, which cannot meet the needs of real-time and low power consumption.
A multi-objective tracking method based on lightweight network is proposed. Multi-scale features are extracted through CNN, and some feature input inputs are used to filter the encoder using the scoring network, and target query and trajectory query are decoded simultaneously in the decoder, which reduces the floating point of the model The number of operations and the amount of parameters are used, and the detection performance is improved by using deformable attention and Hungarian losses.
While maintaining the original accuracy, the number of floating-point operations and parameter volume of the model is greatly reduced, the calculation cost is significantly reduced, and it is suitable for edge device deployment, achieving efficient multi-objective tracking.
Smart Images

Figure FT_1 
Figure FT_2 
Figure SMS_2
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular proposes a multi-target tracking method based on a lightweight network from the perspective of reducing model complexity. Background Art
[0002] In the development of artificial intelligence, computer vision has become a very important branch, especially image recognition and classification technology. Computer vision image recognition tasks refer to the use of computer algorithms to identify and classify objects, scenes, faces, etc. in images. Single target tracking specifically means to associate a target of interest with an input video sequence. Multi-target tracking, as an extended task of target detection, is widely used in intelligent security, smart transportation and autonomous driving. The present invention also focuses on the exploration and research of multi-target tracking tasks.
[0003] Compared with single target tracking, multi-target tracking faces more challenges. On the one hand, due to the diversity of the number of targets, the algorithm needs to be able to detect new targets and terminate old targets. On the other hand, different targets may be blocked or overlapped, and the targets themselves may also be deformed or have lighting changes, which makes it extremely difficult to associate targets. In addition, interference noise in complex dynamic backgrounds is also a major challenge that multi-target tracking needs to deal with.
[0004] Traditional multi-target tracking algorithms mostly adopt a two-stage strategy of "detection-association". First, detectors such as YOLO and Faster R-CNN are used to extract target candidate boxes in each frame to obtain the target appearance and motion features. Then, data association techniques such as the Hungarian algorithm and joint probabilistic data association filtering are used to match the detection results of the current frame with the tracking trajectory of the previous frame to achieve identity transfer. This step-by-step method is simple and intuitive, but since detection and association are relatively independent, it is difficult to transfer context information well, which can easily lead to error accumulation and affect overall performance.
[0005] In recent years, multi-target tracking methods based on end-to-end frameworks have made great progress, among which Transformer-based methods have become a research hotspot. With the help of the self-attention mechanism, Transformer can fully explore the long-range dependencies of input data and effectively solve the problems of target occlusion and overlap. At the same time, detection and tracking are uniformly modeled as a sequence-to-sequence process, and the detection target and the tracked object are automatically bound through query alignment, making the entire process simple and efficient.
[0006] Typical Transformer-based methods usually use the detected target as the target query of the decoder and the tracking of existing targets as the trajectory query. The encoder is used to model the video frames, and the decoder decodes the two types of queries and outputs the prediction results. However, the huge amount of computation and parameters of the Transformer model itself has become a bottleneck for the promotion of these methods, limiting their practicality and promotion capabilities.
[0007] In addition to the above-mentioned computational complexity challenges, the field of multi-target tracking also faces other problems and research directions that deserve attention. For example, how to make full use of multimodal information (images, voice, text, etc.) to improve the robustness of tracking; how to combine prior knowledge to improve the ability to adapt to complex scenarios and uncertain factors; how to design more efficient data association strategies to reduce the risks of target switching and identity drift; how to deploy tracking systems to edge devices to meet real-time and low power requirements, etc. Solving these problems will greatly promote the development and practical application of multi-target tracking technology.
[0008] In general, multi-target tracking, as a core basic problem of computer vision, has not only received widespread attention in academia, but is also closely related to many areas of real life, and has important application value in security monitoring, intelligent transportation, robot navigation, augmented reality, etc. With the rapid improvement of computing power and the development of deep learning technology, I believe that multi-target tracking will surely achieve more and more innovative breakthroughs in the future, giving artificial intelligence stronger perception and understanding capabilities. Therefore, how to reduce the complexity of the model and achieve high efficiency and lightweight while maintaining performance is an important issue that needs to be urgently solved in this field. Summary of the invention
[0009] Due to the huge number of parameters and floating-point operations caused by some current transformer-based multi-target tracking methods, the training of the model requires a lot of time and computing resources. It is also difficult to deploy to edge devices in practical applications. Therefore, the present invention proposes a multi-target tracking method based on a lightweight network, which greatly reduces the number of floating-point operations and parameters of the model while maintaining the original accuracy. First, the video is input into the CNN network frame by frame to extract multi-scale features. Secondly, the features are screened by the scoring network and then input into the encoder. Finally, the target query and trajectory query are decoded simultaneously in the decoder, and the corresponding borders and categories are predicted to form all detection and tracking results of the frame.
[0010] The present invention is achieved through the following technical solutions:
[0011] A multi-target tracking method based on a lightweight network includes the following steps:
[0012] Step 1: The multi-scale features extracted by CNN are input into the encoder through a learnable scoring network to filter the top ρ percent of features.
[0013] Step 2: After the encoder outputs the features, k target object queries are screened out; at the same time, 100 tracking object queries are initialized and input into the decoder together.
[0014] Step 3: Each query is processed by a multi-layer perceptron and input into the decoder to predict the corresponding bounding box and category, forming all the detection and tracking results for the frame. At the same time, a decoder cross-attention map is generated.
[0015] Step 4: The newly detected target will initialize a new tracking object query. The tracking object query of each frame will be passed to the next frame, carrying the spatial and identity information of the target, thus completing the tracking process.
[0016] In step 1, we have a scoring network g that measures the feature map X feat The significance of each feature in . We define a significant region for those features whose scores belong to the top ρ percent , for a given ρ, the size of the salient region is calculated by the following formula. The feature update method of each layer in the encoder is expressed as follows
[0017]
[0018] If the current j is not in the salient region Ω, the features of the previous layer are kept unchanged; if it is in the salient region, it is updated to . Where DefAttn refers to deformable attention, LN refers to layer normalization, and FFN refers to feed-forward network.
[0019] In step 2, we added an auxiliary detection head at the end of the encoder output to calculate the target category score of each encoder output feature to reflect whether each feature contains target information. According to these category scores, we sort all the encoder outputs and select the top k features with the highest scores. At the same time, we apply Hungarian loss on these selected features to speed up the convergence of the encoder and improve the detection performance. Finally, these top k encoder features are directly used as the target query input of the decoder module. n is the number of predicted targets, represents the optimal match between the i-th prediction and the true target, It is the sum of the costs.
[0020]
[0021] In step 3, in order to determine the encoder X featTo calculate the saliency of each feature of , we need to aggregate the decoder cross-attentions between all object queries and the encoder output. This process produces a single map of the same size as the backbone feature map, defined as the decoder cross-attention map. In the case of dense attention, the decoder cross-attention map can be easily obtained by summing the attention maps of each decoder layer. In the case of deformable attention, for each encoder token, the corresponding value of the decoder cross-attention map can be obtained by accumulating the attention weights of the decoder object queries whose attention offsets point to the encoder output token. Since the attention offset calculated in deformable attention is a fractional position, deformable attention uses bilinear interpolation to obtain the value. Therefore, we also use bilinear interpolation to obtain the decoder cross-attention map. Assume that the attention offset, weight, and reference point of the decoder object query q are p, A, and r, respectively. Then, the value of deformable attention is
[0022]
[0023] Where v represents the key value, and x enumerates all integer spatial positions in the feature map. is the bilinear interpolation kernel function, defined as follows.
[0024]
[0025] We accumulate the DAM values at the x position as follows:
[0026]
[0027] To train the scoring network, we binarize the decoder criss-cross attention weights so that only the top ρ percent of the encoder features are retained. This is because our goal is to find a small set of encoder features that are most frequently referenced by the decoder, rather than accurately predicting the extent to which the decoder will reference each encoder feature. This binarized decoder criss-cross attention map implies a one-hot target, indicating whether each encoder feature is contained in the top ρ percent of the encoder tokens that are most frequently referenced. And the network is trained by minimizing the binary cross-entropy loss, which is the loss between the binarized decoder criss-cross attention weights and the predictions. Where N is the total number of features in the feature map, is the score of the ith feature, is the decoder cross attention weight binarization value (0 or 1) of the ith feature, and BCE is the binary cross entropy loss.
[0028]
[0029] In step 4, for subsequent frames that are not the first frame, the queries input by the encoder include not only the target queries for initialization of detection in each frame, but also the trajectory queries that have been successfully detected in the previous frame. After being decoded by the decoder, if the target found by the trajectory query is successfully detected, it is assigned the same label. If it is not detected, it means that the target disappears. Those target queries that are successfully detected are used as new targets. Then, the decoder outputs of these new and old targets are used together as the trajectory query for the next frame. In this way, data association and thus the entire tracking task are completed in a relatively elegant way.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] The present invention proposes a multi-target tracking method based on a lightweight network, which greatly reduces the number of floating-point operations and the amount of parameters of the model while maintaining the original accuracy. On the one hand, by screening some features through a scoring network, the amount of transformer calculation is greatly reduced; on the other hand, by selectively updating encoder features, the computational cost can be reduced without reducing performance. Through verification, the multi-target tracking method based on a lightweight network provided by the present invention can greatly reduce the number of floating-point operations and the amount of parameters of the model, while ensuring the accuracy of tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 The overall framework diagram of a multi-target tracking method based on a lightweight network provided by the present invention;
[0033] Figure 2 A lightweight network structure diagram provided by the present invention; DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0035] like Figure 1As shown, the present invention proposes a multi-target tracking method based on a lightweight network. After each frame of image is input into the network, the image features will be extracted by CNN, and then enter the Transformer encoder for encoding to obtain the encoded features. These encoded features will contain the context information of the entire image scene. Then the encoded features will be input into the Transformer decoder for decoding. The decoder contains two types of queries: one is object query, which is used to detect and initialize new tracking targets; the other is tracking query, which is used to update existing tracking targets. In the first frame t=0, only object queries will be activated, and they will be decoded into output embeddings containing category and bounding box information to detect objects in the scene and initialize tracking. In subsequent frames, query object queries and tracking queries will be activated at the same time. Among them, the tracking query updates the state of the existing tracking target (as shown in the red box, yellow box and blue box in the figure) through self-attention and encoder-decoder attention to achieve tracking. The object query continues to detect newly appearing targets and initialize their tracking (as shown in the purple box in the figure). In this way, the joint optimization of target detection and tracking in the multi-target tracking framework is achieved, and the number of decoder queries also changes dynamically with the push of the video frame, thereby better simulating the tracking process.
[0036] Figure 2 A lightweight network structure diagram provided by the present invention includes the following steps:
[0037] Step 1: The multi-scale features extracted by CNN are input into the encoder through a learnable scoring network to filter the top ρ percent of features. The scoring network is used to measure the feature map X feat The saliency of each feature in is calculated by 4 linear layers, which are connected by Gaussian error linear unit activation functions. Except for the last layer, each layer is followed by layer normalization. The output dimension of the first layer is 256, which is halved to 128 and 64 in the second and third layers, and the output dimension of the fourth linear layer is 1 as the binary classification confidence of each feature. In the first layer, half of the output dimension (128 dimensions) is retained as global features. This part of the features is globally averaged pooled on the feature dimension to extract global statistical information. The pooled global features are concatenated with the other half of the local features (128 dimensions) output by the first linear layer to return the output dimension to 256 dimensions. This preserves both global and local information. When selecting the top ρ% of features based on the score, the features corresponding to the zero-filled area are filtered out to prevent them from affecting the final detection performance. The final binary classification loss comes from the binary cross entropy loss between the confidence of each feature and the target's binary decoder self-attention map.
[0038] We define a salient region for features whose scores belong to the top ρ percent. , for a given , the size of the salient region is calculated by the following formula. This formula calculates the size S of the salient region. Indicates that the number of features in the entire feature map is equal to N. ρ is a proportional coefficient less than 1, so That is, the number of features S in the salient area is much smaller than the number of features N in the entire feature map.
[0039]
[0040] The update method of each layer feature in the encoder is as follows. If the current j is not in the salient region Ω, the feature of the previous layer is kept unchanged; if it is in the salient region, it is updated to . Where DefAttn refers to deformable attention, LN refers to layer normalization, and FFN refers to feed-forward network.
[0041]
[0042] In step 2, we add a fully connected feedforward network and Hungarian matching loss at the end of each layer of encoder output. The addition of this intermediate supervision significantly alleviates the problem of gradient vanishing in deeper encoders based on the Transformer structure, ensuring that the gradient can be effectively propagated to the backbone network layer. At the same time, this explicit distinction of selection marks also optimizes the model's ability to understand foreground targets, thereby improving the final detection performance. The Hungarian loss matching is as follows. n is the number of predicted targets, represents the optimal match between the i-th prediction and the true target, It is the sum of the costs.
[0043]
[0044] Next, we added an auxiliary detection head at the end of the encoder output to calculate the target category score of each encoder output feature to reflect whether each feature contains target information. According to these category scores, we sorted all the encoder outputs and selected the top k features with the highest scores. At the same time, we applied Hungarian loss on these selected features to speed up the convergence of the encoder and improve the detection performance. Finally, these top k encoder features are directly used as the target query input of the decoder module.
[0045] In step 3, in order to determine the encoder X featTo calculate the saliency of each feature of , we need to aggregate the decoder cross-attentions between all target queries and encoder outputs. This process produces a single map of the same size as the backbone feature map, defined as the decoder cross-attention map. In the case of dense attention, the decoder cross-attention map can be easily obtained by summing the attention maps of each decoder layer. In the case of deformable attention, for each encoder token, the corresponding value of the decoder cross-attention map can be obtained by accumulating the attention weights of the decoder object queries with the attention offsets pointing to the encoder output token. Since the attention offset calculated in deformable attention is a fractional position, deformable attention uses bilinear interpolation to obtain the value. Therefore, we also use bilinear interpolation to obtain the decoder cross-attention map. Where x is the spatial position, A is the attention weight, p is the attention offset, r is the reference point, is the key value, and G is the bilinear interpolation kernel function.
[0046]
[0047] Based on the previous formula, the attention weight is accumulated to each x position to obtain the weight of the decoder cross attention map at the x position. We express the accumulation of the decoder cross attention weight at the x position as follows:
[0048]
[0049] To train the scoring network, we binarize the decoder criss-cross attention weights so that only the top ρ percent (by attention weight) of the encoder features are retained. This is because our goal is to find a small fraction of encoder features that are most frequently referenced by the decoder, rather than accurately predicting the extent to which the decoder will reference each encoder feature. This binarized decoder criss-cross attention weight implies a one-hot objective, indicating whether each encoder feature is included in the top ρ percent most frequently referenced encoder features. And the network is trained by minimizing the binary cross-entropy loss, which is the loss between the binarized decoder criss-cross attention weights and the predictions, where N is the total number of features in the feature map, is the score of the ith feature, is the decoder cross attention weight binarization value of the i-th feature, and BCE is the binary cross entropy loss.
[0050]
[0051] In step 4, for subsequent frames that are not the first frame, the queries input by the encoder include not only the target query for detection initialized in each frame, but also the trajectory query that has been successfully detected in the previous frame. After being decoded by the decoder, if the target found by the trajectory query is successfully detected, it is assigned the same label. If it is not detected, it means that the target disappears. Those target queries that are successfully detected are used as new targets. Then, the encoder outputs of these new and old targets are used together as the trajectory query for the next frame. In this way, data association and even the entire tracking task are completed in a relatively elegant way.
[0052] Table 1 is a performance comparison of the invention in this paper and some current mainstream multi-target tracking methods on the MOT17 dataset. The main advantage of the multi-target tracking method based on a lightweight network proposed in this invention is the significant improvement in computational efficiency. Specifically, the number of parameters of the model is only 25.42M, which is 13.6% to 84.5% less than other advanced tracking methods; the number of floating-point operations is 100.56G, which is 17.9% to 55.9% less than that of other advanced tracking methods. These indicators show that the proposed model greatly reduces the demand for computing power.
[0053] At the same time, considering the significant reduction in computational complexity, the proposed method can still achieve an overall tracking accuracy of 71%, which is comparable to or slightly lower than some more complex models. This shows the value of the model in balancing tracking performance and efficiency.
[0054] In summary, the proposed lightweight tracker significantly reduces the model size and greatly reduces the computational burden, achieving an efficient compromise between computation and performance, and is very suitable for real-time tracking applications that have strict requirements on system resources. This efficient and simplified design concept is of great significance in promoting the actual deployment of complex tracking models.
[0055] method MOTA FP FN IDS Params FLOPs MACs TransTrack 74.5 28323 112137 3663 46.87M 179.25G 215.2G TrackFormer 74.10 34602 108777 2829 44.01M 200.81G 358.8G GTR 75.30 36231 93150 2346 43.80M 180.45G - CenterTrack 67.80 18489 160332 3039 29.32M 150.56G 70.88G FairMOT 73.70 27507 117477 3303 29.71M 160.59G 84.98G Ours 71.00 32730 125274 5757 25.42M 100.56G 28.82G
[0056] Table 1
[0057] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A multi-target tracking method based on a lightweight network, characterized by: The following steps are involved: Step 1: The multi-scale features extracted by CNN are input into the encoder through a learnable scoring network to filter the top ρ percent of features. Step 2: After the encoder outputs the features, k target queries are screened out; at the same time, 100 trajectory queries are initialized and input into the decoder together. Step 3: Each query is processed by a multi-layer perceptron and input into the decoder to predict the corresponding bounding box and category, forming all the detection and tracking results for the frame. At the same time, a decoder cross-attention map is generated. Step 4: The newly detected target will initialize a new tracking object query. The tracking object query of each frame will be passed to the next frame, carrying the spatial and identity information of the target, thus completing the tracking process.
2. The multi-target tracking method based on lightweight network according to claim 1 is characterized in that The learnable scoring network in step 1. The feature X extracted by CNN feat Score each local feature in to determine its likelihood of being referenced by the decoder. Based on these scores, select the top ρ percent of features to input into the encoder. If the current j is not in the salient region Ω, directly retain the features of the previous layer unchanged; If it is in a salient area, update to . Where DefAttn refers to deformable attention, LN refers to layer normalization, and FFN refers to feed-forward network.
3. The multi-target tracking method based on lightweight network according to claim 2 is characterized in that In step 2, k target queries are selected. To improve the detection performance, we add an auxiliary detection head on the encoder output. The role of this detection head is to calculate the target category score of each encoder output feature to reflect whether each feature contains target information. According to these category scores, we sort all the encoder outputs and select the top k features with the highest scores. Subsequently, these top k encoder features are directly used as query inputs to the decoder module.
4. The multi-target tracking method based on a lightweight network according to claim 3, characterized in that The decoder cross attention map in step 3. To train the scoring network, we binarize the decoder cross attention map and backpropagate it to the scoring network. The binary attention weight value indicates the probability of whether the feature is among the top ρ percent of features. The scoring network is trained by minimizing the binary cross entropy loss, which measures the difference between the predicted probability and the binary attention weight value. Where N is the total number of features in the feature map, is the score of the ith feature, is the binarized value of the decoder cross attention weight of the i-th feature, and BCE is the binary cross entropy loss.
Citation Information
Cited By
Multi-target tracking method, device and equipment based on Transform and trajectory query group and medium
CN120279119A
