Lightweight video multi-target tracking method and system based on prior content

Abstract

The invention discloses a priori content-based lightweight video multi-target tracking method and system, and the method comprises the steps: building a tracking model, achieving the optimization of model parameters by taking the minimization of an overall loss function as a target, and then achieving the video multi-target tracking through employing the optimized model. A prior content generation strategy based on a multi-scale feature map is provided, a prior content generation module is designed, and the convergence speed of the model is increased while the target detection precision is effectively improved. And then, a model lightweight strategy is used, and a model structure is simplified by directly reducing the number of layers of a Transform decoder, so that the real-time application requirement is met while the calculation cost is reduced. The invention introduces an attention mechanism-based multi-target tracking method based on priori content generation and a model lightweight strategy in detail, and the tracking performance equivalent to that of a reference model is maintained while the model lightweight is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision tracking technology, specifically relating to a lightweight video multi-target tracking method and system based on prior content. Background Technology

[0002] Video multi-object tracking (MOT) is a key subtask in the field of computer vision. It aims to identify multiple targets of interest in a video sequence and perform data association between different frames to generate tracking trajectories for multiple targets. It has important application value in many fields such as autonomous driving, intelligent monitoring, behavior analysis, and sports analysis.

[0003] Currently, video multi-object tracking technology is developing along three paths: detection-association based tracking methods, integrated tracking methods combining joint detection and association, and attention-based tracking methods. Among these, detection-association based methods heavily rely on detector performance, and their object detection and data association tasks cannot be jointly optimized, resulting in high computational complexity and complex models, making them unsuitable for real-time applications. Integrated tracking methods combining joint detection and association have accelerated the development of end-to-end tracking frameworks and improved inference speed while simplifying model architecture, thus meeting deployment requirements in application scenarios. However, because they still include post-processing steps such as non-maximum suppression in the object detection task, these methods have not yet achieved truly end-to-end tracking. Furthermore, since this method only utilizes frame-level feature information and ignores potential spatiotemporal motion information, its robustness in complex scenarios with frequent occlusion and a large number of similar targets remains insufficient. Attention-based tracking methods have further improved multi-object tracking technology and achieved truly end-to-end tracking. However, while the aforementioned methods perform well in tracking tasks, they overlook the lack of prior information for object queries and the inaccuracy of reference points for track queries, limiting further improvements in model performance. Furthermore, the attention mechanism based on the encoder-decoder framework increases model complexity and computational complexity, thus posing challenges to its deployment in real-world scenarios.

[0004] Therefore, when the model structure is relatively complex, there is an imbalance between tracking accuracy and computational efficiency. How to improve the real-time performance and scalability of the algorithm while ensuring tracking accuracy is an urgent problem to be solved. Summary of the Invention

[0005] To address the problems mentioned in the background, this invention first provides a lightweight video multi-target tracking method based on prior content. This method aggregates features from the multi-scale feature maps output by the Transformer encoder using self-attention pooling, generating a content query with feature information of the target to be detected, and then introducing it into the target query, thereby accelerating model convergence and improving target detection accuracy. In addition, it also achieves lightweight model structure by directly reducing the number of Transformer decoder layers, while maintaining its overall performance comparable to the benchmark model, thus effectively accelerating model convergence and achieving significant performance improvement.

[0006] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows:

[0007] A lightweight video multi-object tracking method based on prior content, the key of which includes the following steps:

[0008] S1: Establish a tracking model, which includes a CNN backbone network, a Transformer encoder, a Transformer decoder, and a prior content generation network; firstly, use the CNN backbone network and the Transformer encoder to extract and encode features from the input image to obtain the output feature representation. Secondly, The prior content is input into a network that generates a target query containing information about the target to be detected through its self-attention pooling. Then, and The common input is fed into the Transformer decoder, and the output embedding containing target category and location information is obtained through the deformation cross-attention mechanism. Finally, The prediction head is input, which outputs the final object detection bounding box and category, and the detection result is used to initialize the tracking query for the next frame. In subsequent frames, the merged queries and They are fed into the Transformer decoder together to achieve joint modeling of object detection and data correlation;

[0009] S2: To minimize the overall loss function To optimize the model parameters to achieve the desired result;

[0010] S3: Utilize the optimized model to achieve multi-target tracking in video.

[0011] The prior content generation module aggregates the feature information of the target to be detected through self-attention pooling, thereby realizing content query C. detAdaptive generation. Optionally, the prior content generation network includes an attention map mapping network and a channel reweighting network, and the specific processing procedure is as follows:

[0012] Let all input feature maps be F = (f1, f2, ..., f N ), where the feature map of the nth layer is h and w represent the height and width of the feature map, respectively, and c represents the feature dimension, according to: Generate content query C det In the formula, The attention map mapping network, consisting of a series of convolutional networks, is responsible for outputting the mapping attention map. q represents the initial query count; SoftMax(·) represents the normalization function used to normalize f′. n Generate a weight matrix and compare it with f n Multiplication yields intermediate variables ψ(·) denotes a channel reweighting network used to reweight C′. det The channel weights are refined, specifically represented as: C det =SigMoid(σ(C′) det ))⊙C′ det In the formula, σ(·) represents a multilayer perceptron network based on linear mapping; SigMoid(·) represents the activation function; and ⊙ represents element-wise multiplication.

[0013] By adaptively generating prior content, the target query can be effectively initialized, giving it prior feature information of the target to be detected. Therefore, target search based on rich feature prior information can accelerate model convergence while improving detection performance, thereby improving the model's tracking performance.

[0014] Attention-based multi-object tracking methods primarily incur computational overhead in the Transformer encoder and decoder modules. Typically, in a Transformer architecture, the encoder and decoder are each configured with six layers, optimizing the input query layer by layer to improve model performance. However, as the number of network layers increases, the model's computational resource requirements grow significantly, leading to increased structural complexity and hindering its practicality. Therefore, this invention, based on a priori content generation module, further reduces the number of decoder layers, effectively lowering computational costs and model complexity while maintaining performance comparable to benchmark models, thus enhancing the model's application value. Optionally, the Transformer encoder has six layers, and the Transformer decoder has three layers.

[0015] Optionally, each layer of the Transformer decoder includes a self-attention network, a cross-attention network, and a feedforward neural network, wherein:

[0016] The self-attention network is used for information interaction between input queries to enhance the distinguishability between queries;

[0017] The cross-attention network is used to realize the information interaction between the input query and the feature vector, thereby obtaining the content information and location information of the target in the image;

[0018] The feedforward neural network is used to perform feature mapping and transformation on iteratively updated queries, further improving its feature representation capability.

[0019] This invention reduces the number of layers in the original Transformer decoder from 6 to 3, based on the prior content generation module. This strategy reduces the computational overhead of the Transformer decoder by approximately 50%, significantly accelerating the model's computational efficiency while effectively simplifying the model architecture. Overall, it achieves model lightweighting while maintaining performance comparable to benchmark models.

[0020] Optionally, step S2 uses two adjacent frames from the video sequence as input and optimizes the model based on a binary matching strategy, specifically including:

[0021] For adjacent frames t-1 and t, their predicted sets after processing by the model are represented as follows: Where N = N tck +N det N represents the total number of targets being tracked. tck With the total number of targets N det The sum of the prediction sets, one by one. Prediction results With real tag set Accurately associate the tags in the data;

[0022] The optimal association result based on the binary matching strategy is... Represented as: Let σ(j) represent the matching cost function, where σ(j) is the matching index of the predicted label;

[0023] The matching cost function In the formula, Indicates the target category is c j The predicted probability; λ cls For hyperparameters; Used to measure the true bounding box b j With predicted bounding box The similarities and differences between them; and according to Calculate, where, Denotes the generalized intersection-union ratio function; ||·||1 denotes the L1 norm; λ giou and All of these represent the hyperparameters of the model.

[0024] Alternatively, the overall loss function can be expressed as: In the formula, and These represent the Focalloss, L1loss, and generalized intersection-union (GUC) loss function values, respectively, achieved by minimizing the overall loss function. To optimize model parameters.

[0025] Based on the above method, the present invention also provides a lightweight video multi-target tracking system based on prior content, the key feature of which is that the system is equipped with a computer processing program, which is used to realize video multi-target tracking according to the lightweight video multi-target tracking method based on prior content described above.

[0026] Optionally, the video images received by the computer processing program are autonomous driving images, intelligent monitoring images, behavior analysis images, or sports analysis images.

[0027] Compared with existing technologies, the lightweight video multi-target tracking method and system based on prior content provided by this invention has the following beneficial effects:

[0028] First, a prior content generation strategy based on multi-scale feature maps is proposed, and a prior content generation module is designed to effectively improve object detection accuracy while accelerating model convergence. Then, a model lightweighting strategy is used, and the model structure is simplified by directly reducing the number of Transformer decoder layers, thereby reducing computational costs while meeting the requirements of real-time applications.

[0029] Instruction manual illustrations

[0030] Figure 1 A schematic diagram of a lightweight multi-target tracking method based on prior content provided for embodiments of the invention.

[0031] Figure 2 A schematic diagram of the prior content generation module structure provided for embodiments of the invention.

[0032] Figure 3 This is a schematic diagram of the Transformer decoder network structure provided in an embodiment of the present invention.

[0033] Figure 4 A visual comparison of the tracking results after adding PCGM to the MOT17-01 video sequence;

[0034] Figure 5 A visual comparison of the tracking results when PCGM is added to the MOT17-07 video sequence and the decoder structure is set to 3 layers;

[0035] Figure 6 A visualization comparison of the tracking effect when PCGM and PPEM are added to the MOT17-08 video sequence and the decoder is set to 3 layers. Detailed Implementation

[0036] The present invention will now be described in detail through specific embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention.

[0037] like Figure 1 As shown, this invention provides a lightweight video multi-target tracking method based on prior content, comprising the following steps:

[0038] S1: Establish a tracking model, which includes a CNN backbone network, a Transformer encoder, a Transformer decoder, and a prior content generation network; firstly, use the CNN backbone network and the Transformer encoder to extract and encode features from the input image to obtain the output feature representation. Secondly, The prior content is input into a network that generates a target query containing information about the target to be detected through its self-attention pooling. Then, and The common input is fed into the Transformer decoder, and the output embedding containing target category and location information is obtained through the deformation cross-attention mechanism. Finally, The prediction head is input, which outputs the final object detection bounding box and category, and the detection result is used to initialize the tracking query for the next frame. In subsequent frames, the merged queries and They are fed into the Transformer decoder together to achieve joint modeling of object detection and data correlation;

[0039] S2: To minimize the overall loss function To optimize the model parameters to achieve the desired result;

[0040] S3: Utilize the optimized model to achieve multi-target tracking in video.

[0041] pass Figure 2As can be seen, in specific implementation, the prior content generation network includes an attention map mapping network and a channel reweighting network, and the specific processing procedure is as follows:

[0042] Let all input feature maps be F = (f1, f2, ..., f N ), where the feature map of the nth layer is h and w represent the height and width of the feature map, respectively, and c represents the feature dimension, according to: Generate content query C det In the formula, The attention map mapping network, consisting of a series of convolutional networks, is responsible for outputting the mapping attention map. q represents the initial query count; SoftMax(·) represents the normalization function used to normalize f′. n Generate a weight matrix and compare it with f n Multiplication yields intermediate variables ψ(·) denotes a channel reweighting network used to reweight C′. det The channel weights are refined, specifically represented as: C det =SigMoid(σ(C′) det ))⊙C′ det In the formula, σ(·) represents a multilayer perceptron network based on linear mapping; SigMoid(·) represents the activation function; and ⊙ represents element-wise multiplication.

[0043] In this example, the Transformer encoder is configured with 6 layers, and the Transformer decoder with 3 layers. Figure 3 As can be seen, each layer of the Transformer decoder includes a self-attention network, a cross-attention network, and a feedforward neural network. The self-attention network is used for information interaction between input queries to enhance the discriminability between queries. The cross-attention network is used to realize information interaction between input queries and feature vectors, thereby obtaining the content and location information of the target in the image. The feedforward neural network is used to perform feature mapping and transformation on iteratively updated queries to further improve their feature representation capabilities.

[0044] To achieve model optimization, step S2 in this embodiment uses two adjacent frames from the video sequence as input and optimizes the model based on a binary matching strategy, specifically including:

[0045] For adjacent frames t-1 and t, their predicted sets after processing by the model are represented as follows: Where N = N tck +N det N represents the total number of targets being tracked. tck With the total number of targets N det The sum of the prediction sets, one by one. Prediction results With real tag set Accurately associate the tags in the data;

[0046] The optimal association result based on the binary matching strategy is... Represented as: Let σ(j) represent the matching cost function, where σ(j) is the matching index of the predicted label;

[0047] The matching cost function In the formula, Indicates the target category is c j The predicted probability; λ cls For hyperparameters; Used to measure the true bounding box b j With predicted bounding box The similarities and differences between them; and according to Calculate, where, Denotes the generalized intersection-union ratio function; ||·||1 denotes the L1 norm; λ giou and All of these represent the hyperparameters of the model.

[0048] In implementation, the overall system loss function is expressed as: In the formula, and These represent the Focalloss, L1loss, and generalized intersection-union (GUC) loss function values, respectively, achieved by minimizing the overall loss function. To optimize model parameters.

[0049] Furthermore, this embodiment also provides a lightweight video multi-target tracking system based on prior content. The system is equipped with a computer processing program, which is used to implement video multi-target tracking according to the aforementioned lightweight video multi-target tracking method based on prior content. Depending on the application scenario of the system, the video images received by the computer processing program may be autonomous driving images, intelligent monitoring images, behavior analysis images, or sports analysis images, etc.

[0050] To better demonstrate the effectiveness of the present invention, the applicant also conducted experimental verification and analysis on the publicly available MOT17 and MOT20 datasets during the research process to verify and analyze the effectiveness of the proposed method.

[0051] This experiment uses nine commonly used evaluation metrics and two datasets to quantitatively evaluate the method proposed in this paper. Specific information is as follows:

[0052] The MOT17 dataset consists of pedestrian video clips from seven different public scenes. Each video is divided into two segments, one for training and the other for testing. The video sequences in this dataset exhibit environmental diversity and variations in viewpoint and lighting, making it a commonly used benchmark dataset for evaluating tracking algorithms.

[0053] The MOT20 dataset consists of eight video clips from densely populated scenes, such as stadiums, train stations, and town squares, with 50% used for training and 50% for testing. Compared to MOT17, this dataset features more complex scenes, denser targets, and severe occlusion, making it suitable for evaluating the tracking performance of tracking algorithms in extremely crowded scenarios.

[0054] To verify the effectiveness of the multi-target tracking method, the evaluation metrics used in this invention include MOTA, IDF1, IDSw, MT, ML, HOTA, DetA, FP, and FN. Among them, MOTA and HOTA are used to evaluate the overall tracking performance of the model; IDF1, DetA, FP, and FN are used to measure the target detection performance of the model; and MT, ML, and IDSw focus on evaluating the data association performance of the model.

[0055] (1) Quantitative analysis:

[0056] Table 1 shows the results using common detection on the MOT17 test set, comprehensively evaluating the performance of CNN-based and Transformer-based tracking methods. The blue bold text indicates the best metrics for CNN-based tracking methods, and the black bold text indicates the best metrics for Transformer-based tracking methods. To verify the effectiveness of the prior content generation module, the number of Transformer decoder layers was set to a standard 6 layers to ensure fairness. It can be seen that the proposed method (EfficientFormer) outperforms existing comparative methods on the MOTA evaluation metric. For example, compared to the baseline model TrackFormer, this method achieves significant improvements on multiple evaluation metrics and effectively accelerates the model's convergence process. Specifically, MOTA and IDF1 are improved by 1.4% and 1.3%, respectively, and IDSw is reduced by 10^10; the training cycle is shortened from 50 epochs to 30 epochs, and the model convergence speed is improved by approximately 40%. These experimental results demonstrate that the proposed method, through the adaptive generation of target queries containing target feature information by the prior content generation module, not only helps to accelerate the model convergence speed but also effectively improves the model's tracking performance.

[0057] Table 1. Performance Comparison of Public Detection for Mainstream Video Multi-Object Tracking Methods on the MOT17 Test Set

[0058]

[0059] Table 2. Performance Comparison of Private Detection Methods for Mainstream Video Multi-Object Tracking on the MOT17 Test Set

[0060]

[0061] Specifically, to fully verify the effectiveness of EfficientFormer, the number of decoder layers was adjusted to 3 under the private detection criterion. Table 2 shows the results of using private detection on the MOT17 test set, and the table descriptions are consistent with Table 1. It can be seen that compared to the baseline model TrackFormer, this method improves on both FP and IDSw metrics, while maintaining other performance metrics largely unchanged. Furthermore, it remains competitive compared to other tracking methods.

[0062] Table 3 shows the results of using the proprietary detection method on the MOT20 test set, with the table descriptions consistent with Table 1. It can also be seen that, compared to the baseline model TrackFormer, this method achieves a certain improvement in the FP metric, while maintaining other performance metrics largely unchanged. Furthermore, compared with other tracking methods, this method still maintains a certain performance advantage.

[0063] Table 3. Performance Comparison of Private Detection Methods for Mainstream Video Multi-Object Tracking on the MOT20 Test Set

[0064]

[0065] (2) Qualitative Analysis

[0066] This qualitative comparison is divided into three parts: First, to fully verify the effectiveness of the prior content generation strategy, only the prior content generation module (PCGM) is added to the baseline model, while other settings remain unchanged. Second, with the prior content generation module added, the number of decoder layers is reduced to 3, while other settings remain consistent with the baseline model. Finally, based on the above experiments, the prior location embedding module proposed in Chapter 3 is introduced to analyze the impact of simultaneously adding prior content and prior location information on model performance; in this case, the number of decoder layers is also set to 3.

[0067] like Figure 4As shown, the left half displays the tracking results of the baseline model, while the right half shows the tracking results with the addition of the prior content generation module. It can be seen that in frame 2, the target that the baseline model failed to detect was successfully detected after adding prior content information. Furthermore, even with severe occlusion in frames 90 and 104, the improved model was still able to accurately detect and stably track the target. These visual comparisons validate the effectiveness of the prior content generation module, demonstrating that it significantly improves the model's target detection accuracy and tracking robustness.

[0068] like Figure 5 As shown, the left half presents the tracking results of the baseline model, and the right half presents the tracking results of the lightweight model proposed in this chapter. It can be seen that in frames 8 and 12, the tracking results of the lightweight model are basically consistent with the baseline model, with only slight omissions in the small target edge region marked by the red dashed box in the figure. Furthermore, in frame 13, the edge target undergoes slight movement, yet the lightweight model is still able to accurately detect and continuously track it, maintaining the same tracking performance as the baseline model. These comparative results demonstrate that even with a significantly simplified model architecture, the proposed lightweight model can still effectively maintain its baseline tracking performance, thus indicating the effectiveness of the lightweight design.

[0069] like Figure 6 As shown, the left half presents the tracking results of the baseline model, while the right half shows the tracking results after adding a Prior Location Embedding (PPEM) module to the lightweight model. It can be seen that in frames 11, 22, and 30, targets that the baseline model failed to detect were successfully detected and continuously tracked after adding the PPEM module. This indicates that providing both prior content and prior location information for target queries can further improve the performance of the lightweight model, ultimately surpassing the baseline model.

[0070] Finally, it should be noted that the existing methods and evaluation indicators listed in the above experiments are all well-known technologies in the field. For details, please refer to Reference 1: Fangao Zeng et al., MOTR: End-to-End Multiple-Object Tracking with Transformer and Reference 2: Tim Meinhardt et al., TrackFormer: Multi-Object Tracking with Transformers. The database used in the experiments is a publicly available database in the field. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight video multi-target tracking method based on prior content, characterized in that... Includes the following steps: S1: Establish a tracking model, which includes a CNN backbone network, a Transformer encoder, a Transformer decoder, and a prior content generation network; firstly, use the CNN backbone network and the Transformer encoder to extract and encode features from the input image to obtain the output feature representation. Secondly, The prior content is input into a network that generates a target query containing information about the target to be detected through its self-attention pooling. Then, and The common input is fed into the Transformer decoder, and the output embedding containing target category and location information is obtained through the deformation cross-attention mechanism. Finally, The prediction head is input, which outputs the final object detection bounding box and category, and the detection result is used to initialize the tracking query for the next frame. In subsequent frames, the merged queries and They are fed into the Transformer decoder together to achieve joint modeling of object detection and data correlation; S2: To minimize the overall loss function To optimize the model parameters to achieve the desired outcome; S3: Utilize the optimized model to achieve multi-target tracking in video.

2. The lightweight video multi-target tracking method based on prior content according to claim 1, characterized in that: The prior content generation network includes an attention map mapping network and a channel reweighting network. The specific processing procedure is as follows: Let all input feature maps be F = (f1, f2, ..., f N ), where the feature map of the nth layer is h and w represent the height and width of the feature map, respectively, and c represents the feature dimension, according to: Generate content query C det In the formula, The attention map mapping network, consisting of a series of convolutional networks, is responsible for outputting the mapping attention map. q represents the initial query count; SoftMax(·) represents the normalization function used to normalize f′. n Generate a weight matrix and compare it with f n Multiplication yields intermediate variables ψ(·) denotes a channel reweighting network used to reweight C′. det The channel weights are refined, specifically represented as: C det =SigMoid(σ(C′) det ))⊙C′ det In the formula, σ(·) represents a multilayer perceptron network based on linear mapping; SigMoid(·) represents the activation function; and ⊙ represents element-wise multiplication.

3. The lightweight video multi-target tracking method based on prior content according to claim 1 or 2, characterized in that: The Transformer encoder is set to 6 layers, and the Transformer decoder is set to 3 layers.

4. The lightweight video multi-target tracking method based on prior content according to claim 3, characterized in that: Each layer of the Transformer decoder includes a self-attention network, a cross-attention network, and a feedforward neural network, wherein: The self-attention network is used for information interaction between input queries to enhance the distinguishability between queries; The cross-attention network is used to realize the information interaction between the input query and the feature vector, thereby obtaining the content information and location information of the target in the image; The feedforward neural network is used to perform feature mapping and transformation on iteratively updated queries, further improving its feature representation capability.

5. The lightweight video multi-target tracking method based on prior content according to claim 4, characterized in that: Step S2 uses two adjacent frames from the video sequence as input and optimizes the model based on a binary matching strategy, specifically including: For adjacent frames t-1 and t, their predicted sets after processing by the model are represented as follows: Where N = N tck +N det N represents the total number of targets being tracked. tck With the total number of targets N det The sum of the prediction sets is used to predict each element. Prediction results With real tag set Accurately associate the tags in the data; The optimal association result based on the binary matching strategy is... Represented as: In the formula, Let σ(j) represent the matching cost function, where σ(j) is the matching index of the predicted label; The matching cost function In the formula, Indicates the target category is c j The predicted probability; λ cls For hyperparameters; Used to measure the true bounding box b j With predicted bounding box The similarities and differences between them; and according to Calculate, where, Denotes the generalized intersection-union ratio function; ||·||1 denotes the L1 norm; λ giou and All of these represent the hyperparameters of the model.

6. The lightweight video multi-target tracking method based on prior content according to claim 1 or 5, characterized in that: The overall loss function is expressed as: In the formula, and These represent the values ​​of the Focalloss, L1 loss, and generalized intersection-union (GUC) loss functions, respectively, achieved by minimizing the overall loss function. To optimize model parameters.

7. A lightweight video multi-target tracking system based on prior content, characterized in that: The system is equipped with a computer processing program, which is used to implement video multi-target tracking using the lightweight video multi-target tracking method based on prior content as described in any one of claims 1-6.

8. The lightweight video multi-target tracking system based on prior content according to claim 7, characterized in that: The computer processing program receives video images that are images of autonomous driving, intelligent monitoring, behavior analysis, or sports analysis.

Images