A multi-target tracking method based on feature decoupling and feature enhancement learning

Abstract

A multi-target tracking method based on feature decoupling and feature enhancement learning uses feature decoupling to extract the features required for detection and re-identification tasks, respectively. Two newly designed feature enhancement modules based on an attention mechanism are introduced to further differentiate and strengthen the features of these two components. Based on the obtained enhanced decoupled features, a new lost target retrieval strategy is designed to further retrieve lost targets, thereby enhancing the synergy between tasks and achieving more accurate tracking. This method alleviates the problems of internal task competition within the model and the impact of shared neural networks on feature effects, improving the overall tracking performance of the model and achieving better tracking results.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and artificial intelligence, and in particular relates to a single-stage multi-target tracking method based on feature decoupling and feature enhancement. Background Art

[0002] Multi-object tracking (MOT) aims to accurately estimate the positions and identities of multiple objects in a video sequence. It is one of the most fundamental and challenging tasks in computer vision, with widespread applications in fields such as intelligent surveillance, autonomous driving, and biomedicine. Over the past decade, due to advances in object detection technology, researchers have generally adopted the tracking-by-detection (TBD) paradigm to address this task. Traditional TBD methods typically use two independent models: a detector model to generate bounding boxes for the objects, and an embedding-based tracking model to extract object features and establish object associations between frames based on the features and motion information. Although these TBD methods can achieve good tracking results, the separation of the two networks in TBD methods leads to problems such as the inability to share features between models and insufficient feature utilization, which in turn results in high computational costs and makes real-time tracking difficult to achieve.

[0003] In recent years, joint detection and tracking (JDT) methods have achieved a good balance between inference speed and tracking accuracy by adopting an end-to-end strategy, thereby alleviating some of the issues of tracking-by-detection (TBD) methods. TransTrack establishes a new JDT framework based on the query-key mechanism of the Transformer. It tracks objects in the current frame by querying relevant features from the previous frame in the sequence. Transcenter further proposes a dense pixel-level multi-scale query method based on center point heatmaps to enhance object detection in crowded scenes. FairMOT recognizes the inherent conflict between the objectives of detection and tracking: detection aims to maximize inter-class variance, while tracking aims to maximize intra-class variance. It alleviates this problem by constructing two parallel branches for object detection and feature extraction. RelationTrack further proposes learning re-identification (reID) embeddings with global information to distinguish the features required by the two tasks. However, it still does not fully resolve the internal competition between detection and recognition tasks. In complex scenes with severe occlusion and background clutter, the features extracted due to this task competition are insufficient for accurate object recognition, leading to problems such as object loss and identity switching. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, this paper provides a lightweight and effective multi-target tracking method based on feature decoupling and feature enhancement learning. In this method, a new dual-branch feature enhancement learning network and a new lost target re-finding strategy are proposed. Through the comprehensive application of this network and strategy, a more powerful tool is brought to the field of multi-target tracking.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A multi-target tracking method based on feature decoupling and feature enhancement learning includes the following steps:

[0007] Step 1: Obtain image features and perform feature decoupling and enhancement. The process is as follows:

[0008] S11, input pedestrian multi-target tracking sequence, each frame in the sequence contains multiple pedestrian tracking targets;

[0009] S12: Input pedestrian image data into the backbone network and obtain a four-layer feature map. Then, perform feature decoupling and differentiation based on the four-layer feature map. Then, introduce the modified attention mechanism to enhance the task-oriented nature of the decoupled and differentiated image features.

[0010] S13, two parts of image features after feature decoupling and feature enhancement;

[0011] Step 2: Target tracking. The process is as follows:

[0012] S21, taking the two parts of image features output by S13 as input;

[0013] S22. After the input features pass through a specific classification head, box_size, center_off, heatmap and reid data are obtained. According to these data, target association is completed based on the joint detection and tracking paradigm;

[0014] S23, target association: The multi-target tracking model outputs data from S22 and retains the coordinate position of each tracked target within a specific frame sequence, while considering the re-identification features of each target for target association and re-recovery;

[0015] S24, comprehensive loss: The multi-target tracking model comprehensively considers target detection loss and target association loss to optimize overall performance and balance the competition among tasks within the model.

[0016] S25. After the balance loss calculation, the multi-target tracking model has good robustness and its lightweight size can maintain the real-time tracking requirements.

[0017] Furthermore, in S12, the feature decoupling process is as follows:

[0018] Firstly, DLA-34 is used as the backbone network. Considering the different semantic levels of features required for detection and re-identification tasks, a dual-branch feature fusion network is proposed. This network uses two different feature aggregation methods to obtain the feature representations required for detection and re-identification.

[0019] For the detection part, the reconstructed IDA-UP structure is used, which is based on operating the upsampled three-layer deep feature maps and fusing them with the shallow feature maps, and is called a top-down feature fusion strategy;

[0020] For the re-identification part, a bottom-up feature aggregation strategy is proposed. Represents the feature map input to the network, where N represents the number of layers with feature maps of different resolutions extracted by the backbone network, Fi represents the feature map of the i-th layer, and then the bottom-up feature fusion strategy is expressed as:

[0021]

[0022] in, Represents the final fused features of each layer, represents the features obtained after fusion of the previous layer, UpSample(·) represents the upsampling operation consisting of deformable convolution and deconvolution, Conv 1×1 (·) represents a convolutional layer of size 1*1, which is used to change the number of channels of the feature, and σ(·) represents the Sigmoid activation layer.

[0023] Furthermore, in S12, an attention mechanism is introduced for feature enhancement, and a spatial feature enhancement module SFEM is proposed. SFEM is a superposition of a two-layer recursive spatial attention mechanism. Considering the feature fusion strategy of the constructed top-down detection part, the highest-level feature map obtained by the backbone is passed into the SFEM module. The fusion formula of each layer is:

[0024]

[0025] where Q,K,V∈R C×H×W is the input feature F C×H×W The matrix obtained by three learnable weight vectors, Q u represents any unit eigenvector in the Q matrix, ε i,u ,φ i,u represents the i-th value in the set of all feature vectors in the same row or column as u in the K and V matrices, respectively. Softmax(·) is used to change the number of feature channels.

[0026] This paper proposes an identity recognition feature enhancement module (IEM) adapted to pixel-level fine-grained tasks. IEM completely collapses features in one direction while maintaining high resolution of features in the orthogonal direction. It then increases the dynamic range of attention through Softmax() and Sigmoid() operations on the bottleneck vector, thereby enhancing the feature vector. The channel attention formula of IEM is:

[0027] A c (X) = F SG [W z ((σ1(W v (X))×F SM (σ2(W q (X)))))]

[0028] where X∈R C×H×W , W q ,W v ,W z They are convolutional layers of size 1X1, σ1 and σ2 are two vector reset factors, F SM (·) is the SoftMax operator, W v ,W z The number of internal channels is C / 2, A c (X)∈R C×1×1 , output Z c =A c (X) c X∈R C×H×W ;

[0029] The spatial attention formula of IEM is:

[0030] A s (X) = F SG [σ3(F SM (σ1(F GP (W q (X))))×σ2(W v (X)))]

[0031] where X∈R C×H×W , W q ,W v They are convolutional layers of size 1X1, σ1, σ2, σ3 are three vector reset factors, F SM (·) is the SoftMax operator, output Z p =A p (X) p X∈R C×H×W ;

[0032] The above two branches are added in parallel to get the final result:

[0033] PSAP(X)=Z c +Z p .

[0034] In S23, the multi-target tracking model adopts a tracking strategy based on IOU and appearance embedding distance. First, the appearance embedding information is used to compensate for the drift of the Kalman filter to obtain the final predicted target center position, and the average motion speed (V) of the target center during the retention time is calculated. i ,V j ), considering the average number of frames processed per second t and the number of frames k that the target has lost, the Kalman filter is used to select the prediction center (C i ,C y ) The appearance embedding vectors in the range are calculated and compared with the appearance embedding vector E of the lost target l The distance between them is used to determine whether they belong to the same trajectory according to the formula:

[0035]

[0036] Among them D min Indicates the prediction center (C i ,C y ) in the N×N range near the appearance embedding vector and E l The cosine distance between r Indicates the matching threshold of trajectory recovery. If To represent the appearance embedding vector on pixel (i, j), then D min The calculation formula is as follows:

[0037]

[0038] Among them F cosdistance (·) indicates calculating the cosine distance between two vectors. If the target object of the lost track is finally found, the center point of the found object is set as the center point of the track in the current frame, so that its size is consistent with the size of the tracked target in the previous frame.

[0039] The technical concept of the present invention is: in order to alleviate the problem of internal task competition in the multi-target tracking method of joint detection and re-identification, the present invention proposes a new feature decoupling network, which integrates top-down and bottom-up feature extraction strategies, and differentiates the features required for the internal detection task and re-identification task of the model; by introducing and modifying a specific attention mechanism network, the specified dimensionality of the differentiated features is enhanced, thereby making the acquired features more task-oriented; through the proposed new tracking strategy, the model can maintain robustness in complex scenes with severe occlusion and cluttered background, and fully alleviate the problems of easy target loss and easy identity switching.

[0040] The beneficial effects of the present invention are mainly manifested in:

[0041] (1) A new re-identification feature construction branch is proposed. Compared with existing technologies, this method has significant advantages in obtaining re-identification features and reducing problems such as identity switching and misidentification. The traditional feature decoupling method still divides the unified image features into two parts required for the detection task and the re-identification task. However, this method reconstructs the low-dimensional and high-dimensional feature branches of the two tasks based on the feature map obtained by the backbone network and obtains the corresponding features. This greatly alleviates the problems of task competition within the model and the impact of the shared neural network on the feature effect.

[0042] (2) The attention mechanism suitable for detection / re-identification tasks is introduced into the multi-target tracking method for joint detection and re-identification. Because the method of the present invention adjusts and modifies the specific attention mechanism in terms of dimensions, the feature enhancement of the attention mechanism performs better on the overall task, thereby improving the overall tracking performance of the model.

[0043] (3) A new tracking strategy is proposed. This tracking strategy can combine appearance distance to help recover the detection target lost due to temporary occlusion. At the same time, it can help the Kalman filter better locate the tracking target, update the parameters of the Kalman filter and reduce the cumulative error of the Kalman filter, so that the model can achieve better tracking effect.

[0044] (4) It provides more flexible usage. The feature decoupling reconstruction branch, attention mechanism, and tracking method proposed in this method can be easily inserted into various models to achieve a plug-and-play effect. Extensive experiments conducted on multiple comparative experiments and datasets show that the method of the present invention can enhance the incremental learning performance of existing models without adding additional costs. Ablation studies further verify the effectiveness of each component of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a principle block diagram of the multi-target tracking method based on feature decoupling and feature enhancement learning. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings.

[0047] Reference Figure 1 , a multi-target tracking method based on feature decoupling and feature enhancement learning, comprising the following steps:

[0048] Step 1: Obtain image features and perform feature decoupling and enhancement. The process is as follows:

[0049] S11, input pedestrian multi-target tracking sequence, each frame in the sequence contains multiple pedestrian tracking targets;

[0050] S12: Input pedestrian image data into the backbone network and obtain a four-layer feature map. Then, perform feature decoupling and differentiation based on the four-layer feature map. Then, introduce the modified attention mechanism to enhance the task-oriented nature of the decoupled and differentiated image features.

[0051] The goal of feature decoupling is to provide different task branches with the required features of different dimensions in the same model, thereby alleviating performance issues caused by feature ambiguity due to task competition. The process is as follows:

[0052] First, DLA-34 is adopted as the backbone network to achieve a good balance between detection speed and accuracy. Considering the different semantic levels of features required for detection and re-identification tasks, a dual-branch feature fusion network is proposed, which uses two different feature aggregation methods to obtain the feature representations required for detection and re-identification.

[0053] For the detection part, the reconstructed IDA-UP structure is used, which is based on operating on the upsampled three-layer deep feature maps and fusing them with the shallow feature maps, and is called a top-down feature fusion strategy;

[0054] For the re-identification part, it requires shallower features to distinguish different samples within each category; a bottom-up feature aggregation strategy is proposed, Represents the feature map input to the network, where N represents the number of layers with feature maps of different resolutions extracted by the backbone network, Fi represents the feature map of the i-th layer, and then the bottom-up feature fusion strategy is expressed as:

[0055]

[0056] in, Represents the final fused features of each layer, represents the features obtained after fusion of the previous layer, UpSample(·) represents the upsampling operation consisting of deformable convolution and deconvolution, Conv 1×1 (·) represents a convolutional layer of size 1*1, which is used to change the number of channels of the feature, and σ(·) represents the Sigmoid activation layer.

[0057] The goal of feature enhancement is to enhance the image features obtained by decoupling and differentiation in a task-oriented manner, thereby obtaining more discriminative detection features and re-identification features. Therefore, the attention mechanism is introduced for feature enhancement.

[0058] To strengthen the network's focus on foreground information and suppress background noise, a spatial feature enhancement module (SFEM) is proposed to improve the robustness of feature encoding to complex environments and object sizes, while also increasing its global representation. SFEM is a two-layer recursive spatial attention mechanism stacked together. Compared to other attention mechanisms, this approach is simpler and more efficient. Considering the feature fusion strategy of the constructed top-down detection component, the highest-level feature map obtained by the backbone is passed to the SFEM module. The fusion formula for each layer is:

[0059]

[0060] where Q,K,V∈R C×H×W is the input feature F C×H×W The matrix obtained by three learnable weight vectors, Q u represents any unit eigenvector in the Q matrix, ε i,u ,φ i,u represents the i-th value in the set of all feature vectors in the same row or column as u in the K and V matrices, respectively. Softmax(·) is used to change the number of feature channels.

[0061] Unlike detection tasks, re-identification tasks require obtaining higher-level features of the object to distinguish each instance within the object class. We propose an identity recognition feature enhancement module (IEM) adapted to pixel-level fine-grained tasks. IEM completely collapses features in one direction while maintaining high resolution of features in its orthogonal direction. It then increases the dynamic range of attention through Softmax() and Sigmoid() operations on the bottleneck vector (the smallest feature vector in the attention block), thereby enhancing the feature vector. The channel attention formula of IEM is:

[0062] A c (X) = F SG [W z ((σ1(W v (X))×F SM (σ2(W q (X)))))]

[0063] where X∈R C×H×W , W q ,W v ,W z They are convolutional layers of size 1X1, σ1 and σ2 are two vector reset factors, F SM (·) is the SoftMax operator, W v ,W z The number of internal channels is C / 2, A c (X)∈R C×1×1 , output Z c =Ac (X) c X∈R C×H×W ;

[0064] The spatial attention formula of IEM is:

[0065] A s (X) = F SG [σ3(F SM (σ1(F GP (W q (X))))×σ2(W v (X)))]

[0066] where X∈R C×H×W , W q ,W v They are convolutional layers of size 1X1, σ1, σ2, σ3 are three vector reset factors, F SM (·) is the SoftMax operator, output Z p =A p (X) p X∈R C×H×W ;

[0067] The above two branches are added in parallel to get the final result:

[0068] PSAP(X)=Z c +Z p .

[0069] S13, two parts of image features after feature decoupling and feature enhancement;

[0070] Step 2: Target tracking. The process is as follows:

[0071] S21, taking the two parts of image features output by S13 as input;

[0072] S22. After the input features pass through a specific classification head, box_size, center_off, heatmap and reid data are obtained. According to these data, target association is completed based on the joint detection and tracking paradigm;

[0073] S23, target association: The multi-target tracking model outputs data from S22 and retains the coordinate position of each tracked target within a specific frame sequence, while considering the re-identification features of each target for target association and re-recovery;

[0074] The multi-target tracking model adopts a tracking strategy based on IOU and appearance embedding distance. It follows ByteTrack's principle of secondary matching of high-resolution frames and low-resolution frames. After the secondary matching, it relies on the precise re-identification features obtained by the feature decoupling enhancement framework to re-detect and recover lost targets.

[0075] When the target is lost due to a short blockage or other reasons, the appearance embedding information is first used to compensate for the drift of the Kalman filter to obtain the final predicted target center position, and the average motion speed (V) of the target center in the retention time (30 frames) is calculated. i ,V j ), considering the average number of frames processed per second t and the number of frames k that the target has lost, the Kalman filter is used to select the prediction center (C i ,C y ) The appearance embedding vectors in the range are calculated and compared with the appearance embedding vector E of the lost target l The distance between them is used to determine whether they belong to the same trajectory according to the formula:

[0076]

[0077] Among them D min Indicates the prediction center (C i ,C y ) in the N×N range near the appearance embedding vector and E l The cosine distance between r Indicates the matching threshold of trajectory recovery. If To represent the appearance embedding vector on pixel (i, j), then D min The calculation formula is as follows:

[0078]

[0079] Among them F cosdistance (·) indicates calculating the cosine distance between two vectors. If the target object of the lost track is finally found, the center point of the found object is set as the center point of the track in the current frame, so that its size is consistent with the size of the tracked target in the previous frame.

[0080] S24, comprehensive loss: The multi-target tracking model comprehensively considers target detection loss and target association loss to optimize overall performance and balance the competition among tasks within the model.

[0081] S25. After the balance loss calculation, the multi-target tracking model has good robustness and its lightweight size can maintain the real-time tracking requirements.

[0082] In this example, to evaluate the proposed method, the DLA-34 network was used as the backbone network for feature extraction. The input image size was 1088 × 608, and the feature map size was 272 × 152. The model was trained for 30 epochs on an NVIDIA GeForce RTX 3090 GPU, using the Adam optimizer to tune parameters. The initial learning rate was set to 1 × 10⁻⁵. The batch size was set to 12.

[0083] We use the exact same dataset as FairMOT for training, including ETH, CityPerson, CalTech, CUHK-SYSU, PRW, MOT17, and CrowdHuman, for a total of 20,086 images. We evaluate the tracker on two tracking benchmarks: MOT17 and MOT20.

[0084] Tables 1 and 2 show the ablation study conducted on the MOT17 validation set and the experimental results on different datasets.

[0085]

[0086] Table 1

[0087]

[0088] Table 2

[0089] By incorporating feature decoupling and enhancement into FairMOT, the competition between detection and re-ID tasks is alleviated. The tracker's performance improves modestly in terms of MOTA (+0.6), IDF1 (+1.9), and TPR (+0.3). With the introduction of the spatial feature enhancement (SFE) and identity feature enhancement (IFE) modules, the tracker's performance further improves in terms of MOTA (+1.3), IDF1 (+2.4), and TPR (+0.8). This demonstrates that the features for detection and re-ID tasks are enhanced in a targeted manner, exhibiting stronger task specificity. When combined with the re-ID (RF) strategy, the model framework achieves the highest performance improvement, surpassing the baseline in MOTA (+2.6), IDF1 (+4.3), and TPR (+1.0), while reducing identity switches (IDs) by 187, demonstrating that the RF strategy effectively enhances identification capabilities. Furthermore, the three modules complement each other, collectively contributing to performance improvements in both detection and tracking tasks.

[0090] Table 3 is the experimental results obtained on the MOTChallenge;

[0091]

[0092] Table 3

[0093] The results show that this method can be plug-and-play with existing methods, improving the accuracy of existing models to varying degrees and significantly improving model performance.

[0094] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A multi-target tracking method based on feature decoupling and feature enhancement learning, characterized in that: The method comprises the following steps: Step 1: Obtain image features and perform feature decoupling and enhancement. The process is as follows: S11. Input a pedestrian multi-target tracking sequence, where each frame contains multiple pedestrian tracking targets. S12: Input pedestrian image data into the backbone network and obtain a four-layer feature map. Then, perform feature decoupling and differentiation based on the four-layer feature map. Then, introduce the modified attention mechanism to enhance the task-oriented nature of the decoupled and differentiated image features. S13, two parts of image features after feature decoupling and feature enhancement; Step 2: Target tracking. The process is as follows: S21, taking the two parts of image features output by S13 as input; S22. After the input features pass through a specific classification head, box_size, center_off, heatmap and reid data are obtained. According to these data, target association is completed based on the joint detection and tracking paradigm; S23, target association: The multi-target tracking model outputs data from S22 and retains the coordinate position of each tracked target within a specific frame sequence, while considering the re-identification features of each target for target association and re-recovery; S24, comprehensive loss: The multi-target tracking model comprehensively considers target detection loss and target association loss to optimize overall performance and balance the competition among tasks within the model. S25. After the balanced loss calculation, the multi-target tracking model has good robustness and its lightweight size can maintain the real-time tracking requirements; In S12, an attention mechanism is introduced for feature enhancement, and a spatial feature enhancement module SFEM is proposed. SFEM is a superposition of a recursive spatial attention mechanism with a depth of two layers. Considering the feature fusion strategy of the constructed top-down detection part, the highest-level feature map obtained by the backbone is passed into the SFEM module. The fusion formula of each layer is: , in is the input feature The matrix obtained by three learnable weight vectors, represents any unit eigenvector in the Q matrix, , Respectively represents the i-th value in the set of all eigenvectors in the same row or column as u in the K and V matrices, Used to change the number of channels of the feature; This paper proposes an identity recognition feature enhancement module (IEM) adapted to pixel-level fine-grained tasks. IEM completely collapses features in one direction while maintaining high resolution of features in the orthogonal direction. It then increases the dynamic range of attention through Softmax() and Sigmoid() operations on the bottleneck vector, thereby enhancing the feature vector. The channel attention formula of IEM is: in , They are convolutional layers of size 1X1, are two vector reset factors, is the SoftMax operator, The number of internal channels is C / 2, , output ; The spatial attention formula of IEM is: in , They are convolutional layers of size 1X1, are three vector reset factors, It is the SoftMax operator, output ; The above two branches are added in parallel to get the final result: 。 2. The multi-target tracking method based on feature decoupling and feature enhancement learning according to claim 1, characterized in that: In S12, the process of feature decoupling is as follows: Firstly, DLA-34 is used as the backbone network. Considering the different semantic levels of features required for detection and re-identification tasks, a dual-branch feature fusion network is proposed. This network uses two different feature aggregation methods to obtain the feature representations required for detection and re-identification. For the detection part, the reconstructed IDA-UP structure is used, which is based on operating the upsampled three-layer deep feature maps and fusing them with the shallow feature maps, and is called a top-down feature fusion strategy; For the re-identification part, a bottom-up feature aggregation strategy is proposed. represents the feature map input to the network, where N represents the number of layers with feature maps of different resolutions extracted by the backbone network, Represents the i-th layer feature map, then the bottom-up feature fusion strategy is expressed as: in, Represents the final fused features of each layer, Represents the features obtained after fusion of the previous layer, represents an upsampling operation consisting of deformable convolution and deconvolution, Represents a convolutional layer of size 1*1, which is used to change the number of channels of the feature. Represents a Sigmoid activation layer.

3. The multi-target tracking method based on feature decoupling and feature enhancement learning according to claim 1 or 2, characterized in that: In the S23, the multi-target tracking model adopts a tracking strategy based on IOU and appearance embedding distance. First, the appearance embedding information is used to compensate for the drift of the Kalman filter to obtain the final predicted target center position, and the average movement speed of the target center in the retention time is calculated. , considering the average number of frames processed per second t and the number of frames k that the target has lost, the Kalman filter is used to select the prediction center Nearby The appearance embedding vectors in the range and calculate their difference with the appearance embedding vector of the lost target The distance between them is used to determine whether they belong to the same trajectory according to the formula: in Indicates that the forecast center Nearby N The appearance embedding vector in the range N is The cosine distance between Indicates the matching threshold of trajectory recovery. If To represent the appearance embedding vector on pixel (i, j), then The calculation formula is as follows: in It means calculating the cosine distance between two vectors. If the target object of the lost track is finally found, the center point of the found track is set as the center point of the current frame, so that its size is consistent with the size of the tracked target in the previous frame.

Images