End-to-end multi-target tracking method based on lightweight Transform architecture

Abstract

The invention relates to an end-to-end multi-target tracking method based on a lightweight Transform architecture, and the method optimizes the tracking performance in a complex scene, solves the problems that an existing end-to-end multi-target tracking model is high in calculation complexity and is difficult to deploy in an embedded device, and enables the detection and tracking of an end-to-end normal form to be more accurate. And a higher precision level can be achieved on end side equipment with lower computing power. A lightweight global-local mixed attention mechanism replaces standard multi-head self-attention, reduces computational complexity, and maintains global context modeling capability at the same time. The mechanism adopts low-rank approximation and depth separable convolution to balance global dependency capture and local feature extraction. And proposing loss function bidirectional matching loss and id retention loss. The loss function is put forward on the basis of properties, capable of outputting target features, of transform query features, and the training method for end-to-end matching result optimization of the id is achieved. The tracking precision is improved, and the target identity consistency is ensured.

Description

Technical Field

[0001] This invention relates to an end-to-end multi-target tracking method based on a lightweight Transformer architecture. Background Technology

[0002] Multi-object tracking (MOT) is a crucial field in computer vision, enabling the simultaneous detection of multiple objects in a video sequence while maintaining their identification. Traditional methods often employ a detection-tracking approach (e.g., YOLOv10 + Bytetrack), separating detection and tracking into two independent stages. This approach decouples model training and engineering architecture, requiring optimization of deployment processes to achieve optimal deployment results. This increases the complexity of engineering design during deployment and the workload of debugging track performance across different scenarios.

[0003] Recently, some end-to-end methods based on Transformer (such as TransTrack and TrackFormer) have emerged that unify detection and tracking through a query mechanism, but they suffer from large number of parameters and high computational complexity, making them difficult to deploy on resource-constrained embedded devices. Summary of the Invention

[0004] This invention provides an end-to-end multi-target tracking method based on a lightweight Transformer architecture, comprising the following steps: S1: Input image information into the CNN backbone network, wherein the CNN backbone network extracts feature maps of the image based on the image information. Preferably, the CNN backbone network is a ResNet-50 network. S2: Input the feature map of the image into the Transformer encoder, and the Transformer encoder performs global context modeling on the feature map of the image to obtain a modeling map; S3: The Transformer decoder receives a set of learnable object queries and the modeling graph, interacts with the object queries and the modeling graph, and outputs a prediction result and an identity embedding vector fi; wherein, each object query can be understood as a vector that learns different possible target locations or features in the image of interest; S4: The prediction results are passed through the fully connected layer of the prediction head to output the final class label and bounding box coordinates; S5: Calculate the loss function based on the identity embedding vector fi, category label, and bounding box coordinates. Specifically, jointly optimize the detection and tracking tasks, employing a multi-task loss mechanism. Where Ldet is the detection loss, Ltrack is the tracking loss, and Lid is the identity preservation loss.

[0005] S3 specifically includes the following steps: S31: Processed using a standard self-attention mechanism to calculate the similarity between the query (Q), key (K), and value (V). The processing formula is as follows: S32: Lightweight global-local hybrid layer processing based on the similarity between query (Q), key (K), and value (V); Processing formula: in, U∈Rdk×r A low-rank projection matrix (r≪dk),DWConv This allows depthwise separable convolutions to capture local spatial information.

[0006] The tracking loss uses a two-way matched loss, as shown in the following formula: i represents the i-th ground truth target whose loss is currently being calculated, N represents the total number of ground truth targets in the current frame or current processing unit, σ represents the optimal matching relationship found through algorithms such as the Hungarian algorithm, σ(i) refers to the index of the predicted target that matches the i-th ground truth target, bi is the bounding box coordinates of the i-th ground truth target, usually represented as [center point x, center point y, width w, height h]; b^σ(i) refers to the bounding box coordinates of the model that match the i-th ground truth target.

[0007] Identity Preservation Loss Lid employs contrastive learning loss: Where fi represents the feature vector of the i-th target currently being processed; fpos represents the feature vector of the same target as fi (positive sample); fne represents the feature vector of the same target as fi (negative sample); and τ is a temperature parameter used to adjust the sharpness of the probability distribution.

[0008] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The method of this invention optimizes tracking performance in complex scenarios and solves the problems of high computational complexity and difficulty in deployment on embedded devices in existing end-to-end multi-target tracking models, enabling the end-to-end detection and tracking paradigm to achieve higher accuracy on edge devices with lower computing power.

[0009] 1. Lightweight Global-Local Hybrid Attention Mechanism (GLHA): An alternative to standard Multi-Head Self-Attention (MHSA), it reduces computational complexity while maintaining global context modeling capabilities. The mechanism employs low-rank approximation and depthwise separable convolutions to balance global dependency capture with local feature extraction.

[0010] 2. The proposed loss functions are bidirectional matching loss and ID preservation loss. These loss functions are based on the property of the transformer query feature extraction itself, which can output target features, and implement a training method to optimize the end-to-end ID matching results. This improves tracking accuracy and ensures the consistency of target identity. Detailed Implementation

[0011] This embodiment provides an end-to-end multi-target tracking method based on a lightweight Transformer architecture, including the following steps: S1: Input image information into the CNN backbone network, wherein the CNN backbone network extracts feature maps of the image based on the image information. Preferably, the CNN backbone network is a ResNet-50 network. S2: Input the feature map of the image into the Transformer encoder, and the Transformer encoder performs global context modeling on the feature map of the image to obtain a modeling map; S3: The Transformer decoder receives a set of learnable object queries and the modeling graph, interacts with the object queries and the modeling graph, and outputs a prediction result and an identity embedding vector fi; wherein, each object query can be understood as a vector that learns different possible target locations or features in the image of interest; S3 specifically includes the following steps: S31: Processed using a standard self-attention mechanism to calculate the similarity between the query (Q), key (K), and value (V). The processing formula is as follows: S32: Lightweight global-local hybrid layer processing based on the similarity between query (Q), key (K), and value (V); Processing formula: Where U∈Rdk×r is the low-rank projection matrix (r≪dk), and DWConv is the depthwise separable convolution that captures local spatial information.

[0012] The formula consists of two core parts: Low-rank approximation part (softmax term): The original query (Q), key (K), and value (V) matrices are projected into a low-dimensional space through a low-rank projection matrix U (with dimensions [d_k, r], where r << d_k, N). This enables subsequent attention calculations (computing (Q U)(K U)^T) to be mainly performed on dimension r rather than on the original high-dimensional d_k or sequence length N.

[0013] Effect: The complexity of the core operation (matrix multiplication) in attention calculation is significantly reduced from that of standard attention. The mathematical basis of low-rank approximation is matrix factorization theory, which assumes that a large weight matrix can be effectively approximated by the product of two smaller matrices.

[0014] Local window attention part (DWConv): The value (V) matrix is directly processed using depthwise separable convolution (DWConv). Depthwise separable convolution first performs spatial convolution independently on each channel (capturing local relationships) and then uses 1x1 convolution to integrate channel information. This operation only focuses on the local neighborhood of each position, avoiding global calculations.

[0015] Effect: The receptive field of the convolution operation is fixed (e.g., 3x3), which complements the local detailed information that may be lost in low-rank approximation.

[0016] S4: The prediction result passes through the fully connected layer of the prediction head and is used to output the final class label and bounding box coordinates; S5: Calculate the loss function based on the identity embedding vector f i, class label, and bounding box coordinates. Specifically, jointly optimize the detection and tracking tasks and adopt a multi-task loss where Ldet is the detection loss, Ltrack is the tracking loss, and Lid is the identity preservation loss The detection loss uses Focal Loss to handle class imbalance and has no special design, so it will not be elaborated here.

[0017] Among them, the tracking loss adopts a bipartite matching loss, and the formula is as follows: i represents the i-th real target for which the loss is currently being calculated, N represents the total number of real targets in the current frame or current processing unit, σ represents the optimal matching relationship found through the Hungarian algorithm, etc., σ(i) refers to the index of the predicted target that matches the i-th real target, and b i is the bounding box coordinates of the i-th real target (Ground Truth), usually represented as [center point x, center point y, width w, height h].

[0018] bσ(i) refers to the bounding box coordinates predicted by the model that match the i-th real target. The format is usually [center x, center y, width w, height h].

[0019] Identity Preservation Loss (Lid): Contrastive learning loss is used to reduce identity switching.

[0020] Here, fi represents the feature vector (or identity embedding vector) of the i-th target currently being processed. This vector is typically generated by the feature extractor of the neural network and is designed to encode the target's identity information. fpos represents the feature vector belonging to the same target as fi (positive sample), for example, this might be the feature vector of the same target in the previous frame. fneq represents the feature vector not belonging to the same target as fi (negative sample), for example, this might be the feature vector of the same target in the previous frame. τ is a temperature parameter used to adjust the sharpness of the probability distribution. A smaller τ will cause the model to focus more on those difficult-to-distinguish sample pairs.

[0021] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An end-to-end multi-target tracking method based on a lightweight Transformer architecture, characterized in that, Includes the following steps: S1: Input image information into the CNN backbone network, wherein the CNN backbone network extracts feature maps of the image based on the image information. Preferably, the CNN backbone network is a ResNet-50 network. S2: Input the feature map of the image into the Transformer encoder, and the Transformer encoder performs global context modeling on the feature map of the image to obtain a modeling map; S3: The Transformer decoder receives a set of learnable object queries and the modeling graph, interacts with the object queries and the modeling graph, and outputs a prediction result and an identity embedding vector fi; wherein, each object query can be understood as a vector that learns different possible target locations or features in the image of interest; S4: The prediction results are passed through the fully connected layer of the prediction head to output the final class label and bounding box coordinates; S5: Calculate the loss function based on the identity embedding vector fi, category label, and bounding box coordinates. Specifically, jointly optimize the detection and tracking tasks, employing a multi-task loss mechanism. Where Ldet is the detection loss, Ltrack is the tracking loss, and Lid is the identity preservation loss.

2. The end-to-end multi-target tracking method based on a lightweight Transformer architecture according to claim 1, characterized in that, S3 specifically includes the following steps: S31: Processed using a standard self-attention mechanism to calculate the similarity between the query (Q), key (K), and value (V). The processing formula is as follows: S32: Lightweight global-local hybrid layer processing based on the similarity between query (Q), key (K), and value (V); Processing formula: in, U∈Rdk×r R is a low-rank projection matrix (r≪dk), and DWConv is a depthwise separable convolution that captures local spatial information.

3. The end-to-end multi-target tracking method based on a lightweight Transformer architecture according to claim 1, characterized in that: The tracking loss uses a two-way matched loss, as shown in the following formula: i represents the i-th ground truth target whose loss is currently being calculated, N represents the total number of ground truth targets in the current frame or current processing unit, σ represents the optimal matching relationship found through algorithms such as the Hungarian algorithm, σ(i) refers to the index of the predicted target that matches the i-th ground truth target, bi is the bounding box coordinates of the i-th ground truth target, usually represented as [center point x, center point y, width w, height h]; bσ(i) refers to the bounding box coordinates of the model that match the i-th ground truth target.

4. The end-to-end multi-target tracking method based on a lightweight Transformer architecture according to claim 1, characterized in that: Identity Preservation Loss Lid employs contrastive learning loss: Where fi represents the feature vector of the i-th target currently being processed; fpos represents the feature vector of the same target as fi (positive sample); fne represents the feature vector of the same target as fi (negative sample); and τ is a temperature parameter used to adjust the sharpness of the probability distribution.

Images