A Multi-Target Tracking Method with Reference Point Prior Position Embedding and Adaptive Update

Abstract

This invention relates to the field of machine vision technology, specifically disclosing a multi-target tracking method with reference point prior position embedding and adaptive updating. It constructs a multi-target tracking network for multi-target tracking, within which a prior position generation network based on target queries is designed. This prior position generation network generates target queries with explicit prior position information. This prior position information facilitates effective keypoint sampling, solving the problem of lacking prior position information for target query reference points, thereby improving target detection performance. Furthermore, a tracking reference point prediction network based on tracking query reference points is designed. This tracking reference point prediction network generates new tracking query reference points for tracking queries, effectively reducing the deviation between the tracking reference point position and the target center position, achieving accurate positioning of the tracking reference point, and thus improving target tracking performance.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a multi-target tracking method with reference point prior position embedding and adaptive update. Background Technology

[0002] Multi-object tracking (MOT) is a highly challenging problem in the field of computer vision. MOT technology continuously tracks the states of multiple targets in a video sequence to generate their motion trajectories. This technology holds significant promise for applications in video surveillance, autonomous driving, and human-computer interaction. However, achieving effective multi-object tracking remains a challenging problem due to occlusion, lighting variations, motion blur, and background interference in complex scenes.

[0003] Currently, video multi-target tracking methods are mainly divided into tracking methods based on target detection and data association, integrated tracking methods based on joint target detection and data association, and tracking methods based on attention mechanisms. (1) Tracking methods based on target detection and data association perform data association based on target detection results, which is a two-stage multi-target tracking. Therefore, the model tracking effect largely depends on the detector performance, and the tracking robustness in complex scenarios such as frequent occlusion and appearance changes is still insufficient. (2) Integrated tracking methods based on joint target detection and data association combine target detection and data association to achieve integrated multi-target tracking; the main problem of this type of tracking method is that there is adversarial relationship between the target detection and data association subtasks. (3) Multi-target tracking methods based on attention mechanisms are an end-to-end tracking paradigm, which introduces Object Query and Track Query to achieve integrated target detection and data association. However, the reference points of Object Query and Track Query in this type of tracking method have the problems of lacking position prior and inaccuracy, respectively. First, for Object Query, the reference point is randomly initialized, lacking explicit prior location information, making target localization and model optimization difficult. Second, for Track Query, the reference point is set to the center of the bounding box of the tracked target in the previous frame; however, due to target motion, especially in high-speed and low-frame-rate scenarios, the center position of the same target differs between adjacent video frames, causing a localization error between the Track Query's reference point and the actual center position of the target to be associated, thus affecting data association performance.

[0004] Attention-based multi-object tracking methods are currently the mainstream approach in the field of video multi-object tracking. However, how to address the lack of prior location information and inaccuracy of reference points in these methods has become a key issue that urgently needs to be addressed in current research. Summary of the Invention

[0005] This invention provides a multi-target tracking method with reference point prior position embedding and adaptive update. The technical problem it solves is that traditional attention-based multi-target tracking methods lack prior position information for the reference points used in target queries and the reference points used in tracking queries are inaccurate, which affects the performance of multi-target tracking.

[0006] To address the above technical problems, this invention provides a multi-target tracking method with reference point prior position embedding and adaptive update, comprising: constructing a multi-target tracking network;

[0007] The multi-target tracking network includes a backbone network, a deformation attention network, a priori position generation network, a tracking reference point prediction network, and a prediction head. The deformation attention network includes an encoder and a decoder. The backbone network, the encoder, and the decoder are sequentially connected. The backbone network and the encoder are used to perform feature extraction and feature encoding aggregation on the input image, respectively, to obtain aggregated features which are output to the decoder. The priori position generation network and the tracking reference point prediction network are both connected to the decoder. The priori position generation network is used to generate a target query with priori position information for the initial target query of each frame and output it to the decoder. The tracking reference point prediction network generates the tracking reference point for the current frame from the tracking query output in the previous frame and outputs it to the decoder.

[0008] In the initial frame, the decoder generates an output embedding into the prediction head based on the target query with prior location information and the aggregated features of the initial frame; the prediction head generates the bounding box and category of the corresponding target based on the output embedding of the initial frame, and initializes it as the tracking query for the next frame;

[0009] In the current frame, which is not the initial frame, the decoder generates the output of the current frame and embeds it into the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates the bounding box and category of the corresponding target based on the output embedding of the current frame and initializes it as the tracking query of the next frame.

[0010] Furthermore, the target query q generated by the prior location generation network det Including content query c det and location query p det Location query p det Represented as:

[0011]

[0012] Among them, R det g represents all reference points for the target query. sin (·) indicates a position encoding operation. This indicates a position embedding operation.

[0013] Furthermore, let Then P det Specifically:

[0014]

[0015] Where W1 and W2 are the weights of the first and second linear mappings, respectively, b1 and b2 are the bias terms of the first and second linear mappings, respectively, and ReLU(·) represents the ReLU activation function.

[0016] Furthermore, the tracking reference point prediction network predicts all tracking reference points r in the current frame. tck Adjustments are made to obtain the adjusted reference point.

[0017]

[0018] Where, ΔR tck For tracking query Q tck The predicted tracking reference point offset is specifically expressed as follows:

[0019] ΔR tck =ψ θ (Q tck )

[0020] Where, ψ θ (·) represents the reference point prediction model, where θ is the parameter of the model.

[0021] Furthermore, ΔR tck Specifically, it is expressed as follows:

[0022] ΔR tck =ψ θ (Q tck )=(W5(ReLU(W4(ReLU(W3Q tck +b3))+b4))+b5)

[0023] in, The weights and biases, i.e., parameters θ, represent the first, second, and third layers of the linear mapping in the tracking reference point prediction model.

[0024] Furthermore, the method also includes:

[0025] The constructed multi-target tracking network is trained;

[0026] The prediction sets of adjacent t-1 frames and t frame after passing through the multi-target tracking network are represented as follows: Where N = N tck +N det N represents the number of targets being tracked. tck With the number of targets N det The total number of targets; during training, the optimization objective for the multi-target tracking network is to sequentially optimize the prediction set. Prediction results With real tag set The label y in j To achieve accurate correlation, K represents the total number of real targets in the current frame.

[0027] Furthermore, for the prediction set The optimal association result with the true label set y Represented as:

[0028]

[0029] in, Represents the j-th real label y j With the σ(j)th predicted label The matching cost function between them, where σ(j) is the predicted label. The matching index.

[0030] Furthermore, the matching cost function Represented as:

[0031]

[0032] Among them, c j This indicates the specific target in the actual label corresponding to index j. Indicates the target type is c j The predicted probability, λ cls For hyperparameters, For matching real labels y j The true bounding box b j With predictive labels Predicted bounding box The differences between them.

[0033] Furthermore, Represented as:

[0034]

[0035] in, Represents the true bounding box b jWith predicted bounding box The generalized intersection-union cost function between them, ‖·‖1 denotes the 1-norm, λ giou and This is a hyperparameter.

[0036] Furthermore, for the prediction set The overall loss function is expressed as:

[0037]

[0038] This invention provides a multi-target tracking method with reference point prior position embedding and adaptive update. Building upon the traditional attention-based multi-target tracking network architecture, a multi-target tracking network is constructed for multi-target tracking. Within this network, a prior position generation network based on target queries is designed. This network generates target queries with explicit prior position information, which facilitates effective keypoint sampling and addresses the problem of insufficient prior position information for target query reference points, thereby improving target detection performance. Furthermore, a tracking reference point prediction network based on tracking query reference points is designed. This network generates new tracking query reference points for tracking queries, effectively reducing the deviation between the tracking reference point position and the target center position, achieving accurate positioning of the tracking reference point, and further improving target tracking performance. Attached Figure Description

[0039] Figure 1 This is a flowchart of the multi-target tracking network provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the target detection principle based on deformation attention provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the prior location generation network provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the tracking reference point prediction network provided in an embodiment of the present invention. Detailed Implementation

[0043] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0044] This invention provides a multi-target tracking method with reference point prior position embedding and adaptive update, such as... Figure 1 As shown, the steps include:

[0045] Construct a multi-target tracking network;

[0046] The constructed multi-target tracking network is trained.

[0047] The trained multi-object tracking network can be used in real-world multi-object tracking scenarios.

[0048] like Figure 1 As shown, the multi-target tracking network constructed in this embodiment is based on the traditional deformation attention-based target detection network framework. Building upon this framework, this application designs two novel network structures: a priori position generation network (PPG) and a tracking reference point prediction network (TRP). Overall, the multi-target tracking network constructed in this embodiment is as follows: Figure 1 As shown, the system includes a backbone network (CNN, Convolutional Neural Network), a Deformable Transformer (including an encoder and a decoder), a Prior Location Generation Network (PPG), a Tracking Reference Point Prediction Network (TRP), and a prediction head. The backbone network, encoder, and decoder are sequentially connected. The backbone network and encoder perform feature extraction and feature encoding aggregation on the input image, respectively, to obtain aggregated features that are output to the decoder. Both the Prior Location Generation Network and the Tracking Reference Point Prediction Network are connected to the decoder. The Prior Location Generation Network generates a target query with prior location information for each frame's initial target query and outputs it to the decoder. The Tracking Reference Point Prediction Network generates the tracking reference points for the current frame based on the tracking query output from the previous frame and outputs them to the decoder.

[0049] In the initial frame, the decoder generates an output embedding into the prediction head based on the target query with prior location information and the aggregated features of the initial frame; the prediction head generates the bounding box and category of the corresponding target based on the output embedding of the initial frame, and initializes it as the tracking query for the next frame;

[0050] In the current frame, which is not the initial frame, the decoder generates the output of the current frame and embeds it into the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates the bounding box and category of the corresponding target based on the output embedding of the current frame and initializes it as the tracking query of the next frame.

[0051] like Figure 1 As shown. First, at the initial frame t=0, the backbone network and the Encoder layer of the DeformableTransformer network are used to perform feature extraction and feature encoding aggregation on the initial frame, respectively, to obtain the output features of the initial frame. Secondly, the object query of the initial frame initialized by the model is input into the prior position generation network (PPG) to generate object queries for the initial frame with prior information. and The initial frame's output embedding is obtained by inputting it into the Decoder layer of the Deformable Transformer network. Then, embed the output of the initial frame. The bounding box and category of the corresponding target are obtained from the input prediction header and initialized as the Track Query for the next frame. Finally, in the subsequent current frame, the track query for the current frame is performed. And target query of the current frame with prior information Perform the merging, and then perform a track query on the current frame. The tracking reference point (TRP) network is used to predict the tracking reference point for the current frame. And the tracking reference point Output features of the current frame and the merged Together with the input into the decoder, new targets are detected and existing targets are tracked. By repeating the above process, a complete multi-target tracking workflow is achieved.

[0052] Deformation-attention-based target detection uses keypoint sampling around a reference point to achieve target localization and recognition, such as... Figure 2 As shown in the diagram, the keypoint sampling position is determined by the positional offset (Δp = (Δx, Δy)) relative to the reference point (p = (x, y)), which is obtained by linear mapping from the object query. Therefore, assigning a clear positional prior to the object query is beneficial for keypoint sampling, thereby achieving efficient object detection.

[0053] In view of this, embodiments of the present invention propose a priori position generation network based on reference points, the principle of which is as follows: Figure 3 As shown, the target query Q det It can be represented as:

[0054] Q det =C det +P det (1)

[0055] In the formula, C det and P detThese represent Content Query and Position Query, respectively. To assign the target query Q... det With explicit prior location information, this invention utilizes the location information of reference points to generate a location query P. det Specifically, let all reference points of the Object Query be denoted as... Then the m-th reference point is (The superscript T indicates transpose), x m and y m Let P represent the x and y coordinates of the m-th target reference point in the 2D plane. Then, based on the prior position of the reference point, P is generated. det It can be represented as:

[0056]

[0057] Among them, g sin (·) indicates a position encoding operation; This represents a location embedding operation used for alignment in the feature map's location space. Let... but for:

[0058]

[0059] Where w1 and W2 are the weights of the first and second linear mappings, respectively, b1 and b2 are the bias terms of the first and second linear mappings, respectively, and ReLU(·) represents the ReLU activation function.

[0060] The multi-head deformation attention mechanism of the Deformable Transformer network is based on all reference points R of the ObjectQuery (target query). det The input frame image's feature image F and the feature image with prior location information (P) det ) Target query Q det This yields the target detection key points of the input frame image.

[0061] By embedding the aforementioned reference point location information, the target detection based on deformation attention searches for targets around the reference points, which has clear location constraints, such as... Figure 3 As shown, since the object query has explicit reference point prior location information, the offset of keypoint sampling can be explicitly represented as the positional offset relative to the reference point. This positional prior helps in effective keypoint sampling, thereby improving the object detection performance based on deformation attention.

[0062] Deformation-attention-based target tracking associates the current target with historical targets by sampling keypoints around a tracking reference point. However, since this tracking reference point is set to the center of the bounding box of a target tracked in the previous frame, a positioning error occurs when there are positional differences between adjacent frames, especially in high-speed target movement and low frame rate scenarios. Therefore, this invention proposes a Tracking Reference Point Prediction Network (TRP) based on Track Query to achieve accurate positioning of the tracking reference point, thereby improving target tracking performance.

[0063] The Tracking Reference Point Prediction Network (TRP) achieves adaptive adjustment of the tracking reference point by predicting its offset. For example... Figure 4 The flowchart of the tracking reference point prediction network is shown below. Let all tracking reference points in the current frame be denoted as... in Let x represent the nth tracking reference point. n and y n Let x and y represent the x and y coordinates of the nth tracking reference point in the 2D plane. Then, the reference point adjusted by the Tracking Reference Point Prediction Network (TRP) is... It can be represented as:

[0064]

[0065] Where, ΔR tck The tracking reference point offset predicted based on the Track Query can be specifically represented as:

[0066] ΔR tck =ψ θ (Q tck (5)

[0067] Where, ψ θ (·) represents the reference point prediction model, where θ is the parameter of the model; Q tck This represents a track query. This invention designs a tracking reference point prediction model based on a multilayer perceptron, where the tracking reference point offset ΔR... tck It can be represented as:

[0068]

[0069] in, The weights and biases, i.e., parameters θ, represent the first, second, and third layers of the linear mapping in the tracking reference point prediction model.

[0070] By adaptively adjusting the tracking reference point as described above, the deviation between the tracking reference point position and the target center position can be effectively reduced. Therefore, key point sampling based on the adjusted tracking reference point is beneficial for sampling effective key features, thereby improving target tracking performance.

[0071] During training, this invention uses two adjacent frames of video data as model input to optimize the parameters of the multi-target tracking network.

[0072] The prediction sets of adjacent t-1 frames and t frame after passing through the multi-target tracking network can be represented as follows: Where N = N tck +N det N represents the number of targets being tracked. tck With the number of targets N det The total number of targets. Therefore, the optimization objective for a multi-target tracking network is: to sequentially optimize the prediction set... Prediction results With real tag set The label y in j For accurate association, K represents the total number of real targets in the current frame. The predicted total number of targets N needs to be matched with K ground truth labels to determine whether the predicted target appears in the current frame, and this is used for data association. However, in real-world scenarios, the model may not be able to successfully detect and associate all targets (K) 100% of the time; therefore, N ≤ K.

[0073] Then, for the prediction set The optimal association result with the true label set y It can be represented as:

[0074]

[0075] in, Let σ(j) represent the matching cost function, where σ(j) is the matching index of the predicted label. The current frame contains a prediction set [1, N] and a ground truth set [1, K]. In the training dataset, an image contains multiple types of targets, such as vehicles and pedestrians, each with a ground truth label. For each ground truth label, there are multiple prediction results. To obtain the optimal matching result—that is, the one with the highest similarity to the ground truth label—a unique match needs to be selected from the multiple prediction results. This unique match corresponds to a value in the prediction set, represented using an array index. The range of the matching index is the size of the prediction set.

[0076] Due to predicted labels The optimal matching combination (calculated from the prediction set and the true labels) includes the target type and bounding box, hence the matching cost function. It can be represented as:

[0077]

[0078] Among them, c j This indicates the specific target among the real labels corresponding to index j, where j corresponds to the total number of real targets. Indicates the target type is c j The predicted probability, λ cls For hyperparameters; Used to measure the true label y that matches. j The true bounding box b j With predictive labels Predicted bounding box The differences between them can be specifically expressed as follows:

[0079]

[0080] in, Represents the true bounding box b j With predicted bounding box The generalized intersection-union cost function between them, ‖·‖1 denotes the 1-norm, λ giou and This is a hyperparameter.

[0081] For the prediction set The overall loss function can be expressed as:

[0082]

[0083] By minimizing the overall loss function The optimization of network parameters for multi-target tracking is achieved, primarily including the core parameters weights W1 to W5 and biases b1 to b5. The hyperparameters are pre-set by default and do not change during model training.

[0084] In summary, the multi-target tracking method with reference point prior position embedding and adaptive update provided by this invention constructs a multi-target tracking network for multi-target tracking based on the traditional attention-based multi-target tracking network architecture. This multi-target tracking network includes a prior position generation network for reference points based on target queries. This prior position generation network generates target queries with explicit prior position information. This prior position information facilitates effective keypoint sampling and solves the problem of lack of prior position information for reference points in target queries, thereby improving target detection performance. Furthermore, a tracking reference point prediction network based on tracking query reference points is designed. This tracking reference point prediction network generates new tracking query reference points for tracking queries, effectively reducing the deviation between the tracking reference point position and the target center position, achieving accurate positioning of the tracking reference point, and thus improving target tracking performance.

[0085] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-target tracking method with reference point prior position embedding and adaptive update, characterized in that, include: Construct a multi-target tracking network; The multi-target tracking network includes a backbone network, a deformation attention network, a priori position generation network, a tracking reference point prediction network, and a prediction head. The deformation attention network includes an encoder and a decoder. The backbone network, the encoder, and the decoder are sequentially connected. The backbone network and the encoder are used to perform feature extraction and feature encoding aggregation on the input image, respectively, to obtain aggregated features which are output to the decoder. The priori position generation network and the tracking reference point prediction network are both connected to the decoder. The priori position generation network is used to generate a target query with priori position information for the initial target query of each frame and output it to the decoder. The tracking reference point prediction network generates the tracking reference point for the current frame from the tracking query output in the previous frame and outputs it to the decoder. In the initial frame, the decoder generates an output that is embedded into the prediction head based on the target query with prior location information and the aggregated features of the initial frame; The prediction head generates the bounding box and category of the corresponding target based on the output of the initial frame, and initializes it for the tracking query of the next frame. In the current frame, which is not the initial frame, the decoder generates the output of the current frame and embeds it into the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates the bounding box and category of the corresponding target based on the output embedding of the current frame and initializes it as the tracking query of the next frame. The target query generated by the prior location generation network Including content search and location query Location search Represented as: ， in, This represents all reference points for the target query. This indicates a position encoding operation. This indicates a position embedding operation; make ,but Specifically: ， in, and These are the weights of the first-level linear mapping and the second-level linear mapping, respectively. and These are the bias terms for the first-level linear mapping and the second-level linear mapping, respectively. Represents the ReLU activation function; The tracking reference point prediction network predicts all tracking reference points in the current frame. Adjustments are made to obtain the adjusted reference point. : ， in, For tracking queries The predicted tracking reference point offset is specifically expressed as follows: ， in, This indicates a prediction model that tracks reference points. These are the parameters of the model. This represents the weights and biases of the first, second, and third layers of the linear mapping in the reference point prediction model, i.e., the parameters. .

2. The multi-target tracking method with reference point prior position embedding and adaptive update according to claim 1, characterized in that, The method also includes: The constructed multi-target tracking network is trained; Adjacent Frames and The prediction set of a frame after passing through a multi-object tracking network is represented as follows: ,in Indicates the number of tracked targets With the number of detection targets The total number of targets; during training, the optimization objective for the multi-target tracking network is to sequentially optimize the prediction set. Prediction results With real tag set Tags in To make accurate associations, This represents the total number of real targets in the current frame.

3. The multi-target tracking method with reference point prior position embedding and adaptive update according to claim 2, characterized in that, For the prediction set and real tag set Optimal association results Represented as: ， in, Indicates the first A real label With the Predicted labels The matching cost function between them For predicting labels The matching index.

4. The multi-target tracking method with reference point prior position embedding and adaptive update according to claim 3, characterized in that, Matching cost function Represented as: ， in, This indicates that The specific target in the actual label corresponding to the index. Indicates the target type is The predicted probability, For hyperparameters, For matching real labels True bounding box With predictive labels Predicted bounding box The differences between them.

5. The multi-target tracking method with reference point prior position embedding and adaptive update according to claim 4, characterized in that, Represented as: ， in, Represents the true bounding box With predicted bounding box The generalized intersection-union cost function between them Denotes the 1-norm. and This is a hyperparameter.

6. The multi-target tracking method with reference point prior position embedding and adaptive update according to claim 5, characterized in that, For the prediction set The overall loss function is expressed as: 。

Images