Multi-target tracking method for embedding and adaptively updating prior position of reference point

Abstract

The invention relates to the technical field of machine vision, and particularly discloses a reference point prior position embedding and self-adaptive updating multi-target tracking method, which constructs a multi-target tracking network for multi-target tracking, and designs a prior position generation network of a reference point based on target query in the multi-target tracking network. The priori position generation network generates a target query with clear priori position information, and the position priori information is beneficial to effective key point sampling, so that the problem that a reference point of the target query lacks priori position information can be solved, and the target detection performance is further improved; in addition, a tracking reference point prediction network based on reference points of tracking query is also designed, the tracking reference point prediction network generates new tracking query reference points for tracking query, the deviation between the tracking reference point position and the target center position can be effectively reduced, accurate positioning of the tracking reference points is achieved, and then the target tracking performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a multi-target tracking method for embedding and adaptively updating reference point priori positions. Background Art

[0002] Video multi-object tracking (MOT) is a very challenging problem in the field of computer vision. MOT technology continuously tracks multiple target states in a video sequence to generate the target's motion trajectory. This technology has important application prospects in video surveillance, autonomous driving, human-computer interaction and other fields. Due to occlusion, lighting changes, motion blur and background interference in complex scenes, achieving effective multi-target tracking remains a challenging problem.

[0003] At present, video multi-target tracking methods are mainly divided into tracking methods based on target detection and data association, integrated tracking methods based on joint target detection and data association, and tracking methods based on attention mechanism. (1) Tracking methods based on target detection and data association perform data association based on target detection results, which belongs to two-stage multi-target tracking. Therefore, the model tracking effect depends largely on the performance of the detector, and the tracking robustness in complex scenes such as frequent occlusion and appearance changes is still insufficient. (2) Integrated tracking methods based on joint target detection and data association combine target detection and data association to achieve integrated multi-target tracking; the main problem of this type of tracking method is that there is a confrontation between the target detection and data association subtasks. (3) Multi-target tracking methods based on attention mechanism are an end-to-end tracking paradigm, which introduces Object Query and Track Query to achieve integrated target detection and data association. However, the reference points of Object Query and Track Query in this type of tracking method lack position priors and are inaccurate, respectively. First, for Object Query, its reference point is randomly initialized and lacks clear location prior information, which makes it difficult to locate and optimize the model. Second, for Track Query, its reference point is set to the center of the bounding box of the tracked target in the previous frame; however, due to the movement of the target, especially in high-speed target movement and low frame rate scenarios, the center position of the same target between adjacent video frames is different, resulting in a positioning error between the reference point of Track Query and the actual center position of the target to be associated, which in turn affects the data association performance.

[0004] The multi-target tracking method based on the attention mechanism is the mainstream method in the current field of multi-target tracking in videos. However, how to solve the problem of lack of position prior and inaccurate reference points in this type of tracking method has become a key issue that needs to be solved urgently in current research. Summary of the invention

[0005] The present invention provides a multi-target tracking method with reference point prior position embedding and adaptive updating, and solves the technical problem that the reference points of target queries in traditional multi-target tracking methods based on attention mechanism lack position prior information, and the reference points of tracking queries are inaccurate, which affects the multi-target tracking performance.

[0006] In order to solve the above technical problems, the present invention provides a multi-target tracking method with reference point prior position embedding and adaptive updating, which comprises: constructing a multi-target tracking network;

[0007] The multi-target tracking network includes a backbone network, a deformation attention network, a priori position generation network, a tracking reference point prediction network and a prediction head, and the deformation attention network includes an encoder and a decoder; the backbone network, the encoder and the decoder are connected in sequence, and the backbone network and the encoder are used to perform feature extraction and feature encoding aggregation on the input image respectively, and the aggregated features are output to the decoder; the priori position generation network and the tracking reference point prediction network are both connected to the decoder; the priori position generation network is used to generate a target query with priori position information for the initialized target query of each frame and output it to the decoder; the tracking reference point prediction network generates the tracking reference point of the current frame for the tracking query outputted by the previous frame and outputs it to the decoder;

[0008] At the initial frame, the decoder generates an output embedding to the prediction head according to the target query with prior position information and the aggregated features of the initial frame; the prediction head generates a bounding box and a category of the corresponding target according to the output embedding of the initial frame, and initializes it as a tracking query for the next frame;

[0009] When the current frame is not the initial frame, the decoder generates an output embedding of the current frame to the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates a bounding box and category of the corresponding target based on the output embedding of the current frame, and initializes it as a tracking query for the next frame.

[0010] Furthermore, the target query q generated by the prior position generation network det Include content queryc det and location query p det , location query p det It is expressed as:

[0011]

[0012] Among them, R det represents all reference points of the target query, g sin (·) represents the position encoding operation, Represents a positional embedding operation.

[0013] Further, let Then P det Specifically:

[0014]

[0015] Where W1 and W2 are the weights of the first-layer linear mapping and the second-layer linear mapping, respectively; b1 and b2 are the bias terms of the first-layer linear mapping and the second-layer linear mapping, respectively; and ReLU(·) represents the ReLU activation function.

[0016] Furthermore, the tracking reference point prediction network predicts all tracking reference points r of the current frame. tck Make adjustments and get the adjusted reference point

[0017]

[0018] Among them, ΔR tck Based on the tracking query Q tck The predicted tracking reference point offset is expressed as:

[0019] ΔR tck =ψ θ (Q tck )

[0020] Among them, ψ θ (·) represents the tracking reference point prediction model, and θ is the parameter of the model.

[0021] Furthermore, ΔR tck Specifically expressed as:

[0022] ΔR tck =ψ θ (Q tck )=(W5(ReLU(W4(ReLU(W3Q tck +b3))+b4))+b5)

[0023] in, Represents the weights and bias terms of the first, second, and third linear mappings of the tracking reference point prediction model, i.e., parameter θ.

[0024] Furthermore, the method further comprises:

[0025] Training the constructed multi-target tracking network;

[0026] The prediction set of adjacent t-1 frames and t frames after the multi-target tracking network is expressed as Where N = N tck +N det Indicates the number of tracked targets N tck and the number of detected targets N det The total number of; During the training process, the optimization goal of the multi-target tracking network is: one by one to predict the set The prediction results in With the true label set The label y in j Perform accurate association, K is the total number of real targets in the current frame.

[0027] Furthermore, for the prediction set The optimal correlation result with the true label set y It is expressed as:

[0028]

[0029] in, Represents the jth true label y j and the σ(j)th predicted label The matching cost function between them, σ(j) is the predicted label The matching index of .

[0030] Furthermore, the matching cost function It is expressed as:

[0031]

[0032] Among them, c j Indicates the specific target in the real label corresponding to the j index, Indicates that the target type is c j The predicted probability, λ cls is a hyperparameter, is the matching true label y j The ground-truth bounding box b j With predicted labels The predicted bounding box The difference between.

[0033] Further, It is expressed as:

[0034]

[0035] in, represents the ground-truth bounding box b jWith the predicted bounding box The generalized intersection-over-union cost function between , ‖·‖1 represents the 1-norm, λ giou and is a hyperparameter.

[0036] Furthermore, for the prediction set The overall loss function is expressed as:

[0037]

[0038] The present invention provides a multi-target tracking method with reference point prior position embedding and adaptive updating. On the basis of the traditional multi-target tracking network architecture based on the attention mechanism, a multi-target tracking network is constructed for multi-target tracking. In the multi-target tracking network, a priori position generation network based on the reference point of the target query is designed. The prior position generation network generates the target query with clear prior position information. The position prior information is helpful for effective key point sampling, which can solve the problem of lack of prior position information of the reference point of the target query, thereby improving the target detection performance. A tracking reference point prediction network based on the reference point of the tracking query is also designed. The tracking reference point prediction network generates a new tracking query reference point for the tracking query, which can effectively reduce the deviation between the tracking reference point position and the target center position, realize the accurate positioning of the tracking reference point, thereby improving the target tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a processing flow chart of a multi-target tracking network provided by an embodiment of the present invention;

[0040] Figure 2 is a principle diagram of target detection based on deformation attention provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a priori position generation network provided by an embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of a tracking reference point prediction network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0044] The embodiment of the present invention provides a multi-target tracking method for embedding and adaptively updating reference point prior positions, such as Figure 1 As shown, the steps include:

[0045] Construct a multi-target tracking network;

[0046] The constructed multi-target tracking network is trained.

[0047] The trained multi-target tracking network can be used in practical multi-target tracking scenarios.

[0048] like Figure 1 As shown in the figure, the multi-target tracking network constructed in this embodiment is based on the traditional target detection network framework based on deformation attention. On this basis, this application designs two new network structures, one is the prior position generation network (PPG), and the other is the tracking reference point prediction network (TRP). Overall, the multi-target tracking network constructed in this embodiment is as follows Figure 1 As shown, it includes a backbone network (CNN, convolutional neural network), a deformable attention network (Deformable Transformer, including an encoder and a decoder), a priori position generation network (PPG), a tracking reference point prediction network (TRP) and a prediction head. Among them, the backbone network, encoder (Encoder) and decoder (Decoder) are connected in sequence, and the backbone network and encoder are used to perform feature extraction and feature encoding aggregation on the input image respectively, and obtain aggregated features and output them to the decoder. The priori position generation network and the tracking reference point prediction network are both connected to the decoder. The priori position generation network is used to generate a target query with priori position information for the initialized target query of each frame and output it to the decoder. The tracking reference point prediction network generates the tracking reference point of the current frame for the tracking query output of the previous frame and outputs it to the decoder.

[0049] At the initial frame, the decoder generates an output embedding to the prediction head according to the target query with prior position information and the aggregated features of the initial frame; the prediction head generates a bounding box and a category of the corresponding target according to the output embedding of the initial frame, and initializes it as a tracking query for the next frame;

[0050] When the current frame is not the initial frame, the decoder generates an output embedding of the current frame to the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates a bounding box and category of the corresponding target based on the output embedding of the current frame, and initializes it as a tracking query for the next frame.

[0051] like Figure 1 First, at the initial frame t = 0, the backbone network and the encoder layer of the deformable attention network (DeformableTransformer) are used to perform feature extraction and feature encoding aggregation on the initial frame to obtain the output feature of the initial frame Secondly, the object query of the initial frame initialized by the model is input into the prior position generation network (PPG) to generate the object query of the initial frame with prior information. and with Input together into the Decoder layer of the Deformable Transformer to get the output embedding of the initial frame The output of this initial frame is then embedded into The bounding box and category of the corresponding target are obtained in the input prediction head and initialized as the tracking query (Track Query) of the next frame Finally, in the subsequent current frame, the Track Query of the current frame is and the target query of the current frame with prior information Merge, then set the Track Query of the current frame Input the tracking reference point prediction network (TRP) to predict the tracking reference point of the current frame And the tracking reference point The output features of the current frame and the merged The above process is repeated to realize the complete multi-target tracking process.

[0052] Deformation-attention-based object detection achieves object positioning and recognition by sampling key points around the reference point, such as Figure 2 As shown in the figure, the key point sampling position is determined by the position offset (△p=(△x,△y)) compared to the reference point (p=(x,y)), and the position offset is obtained by linear mapping of the object query. Therefore, giving a clear position prior to the object query is beneficial to key point sampling, thereby achieving efficient object detection.

[0053] In view of this, the embodiment of the present invention proposes to generate a network based on the prior position of the reference point, and its principle is as follows: Figure 3 As shown, the target query Q det It can be expressed as:

[0054] Q det =C det +P det (1)

[0055] In the formula, C det and P detThey represent Content Query and Position Query respectively. In order to assign the target query Q det Clear location prior information, the present invention uses the location information of the reference point to generate the location query P det Specifically, all reference points of the Object Query are Then the mth reference point is (superscript T indicates transpose), x m and m represents the horizontal and vertical coordinates of the mth target reference point in the 2D plane. Then, P is generated based on the position of the reference point a priori. det It can be expressed as:

[0056]

[0057] Among them, g sin (·) indicates position encoding operation; represents the position embedding operation, which is used to align the feature map position space. but for:

[0058]

[0059] Among them, w1 and W2 are the weights of the first layer linear mapping and the second layer linear mapping, b1 and b2 are the bias terms of the first layer linear mapping and the second layer linear mapping, respectively, and ReLU(·) represents the ReLU activation function.

[0060] The multi-head deformation attention mechanism of the Deformable Transformer is based on all reference points R of the ObjectQuery (target query). det , the feature image F of the input frame image and the prior position information (P det ) Target query Q det , get the target detection key points of the input frame image.

[0061] By embedding the above reference point position information, the target detection constraint based on deformation attention searches for the target around the reference point with clear position constraints, such as Figure 3 As shown in Figure 2. Since the object query has clear reference point prior position information, the offset of key point sampling can be explicitly expressed as the position offset compared to the reference point. This position prior helps to effectively sample key points and thus improve the performance of object detection based on deformation attention.

[0062] Target tracking based on deformation attention achieves the association between the current target and the historical target by sampling key points around the tracking reference point. However, since the tracking reference point is set to the center position of the bounding box of the tracked target in the previous frame, when there is a position difference between the targets in adjacent frames, especially in the high-speed motion and low frame rate scenes of the target, there is a positioning error between the reference point and the center position of the current target. Therefore, an embodiment of the present invention proposes a tracking reference point prediction network (TRP) based on tracking query (Track Query) to achieve accurate positioning of the tracking reference point, thereby improving the target tracking performance.

[0063] The Tracking Reference Point Prediction Network (TRP) achieves adaptive adjustment of the tracking reference point by predicting the offset of the reference point. Figure 4 As shown in the flowchart of the tracking reference point prediction network, all the tracking reference points of the current frame are recorded as in Indicates the nth tracking reference point, x n and n represents the horizontal and vertical coordinates of the nth tracking reference point in the 2D plane. Then, the reference point adjusted by the tracking reference point prediction network (TRP) It can be expressed as:

[0064]

[0065] Among them, ΔR tck is the tracking reference point offset predicted based on the tracking query (Track Query), which can be specifically expressed as:

[0066] ΔR tck =ψ θ (Q tck ) (5)

[0067] Among them, ψ θ (·) represents the tracking reference point prediction model, θ is the parameter of the model; Q tck The present invention designs a tracking reference point prediction model based on a multi-layer perceptron. The tracking reference point offset ΔR tck It can be expressed as:

[0068]

[0069] in, Represents the weights and bias terms of the first, second, and third linear mappings of the tracking reference point prediction model, i.e., parameter θ.

[0070] Through the adaptive adjustment of the tracking reference point, the deviation between the tracking reference point and the target center position can be effectively reduced. Therefore, the key point sampling based on the adjusted tracking reference point is beneficial to sampling effective key features, thereby improving the target tracking performance.

[0071] During the training process, the present invention uses two adjacent frames of video data as model input to optimize the parameters of the multi-target tracking network.

[0072] The prediction set of adjacent t-1 frames and t frames after the multi-target tracking network can be expressed as Where N = N tck +N det Indicates the number of tracked targets N tck and the number of detected targets N det The total number of. Then the optimization goal of the multi-target tracking network is: one by one to predict the set The prediction results in With the true label set The label y in j Accurately associate, K is the total number of real targets in the current frame. The total number of predicted targets N needs to be matched with K real labels (Ground Truth) to determine whether the predicted target appears in the current frame, and then perform data association. In real scenes, the model may not be able to successfully detect and associate all targets (K) 100%, so N≤K.

[0073] Then, for the prediction set The optimal correlation result with the true label set y It can be expressed as:

[0074]

[0075] in, Represents the matching cost function, σ(j) is the matching index of the predicted label. The current frame has a prediction set [1, N] and a real label set [1, K]. In the training data set, a picture contains multiple types of targets, such as vehicles, pedestrians, etc., and each type of target has a real annotation, which is the label (ground truth). For the real label, there are multiple prediction results. In order to obtain the best matching result, that is, the similarity between the predicted result and the real label is the highest, it is necessary to select the only match from multiple prediction results, and the prediction result corresponds to the value in the prediction set, that is, it is represented by an array index. The range of the matching index is the size of the prediction set.

[0076] Since the predicted label The optimal matching combination (calculated from the predicted set and the true label) includes the type and bounding box of the target, so the matching cost function It can be expressed as:

[0077]

[0078] Among them, c j Indicates the specific target in the real label corresponding to the j index, j corresponds to the total number of real targets, Indicates that the target type is c j The predicted probability, λ cls is a hyperparameter; The true label y that is matched to the metric j The ground-truth bounding box b j With predicted labels The predicted bounding box The difference between them can be specifically expressed as:

[0079]

[0080] in, represents the ground-truth bounding box b j With the predicted bounding box The generalized intersection-over-union cost function between , ‖·‖1 represents the 1-norm, λ giou and is a hyperparameter.

[0081] For the prediction set The overall loss function can be expressed as:

[0082]

[0083] By minimizing the overall loss function The optimized network parameters of multi-target tracking are optimized, including core parameter weights W1 to W5 and biases b1 to b5. Hyperparameters are pre-set parameters and will not change due to model training.

[0084] In summary, an embodiment of the present invention provides a multi-target tracking method with reference point prior position embedding and adaptive updating. On the basis of the traditional multi-target tracking network architecture based on the attention mechanism, a multi-target tracking network is constructed for multi-target tracking. In the multi-target tracking network, a priori position generation network based on the reference point of the target query is designed. The prior position generation network generates a target query with clear prior position information. The position prior information helps to effectively sample key points, which can solve the problem of lack of prior position information of the reference point of the target query, thereby improving the target detection performance. A tracking reference point prediction network based on the reference point of the tracking query is also designed. The tracking reference point prediction network generates a new tracking query reference point for the tracking query, which can effectively reduce the deviation between the tracking reference point position and the target center position, realize accurate positioning of the tracking reference point, thereby improving the target tracking performance.

[0085] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-target tracking method with reference point prior position embedding and adaptive updating, characterized in that: include: Construct a multi-target tracking network; The multi-target tracking network includes a backbone network, a deformation attention network, a priori position generation network, a tracking reference point prediction network and a prediction head, and the deformation attention network includes an encoder and a decoder; the backbone network, the encoder and the decoder are connected in sequence, and the backbone network and the encoder are used to perform feature extraction and feature encoding aggregation on the input image respectively, and the aggregated features are output to the decoder; the priori position generation network and the tracking reference point prediction network are both connected to the decoder; the priori position generation network is used to generate a target query with priori position information for the initialized target query of each frame and output it to the decoder; the tracking reference point prediction network generates the tracking reference point of the current frame for the tracking query outputted by the previous frame and outputs it to the decoder; At the initial frame, the decoder generates output embedding into the prediction head according to the target query with prior position information and the aggregated features of the initial frame; The prediction head generates a bounding box and category of the corresponding target based on the output embedding of the initial frame, and initializes it as a tracking query for the next frame; When the current frame is not the initial frame, the decoder generates an output embedding of the current frame to the prediction head based on the tracking reference point of the current frame, the target query of the previous frame, and the target query with prior position information of the current frame. The prediction head generates a bounding box and category of the corresponding target based on the output embedding of the current frame, and initializes it as a tracking query for the next frame.

2. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 1, characterized in that: The prior position generates the target query Q generated by the network det Include content query C det and location query P det , location query P det It is expressed as: Among them, R det represents all reference points of the target query, g sin (·) represents the position encoding operation, Represents a positional embedding operation.

3. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 2, characterized in that: make Then P det Specifically: Where W1 and W2 are the weights of the first-layer linear mapping and the second-layer linear mapping, respectively; b1 and b2 are the bias terms of the first-layer linear mapping and the second-layer linear mapping, respectively; and ReLU(·) represents the ReLU activation function.

4. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 3, characterized in that: The tracking reference point prediction network predicts all tracking reference points R in the current frame. tck Make adjustments and get the adjusted reference point Among them, ΔR tck Based on the tracking query Q tck The predicted tracking reference point offset is expressed as: ΔR tck =ψ θ (Q tck ) Among them, ψ θ (·) represents the tracking reference point prediction model, and θ is the parameter of the model.

5. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 4, characterized in that: ΔR tck Specifically expressed as: ΔR tck =ψ θ (Q tck )=(W5(ReLU(W4(ReLU(W3Q) tck +b3))+b4))+B5) in, Represents the weights and bias terms of the first, second, and third linear mappings of the tracking reference point prediction model, i.e., parameter θ.

6. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 5, characterized in that: The method further includes: Training the constructed multi-target tracking network; The prediction set of adjacent t-1 frames and t frames after the multi-target tracking network is expressed as Where N = N tck +N det Indicates the number of tracked targets N tck and the number of detected targets N det The total number of; During the training process, the optimization goal of the multi-target tracking network is: one by one to predict the set The prediction results in With the true label set The label yj in is accurately associated, and K is the total number of real targets in the current frame.

7. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 6, characterized in that: For the prediction set The optimal correlation result with the true label set y It is expressed as: in, Represents the jth true label yj and the σ(j)th predicted label The matching cost function between them, σ(j) is the predicted label The matching index of .

8. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 7, characterized in that: Matching cost function It is expressed as: Among them, c j Indicates the specific target in the real label corresponding to the j index, Indicates that the target type is c j The predicted probability, λ cls is a hyperparameter, is the matching true label y j The ground-truth bounding box b j With predicted labels The predicted bounding box The difference between.

9. The method for multi-target tracking with embedding and adaptive updating of reference point prior positions according to claim 8, characterized in that: It is expressed as: in, represents the ground-truth bounding box b j With the predicted bounding box The generalized intersection-over-union cost function between , ‖·‖1 represents the 1-norm, λ giou and is a hyperparameter.

10. The multi-target tracking method of embedding and adaptively updating reference point prior positions according to claim 9, characterized in that: For the prediction set The overall loss function is expressed as:

Images