Three-dimensional multi-target tracking method, system and medium based on feature-enhanced Transformer

Abstract

The present invention discloses a three-dimensional multi-target tracking method, system, and medium based on a feature-enhanced Transformer. The method comprises the following steps: acquiring three-dimensional point cloud data collected by a laser radar, generating three-dimensional target detection results for the current frame using a target detector; constructing a detection map, a trajectory map, and a trajectory-detection association map based on historical trajectories and the current frame detection results, and extracting initial features; dynamically fusing and mining the three types of initial features through a feature-enhanced Transformer network to obtain trajectory features, detection features, and association features after attention-weighted aggregation; calculating the similarity between trajectory-detection results using association map features and performing data association; estimating velocity using detection map features, and estimating the trajectory position in the next frame based on this velocity; and managing and updating trajectories using trajectory map features and detection map features.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and computer vision, and in particular to a three-dimensional multi-target tracking method, system and medium based on feature-enhanced Transformer. Background Art

[0002] Thanks to societal progress and technological advancements, transportation is gradually evolving towards intelligent and even autonomous driving. Mainstream autonomous driving systems consist of three key components: environmental perception, decision-making and planning, and execution control, all arranged in a cascaded architecture. The environmental perception module utilizes sensors such as cameras, lidar, and millimeter-wave radar to dynamically detect and identify the vehicle's surroundings. The decision-making module determines the optimal motion path for the autonomous vehicle based on the perceived results, while the execution module controls the vehicle's trajectory. The environmental perception module is the first step in autonomous driving and a crucial component of the system. Three-dimensional multi-target tracking, a fundamental task in environmental perception, requires locating objects in a data stream, estimating their size and orientation, and assigning unique identification tags to each object.

[0003] The mainstream framework for multi-target tracking is detection-based tracking, which can be roughly divided into four parts: target detection, feature extraction, data association, and trajectory management. The most important component is data association, which involves matching and associating historical trajectories with the detection results of the current frame. Existing 3D multi-target tracking algorithms mostly use non-learning algorithms in the data association module. For example, they use the constant velocity hypothesis model and the Kalman filter algorithm to estimate the position of historical trajectories in the current frame, and then use geometric correlation metrics such as the intersection of union (IoU) and center-to-center distance (CDM) of 3D bounding boxes to associate and match historical trajectories with the detection results of the current frame. This heuristic approach requires extensive manual parameterization and is not very versatile. For example, the AB3DMOT algorithm proposed by Xinshuo Weng et al. combines the Kalman filter algorithm and the Hungarian matching algorithm for state estimation and data association. The Hungarian matching algorithm only uses the 3D IoU for similarity calculation and does not consider the deeper feature correlations between detections and trajectories. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the existing technology by providing a 3D multi-target tracking method based on a feature-enhanced Transformer. This method aims to introduce richer feature information by extracting appearance and motion features. Furthermore, the Transformer is used for deep feature mining, capturing the feature dependencies between trajectories and detections, thereby forming a more complete feature representation of the target. Furthermore, the use of the Transformer reduces the need for manual parameterization, improves the versatility of the network, and achieves better tracking results.

[0005] To achieve the purpose of the present invention, the present invention provides a three-dimensional multi-target tracking method based on feature enhancement Transformer, the main steps of which include:

[0006] S1. Obtain the 3D point cloud data collected by the lidar and use the target detector to generate the 3D target detection results of the current frame, including the category, position parameters and other information of the target bounding box;

[0007] S2. Construct a detection map, a trajectory map, and a trajectory-detection association map based on the historical trajectory and the current frame detection results;

[0008] S3, extracting initial features of the detection map, trajectory map and trajectory-detection association map;

[0009] S4. Dynamically fuse and mine the three types of initial features and the edge values ​​of the graph through a feature-enhanced Transformer network to obtain trajectory features, detection features, and correlation features after attention-weighted aggregation. The feature-enhanced Transformer network includes a multi-head self-attention mechanism, an enhanced multi-head cross-attention mechanism, and a feedforward network. The multi-head self-attention mechanism is used for the interaction between different trajectory features and trajectory features, the enhanced multi-head cross-attention module is used for the interaction between different trajectory features and detection features, and the introduction of correlation features enhances the correlation between detection features and trajectory features. The feedforward network is used to improve the nonlinear ability of the network.

[0010] S5. Calculate the similarity between the trajectory and the detection result using the correlation features and perform data association;

[0011] S6. Use the detected features to estimate the velocity, and estimate the position of the trajectory in the next frame based on the velocity;

[0012] S7, using trajectory features and detection features to manage and update trajectories;

[0013] S8. Define the loss function and use the dataset to train the proposed model.

[0014] Furthermore, in step S1, a CenterPoint object detector is used to obtain all object detection box information in the three-dimensional point cloud, including position parameters, confidence scores, and one-hot encoding classification.

[0015] Furthermore, the specific process in step S2 is as follows:

[0016] S21, build detection graph G D =(V D ,E D ): In the 3D point cloud, all detection results are marked as nodes V in the detection graph D, if two nodes belong to the same type of object and the distance between them is within the threshold T D If the two nodes are within , the two nodes are connected, and the weight of the edge is the distance value;

[0017] S22. Construct trajectory graph G T =(V T ,E T ): The construction process is the same as step S21, the node V in the trajectory graph T is the predicted location of the historical trajectory in the current frame. If two nodes belong to the same type of object and the distance between them is within the threshold T T If the two nodes are within , the two nodes are connected, and the weight of the edge is the distance value;

[0018] S23. Construct trajectory-detection association graph G A =(V D ,V T ,E A ): association graph G A It is a bipartite graph, with edges connecting detection nodes V D and trajectory node V T , where edges are only built on objects that belong to the same category and whose distance is within the threshold T T Detection node V within the range D and trajectory node V T between.

[0019] Furthermore, the specific process in step S3 is as follows:

[0020] S31. Extract initial features of the detection image: Initial features include appearance features and motion features. For appearance features, first extract the point cloud information enclosed by the point cloud detection box and use the PointNet algorithm to extract appearance features from these point clouds. For motion features, directly use the 3D point cloud detection box as input and use the Multi-Layer Perceptron (MLP) algorithm to extract motion features from the detection box. To balance the contributions of appearance and motion features, adjust them to the same size.

[0021] S32, extracting initial features of the trajectory graph: using the trajectory features stored in the historical trajectory library as the initial features of the trajectory graph of the current frame;

[0022] S33. Extracting initial features of the trajectory-detection association graph: Using the detection box information difference between the predicted position point of the historical trajectory in the current frame and the detection result of the current frame, including the detection box center point position difference, detection box size difference, and orientation angle difference, as the initial features of the trajectory-association graph.

[0023] Furthermore, the feature enhancement Transformer network in step S4 includes a self-attention module, a cross-attention module and a feedforward network; the self-attention module is used for interaction between different trajectory features, the cross-attention module is used for interaction between different trajectory features and detection features, and the feedforward network is used to improve the nonlinear ability of the network.

[0024] Furthermore, in step S5, the association features are input into the multi-layer perceptron (MLP) network to calculate the similarity score between the trajectory and the detection result. The higher the similarity score, the more likely it is that the trajectory and the detection result belong to the same target. Based on the similarity score, a greedy algorithm is used to perform association matching, thereby obtaining successfully associated trajectory-detection pairs, unsuccessfully associated trajectories, and unsuccessfully associated detection results.

[0025] Furthermore, in step S6, the detection features are input into a multi-layer perceptron (MLP) network to calculate the speed of the current frame detection result for subsequent position estimation.

[0026] Furthermore, in step S7, the historical trajectory library is managed and updated using trajectory features and detection features based on the successfully associated trajectory-detection pairs, unsuccessfully associated trajectories, and unsuccessfully associated detection results obtained by the data association module. For successfully associated trajectory-detection pairs, the position of the historical trajectory in the current frame is updated, and the trajectory features are updated using the detection result features. For unsuccessfully associated trajectories, if they have been unsuccessfully associated for more than κ frames, they are considered to have disappeared and deleted from the historical trajectory library. For unsuccessfully associated detection results, if they have appeared for more than κ frames, they are considered to be newly appeared targets, and a new trajectory is created for them and added to the trajectory library.

[0027] Furthermore, we define the loss function The loss function includes similarity loss and speed loss

[0028]

[0029] Among them, λ a ,λ v are the weights of each type of loss respectively; the similarity loss acts on the output of the similarity matrix. To address the imbalance problem between positive and negative samples, FocalLoss is used to calculate the similarity loss. The speed loss acts on the output of the target speed, and L1 loss is used to calculate the error between the parameters of the positive sample and the true value.

[0030] The present invention also provides a three-dimensional multi-target tracking system based on feature-enhanced Transformer, which is used to implement the above method and includes the following modules:

[0031] An object detection module, which is used to generate a three-dimensional object detection result of the current frame using an object detector, including information such as the category and position parameters of the object bounding box;

[0032] A graph construction module, which is used to dynamically construct a detection graph, a trajectory graph, and a trajectory-detection association graph based on historical trajectories and current frame detection results;

[0033] A feature extraction module, wherein the feature extraction module is used to extract initial features of the detection map, the trajectory map, and the trajectory-detection association map;

[0034] The feature-enhanced Transformer module is used to dynamically fuse and mine initial features to output richer feature information. It includes a multi-head self-attention mechanism, an enhanced multi-head cross-attention mechanism, and a feedforward network. The self-attention module is used for interaction between different trajectory features, the cross-attention module is used for interaction between different trajectory features and detection features, and the feedforward network is used to improve the nonlinear ability of the network.

[0035] The data association module calculates the similarity between the trajectory and the detection results based on the association features through the MLP multi-layer perceptron, and uses the greedy algorithm to perform data association, reducing the formulation of manual parameters;

[0036] A speed estimation module, which uses an MLP multi-layer perceptron to estimate the speed based on the detection features and predicts the position of the trajectory in the next frame based on the speed;

[0037] The trajectory management and update module uses trajectory features and detection features to manage and update the trajectory.

[0038] The present invention also provides a three-dimensional multi-target tracking device based on feature-enhanced Transformer, comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the aforementioned method.

[0042] The present invention also provides a storage medium storing a program executable by a processor. When the program is executed by the processor, it is used to perform the above method.

[0043] The present invention has the following advantages and beneficial effects compared to the prior art:

[0044] (1) The present invention discloses a 3D multi-target tracking method based on a feature-enhanced Transformer. This method takes into account the appearance and motion characteristics of the target and designs a feature extraction unit to enhance the feature expression of the target, thereby improving the accuracy and stability of 3D multi-target tracking.

[0045] (2) The present invention discloses a three-dimensional multi-target tracking method based on feature-enhanced Transformer, which uses the self-attention and cross-attention of Transformer to mine target correlation features between trajectories and detections, providing more accurate measurement data for subsequent similarity calculation and data association;

[0046] (3) The present invention discloses a 3D multi-target tracking method based on feature-enhanced Transformer. In the design of the Transformer fusion mechanism, an enhanced cross-attention mechanism is proposed to take into account the edge relationship of the trajectory-detection association graph and better capture the feature dependency between trajectories and detections.

[0047] (4) The present invention discloses a three-dimensional multi-target tracking method based on feature-enhanced Transformer, which uses Transformer for feature mining and speed estimation, reduces the number of manually formulated parameters, and improves the versatility of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a structural diagram of a three-dimensional multi-target tracking system based on feature-enhanced Transformer in an embodiment of the present invention;

[0049] Figure 2 This is a diagram of the initial feature extraction network structure in an embodiment of the present invention;

[0050] Figure 3 This is a diagram of the multi-head self-attention network structure in an embodiment of the present invention;

[0051] Figure 4 This is a diagram of the enhanced multi-head cross attention network structure in an embodiment of the present invention;

[0052] Figure 5 A diagram showing the structure of a feedforward network in an embodiment of the present invention;

[0053] Figure 6 This is a diagram of the feature mining network structure of the feature enhancement Transformer in an embodiment of the present invention;

[0054] Figure 7 This is a diagram of a data association network structure in an embodiment of the present invention;

[0055] Figure 8This is a diagram of a speed estimation network structure in an embodiment of the present invention;

[0056] Figure 9 This is a flowchart of three-dimensional multi-target tracking based on feature-enhanced Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] like Figure 9 As shown in Figure 1, a 3D multi-target tracking method based on feature-enhanced Transformer includes building and training a tracking model. The specific steps are as follows:

[0059] S1. Obtain the three-dimensional point cloud data collected by the lidar and use the CenterPoint target detector to generate the three-dimensional target detection result (x, y, z, l, w, h, θ, v, o, s) of the current frame, where (x, y, z) are the coordinates of the center point of the detection box, (l, w, h) are the length, width and height of the detection box, θ is the orientation angle, v is the speed, o is the one-hot encoding classification, and s is the confidence score of the detection result.

[0060] S2. Construct a detection map, a trajectory map, and a trajectory-detection association map based on the historical trajectory and the current frame detection results.

[0061] Step S2 includes the following sub-steps:

[0062] S21, build detection graph G D =(V D ,E D ): In the 3D point cloud, all detection results are marked as detection nodes V in the detection graph D , detect the edge E of the graph D Then it is the detection node V D If two detection nodes belong to the same type of object, that is, the classification results given by the target detector are the same, in the autonomous driving task, the classification results are mainly vehicles or pedestrians, and the distance between the two is within the preset first threshold T, then the two detection nodes are connected and the detection edge E is connected. D The weight of is the distance value. In some embodiments of the present invention, the first threshold T D It is determined based on the statistical value of the maximum speed of a certain category (vehicle or pedestrian) in the data set.

[0063] S22. Construct trajectory graph G T =(V T ,ET ): The construction process is the same as step S21, the trajectory node V in the trajectory graph T is the predicted position point of the historical trajectory in the current frame, and the edge E of the trajectory graph T Then it is the trajectory node V T If the two trajectory nodes belong to the same type of object and the distance between them is within the preset second threshold T T If the two trajectory nodes are connected, the edge E T The weight of is the distance value; in some embodiments of the present invention, the second threshold T T Determined based on the maximum velocity of a category in the dataset.

[0064] S23. Construct trajectory-detection association graph G A =(V D ,V T ,E A ): Trajectory-detection association graph G A For a bipartite graph, edge E A Connected to the detection node V D and trajectory node V T , where edge E A It is only based on objects belonging to the same category and whose distance is within the preset third threshold T A Detection node V within the range D and trajectory node V T between.

[0065] S3. Extract initial features of the detection map, trajectory map, and trajectory-detection association map.

[0066] Step S3 includes the following sub-steps:

[0067] S31, extracting initial features of the detection image;

[0068] Use as Figure 2 The feature extraction network shown performs an initial feature extraction operation on the detection results. The initial features include appearance features and motion features. For appearance features, the point cloud information surrounded by the point cloud detection box is first extracted, and appearance features are extracted from these point clouds. The input point cloud scale is P×4, where P is the number of point clouds surrounded by the bounding box, and 4 represents the four dimensions of (x, y, z, reflectance). After the PointNet algorithm containing 6 one-dimensional convolutional layers and a maximum pooling layer, the final output feature is a 64-dimensional vector f app ; For motion features, we directly use the 3D point cloud detection frame as input and use a two-layer perceptron MLP algorithm to extract motion features from the detection frame to obtain a 64-dimensional vector f mo, the appearance feature vector and the motion feature vector are spliced ​​together to finally obtain a 128-dimensional vector, which is the final extracted initial vector of the detection image, that is, the initial feature of the detection image.

[0069] S32, extracting initial features of the trajectory graph;

[0070] In this step, the trajectory features stored in the historical trajectory library are used as the initial features of the current frame trajectory map.

[0071] S33, extracting initial features of the trajectory-detection association graph.

[0072] In this step, the detection box information difference between the predicted position point of the historical trajectory in the current frame and the detection result of the current frame, including the detection box center point position difference, detection box size difference, and orientation angle difference, is used as the initial feature of the trajectory-association graph.

[0073] S4. Use the feature-enhanced Transformer network to further extract and mine features from the detection map, trajectory map, and detection-trajectory association map, capturing the feature dependency between trajectories and detections, thereby forming a more complete feature representation of the target.

[0074] The overall architecture of the feature-enhanced Transformer network is as follows Figure 6 As shown in the figure, the feature-enhanced Transformer network includes a multi-head self-attention module, an enhanced multi-head cross-attention module, and a feedforward network module. The multi-head self-attention module is used to interact between different detection features and trajectory features, while the enhanced multi-head cross-attention module is used to interact between different trajectory features and detection features. Correlation features are introduced to enhance the correlation between detection features and trajectory features. The feedforward network is used to improve the network's nonlinear capabilities.

[0075] As a preferred embodiment, the feature mining of the target is realized, and the specific process is as follows:

[0076] S41, the multi-head self-attention module includes C self-attention heads, C = 6, and its network structure is as follows Figure 3 As shown. Calculate the query matrix Q, keyword matrix K and value matrix V:

[0077]

[0078]

[0079]

[0080] In the above formula, i indicates that the input feature belongs to the i-th node in the graph, c indicates the c-th attention head, is the learnable parameter of the Query, Key, and Value three-layer fully connected layer of the c-th attention head, f i is the input feature, V c i are the query vector, keyword vector, and value vector of the i-th node of the c-th attention.

[0081] Then, the query matrix Q is correlated with the keyword matrix K, and the feature channel number d is scaled. The Softmax function is used to normalize the feature and obtain an attention matrix that represents the correlation between the targets. Represents the attention value of the i-th node to the j-th node under the c-th attention head:

[0082]

[0083] In the above formula, represents the transpose of the query vector q of the i-th node, k j represents the keyword vector k of the j-th node, and T represents the transposed matrix.

[0084] It is worth noting that during the normalization process, only the neighborhood points of the i-th node are considered, that is, the points connected to the i-th node in the graph, rather than all the points in the graph.

[0085] The attention matrix α acts on the value matrix V to obtain the feature output of the cth attention head, and the features of multiple attention heads are concatenated to obtain the output feature

[0086]

[0087] In the above formula, Represented as the feature of the i-th node in the trajectory graph T, W O is the parameter of the output fully connected layer, || represents the concatenation operation of each attention head vector, N (i) represents all neighboring points of the i-th node, Represents the value vector v of the j-th node.

[0088] Finally, as shown in the following formula, using Figure 5 The feedforward neural network shown outputs features Further processing is performed to obtain the final output f i T .

[0089]

[0090] Through the above operations, all target features interact with each other, which helps the targets obtain global information and learn the position dependencies between each other, and obtain the corrected target features, which will serve as the input of the subsequent enhanced cross-attention module.

[0091] S42, the enhanced multi-head cross attention module includes C = 6 cross attention heads, and its network structure is as follows Figure 4 Specifically, each attention head will perform the following operations in parallel. It should be noted that the letter c in the variables below represents the cth attention head.

[0092] First enter the detection feature Generate a query matrix through the fully connected layer Next, the trajectory features output from the encoder Generate keyword matrices through the fully connected layer and value matrix Will and Perform dot product operation and add the associated features of the trajectory-association graph, then perform feature scaling on the number of feature channels d, and finally normalize through the Softmax layer to generate an attention weight matrix The attention matrix acts on the value matrix The V values ​​of all neighborhood points are weighted and summed, and processed through a feedforward neural network to obtain the feature output of the detection image.

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] In the above formula, W v c are the parameters of the Query, Key, and Value fully connected layers of the c-th attention head. is the association feature between the i-th node and the j-th node in the trajectory-detection association graph, It is a one-dimensional vector obtained by projecting the associated features, representing part of the attention.

[0100] Similarly, the feature update of the trajectory-detection association graph is shown in the following formula.

[0101]

[0102]

[0103] This cross-attention mechanism adaptively finds the strength of feature associations between different nodes based on feature relevance, assigning higher weights to features at highly correlated locations and weakening features at less correlated locations, achieving better fusion. Furthermore, compared to the conventional Transformer cross-attention mechanism, this enhanced cross-attention mechanism allows the network to learn the associations between trajectories and detections and perform weighted fusion. Therefore, the proposed deformable cross-attention mechanism is more robust in a variety of real-world scenarios.

[0104] S43, the network structure diagram of the feedforward network is as follows Figure 5 As shown in the figure, it includes two fully connected layers, and the output of the first fully connected layer is activated by the RELU activation function. Since self-attention and cross-attention are basically linear transformations, adding a feedforward network here provides more nonlinear properties to the model and improves its overall expressiveness.

[0105] S5, such as Figure 7 As shown in the figure, the MLP network is used to calculate the associated features and obtain the similarity between the trajectory and the detection result. The higher the similarity, the more likely it is that the trajectory and the detection result belong to the same target. The greedy algorithm is used to associate the data, and then the matched trajectory-detection pairs, unmatched trajectories, and unmatched detections are obtained.

[0106] S6, such as Figure 8 As shown in Figure 1, the detection features are calculated through the MLP network to estimate the speed of the detection result, and the position of the trajectory in the next frame is estimated based on this speed.

[0107] S7. Based on the successfully associated trajectory-detection pairs, unsuccessfully associated trajectories, and unsuccessfully associated detection results obtained through data association, the historical trajectory database is managed and updated using trajectory features and detection features.

[0108] For successfully associated trajectory-detection pairs, the position of the historical trajectory in the current frame is updated, and the trajectory features are updated with the features of the detection results. For unsuccessfully associated trajectories, if they have not been successfully associated for more than κ frames, they are considered to have disappeared and deleted from the historical trajectory library. For unsuccessfully associated detection results, if they appear for more than κ frames, they are considered to be new targets, and a new trajectory is created for them and added to the trajectory library.

[0109] S8. Define the loss function and use the nuScenes dataset to train the proposed model.

[0110] The loss function includes similarity loss and speed loss

[0111]

[0112] Among them, λ a ,λ v are the weights of each type of loss.

[0113] As a preferred embodiment, in this embodiment, a and λ v Both are set to 1.

[0114] Similarity loss acts on the output of the similarity matrix Among them, m and n are the number of historical trajectories and the number of current frame detection results respectively. To solve the problem of imbalance between positive and negative samples, FocalLoss is used to calculate the similarity loss:

[0115]

[0116] In the formula, p represents the probability that the model predicts that a certain test result is the correct result, that is, p reflects the degree of closeness to the true value. The larger the value, the closer it is to the true value and the more accurate the prediction. α is the weight factor, α∈[0,1]. When it is a positive sample, the weight factor is α, and when it is a positive sample, the weight factor is 1-α. γ is the modulation factor, which controls the difficulty of classifying samples.

[0117] As a preferred embodiment, in this embodiment, α=0.5 and γ=1.

[0118] The speed loss acts on the output of the target speed and uses L1 loss to calculate the error between the parameters of the positive sample and the true value. The loss calculation formula is:

[0119]

[0120] Where, Represents the model's estimation of the detection speed, x gt represents the true value, N pos is the number of positive samples.

[0121] Finally, the proposed method was trained on the nuScenes public dataset for autonomous driving, which focuses on perception and prediction for autonomous vehicles. This dataset contains 1,000 20-second scene sequences, divided into 700, 150, and 150 scene sequences for training, validation, and testing, respectively. Data were collected from diverse locations, including urban, residential, suburban, and industrial areas in Boston and Singapore, and across various weather conditions, including daytime, nighttime, sunny, rainy, and cloudy. Driving data was collected using a variety of sensors, including multi-view cameras and a 32-beam lidar, providing 360° field of view perception.

[0122] In terms of indicator evaluation, the dataset proposes the main evaluation indicators of average tracking accuracy (AMOTA) and average tracking error (AMOTP), and the calculation formula is as follows:

[0123]

[0124]

[0125] Where n represents the number of tracked targets. i Represents the number of frames in which target i is successfully tracked, fp i represents the number of frames in which target i is detected incorrectly, miss i represents the number of frames where target i is missed, gt i represents the true value of target i.

[0126]

[0127]

[0128] Where n represents the number of tracked targets, c i represents the number of frames in which target i appears, d ij Represents the Euclidean distance between target i and the true value in the jth frame.

[0129] The experimental results of this embodiment on the nuScenes dataset are shown in Table 1. Obviously, the tracking accuracy of the method of the present invention is higher than that of the existing method, and the error is lower.

[0130]

[0131] In some embodiments, as Figure 2As shown, a three-dimensional multi-target tracking system based on feature-enhanced Transformer is provided, which is used to implement the method provided in the aforementioned embodiment, including a target detection module, a graph construction module, a feature extraction module, a feature-enhanced Transformer feature mining module, a data association module, a speed estimation module, and a trajectory management and update module.

[0132] The object detection module uses the object detector to generate 3D object detection results for the current frame, including information such as the target bounding box category and location parameters. The graph construction module constructs the detection graph, trajectory graph, and detection-trajectory association graph. The feature extraction module extracts the initial appearance and motion features of the three types of graphs, enriching the feature information. The feature-enhanced Transformer feature mining module captures the dependencies between trajectories and detections, forming a more complete feature representation of the target. The data association module calculates the similarity between historical trajectories and the current frame detection results and performs matching and association. The velocity estimation module estimates the velocity of the detection results to support subsequent trajectory position prediction. The trajectory management and update module updates successfully matched trajectories, creates new trajectories, or deletes disappeared trajectories.

[0133] The system introduces richer feature information by extracting appearance and motion features, and uses feature-enhanced Transformer to perform deep feature mining, capturing the feature dependencies between trajectories and detections, thereby forming a more complete feature expression of the target. At the same time, it abandons the traditional heuristic data association method, reduces manual parameter setting, and improves the versatility of the network.

[0134] In some embodiments, a three-dimensional multi-target tracking apparatus based on a feature-enhanced Transformer is provided, comprising:

[0135] at least one processor;

[0136] at least one memory for storing at least one program;

[0137] When the at least one program is executed by the at least one processor, the at least one processor implements the method provided in the aforementioned embodiment.

[0138] In some embodiments, a storage medium stores a program executable by a processor, and when the program is executed by a processor, it is used to perform the method provided in the above embodiments.

[0139] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A three-dimensional multi-target tracking method based on feature-enhanced Transformer, characterized by: The following steps are involved: Obtain 3D point cloud data and use the object detector to generate the 3D object detection results of the current frame, including the category and position parameter information of the object bounding box; Dynamically construct detection graphs, trajectory graphs, and trajectory-detection association graphs based on historical trajectories and current frame detection results; Extracting initial features of the detection map, trajectory map, and trajectory-detection association map; The feature-enhanced Transformer network dynamically fuses and mines initial features to output richer feature information. This includes a multi-head self-attention mechanism, an enhanced multi-head cross-attention mechanism, and a feedforward network. The multi-head self-attention mechanism is used for the interaction between different detection features and trajectory features, while the enhanced multi-head cross-attention mechanism is used for the interaction between different trajectory features and detection features. Correlation features are introduced to enhance the correlation between detection features and trajectory features, and the feedforward network is used to improve the network's nonlinear capabilities. Calculate the similarity between the trajectory and the detection results based on the associated features and perform data association; According to the detected features, the velocity is estimated and the position of the trajectory in the next frame is predicted based on the estimated velocity; Use trajectory features and detection features to manage and update trajectories; The steps of constructing the detection graph, trajectory graph, and trajectory-detection association graph include: Building a detection graph : In the 3D point cloud, all detection results are marked as nodes in the detection graph , detect the edges of the graph The detection node If the two detection nodes belong to the same type of object and the distance between them is within the preset first threshold If the two detection nodes are within , the two detection nodes are connected, and the weight of the edge is the distance value; Build a trajectory map : Trajectory nodes in the trajectory graph is the predicted position point of the historical trajectory in the current frame, and the edge of the trajectory graph The trajectory node If the two trajectory nodes belong to the same type of object and the distance between them is within the preset second threshold If the two trajectory nodes are within , the edge weight is the distance value; Constructing trajectory-detection association graph : Trajectory-detection association graph For a bipartite graph, the edge Connected to the detection node and trajectory nodes , where the edge Only based on objects belonging to the same category and whose distance is within the preset third threshold Detection nodes within range and trajectory nodes between.

2. The three-dimensional multi-target tracking method based on feature-enhanced Transformer according to claim 1, characterized in that: The extracting of initial features of the detection graph, trajectory graph, and trajectory-detection association graph includes: Extract the initial features of the detection image, which include appearance features and motion features. For appearance features, first extract the point cloud information surrounded by the point cloud detection box and perform appearance feature extraction on these point clouds. For motion features, use the 3D point cloud detection box as input and use the multi-layer perceptron (MLP) algorithm to extract motion features of the detection box. Extract the initial features of the trajectory map: use the trajectory features saved in the historical trajectory library as the initial features of the trajectory map of the current frame; Extracting initial features of the trajectory-detection association graph: The difference in detection box information between the predicted position of the historical trajectory in the current frame and the detection result of the current frame is used as the initial feature of the trajectory-detection association graph.

3. The three-dimensional multi-target tracking method based on feature-enhanced Transformer according to claim 1, characterized in that: Based on the trajectory-detection correlation features, the similarity between the trajectory and the detection results is calculated and data association is performed, including: The correlation features are input into the multi-layer perceptron (MLP) network to calculate the similarity score between the trajectory and the detection result. Based on the similarity score, the greedy algorithm is used for correlation matching to obtain successfully associated trajectory-detection pairs, unsuccessfully associated trajectories, and unsuccessfully associated detection results.

4. The three-dimensional multi-target tracking method based on feature-enhanced Transformer according to claim 1, characterized in that: The speed estimation is performed based on the detection features, including: inputting the detection features into a multi-layer perceptron (MLP) network to calculate the speed of the current frame detection result for subsequent position estimation.

5. The three-dimensional multi-target tracking method based on feature-enhanced Transformer according to claim 1, characterized in that: The use of trajectory features and detection features to manage and update trajectories includes: According to the successfully associated trajectory-detection pairs, unsuccessfully associated trajectories, and unsuccessfully associated detection results, the historical trajectory library is managed and updated using trajectory features and detection features. For successfully associated trajectory-detection pairs, the position of the historical trajectory in the current frame is updated, and the trajectory features are updated with the features of the detection results. For unsuccessfully associated trajectories, if the unsuccessful association exceeds frames, it is considered to have disappeared and deleted from the historical trajectory library; for the detection results that are not successfully associated, if it appears for more than If a frame is found, it is considered a new target, and a new trajectory is created for it and added to the trajectory library.

6. The three-dimensional multi-target tracking method based on feature-enhanced Transformer according to any one of claims 1 to 5, characterized in that: Define the loss function , the loss function includes similarity loss and speed loss : in, 、 are the weights of each type of loss respectively; the similarity loss acts on the output of the similarity matrix, and the similarity loss is calculated to address the imbalance problem of positive and negative samples; the speed loss acts on the output of the target speed, and the L1 loss is used to calculate the error between the parameters of the positive sample and the true value.

7. A 3D multi-target tracking system based on feature-enhanced Transformer, characterized by: The method for implementing any one of claims 1 to 6 comprises the following modules: An object detection module is configured to generate a three-dimensional object detection result for the current frame using an object detector, including the category and position parameter information of the object bounding box; A graph construction module is used to dynamically construct a detection graph, a trajectory graph, and a trajectory-detection association graph based on historical trajectories and current frame detection results. The steps of constructing the detection graph, the trajectory graph, and the trajectory-detection association graph include: constructing the detection graph : In the 3D point cloud, all detection results are marked as nodes in the detection graph , detect the edges of the graph The detection node If the two detection nodes belong to the same type of object and the distance between them is within the preset first threshold If the distance between the two detection nodes is less than , the two detection nodes are connected, and the edge weight is the distance value; construct a trajectory graph : Trajectory nodes in the trajectory graph is the predicted position point of the historical trajectory in the current frame, and the edge of the trajectory graph The trajectory node If the two trajectory nodes belong to the same type of object and the distance between them is within the preset second threshold If the two trajectory nodes are within , the edge weight is the distance value; construct the trajectory-detection association graph : Trajectory-detection association graph For a bipartite graph, the edge Connected to the detection node and trajectory nodes , where the edge Only based on objects belonging to the same category and whose distance is within the preset third threshold Detection nodes within range and trajectory nodes between; A feature extraction module, wherein the feature extraction module is used to extract initial features of the detection map, the trajectory map, and the trajectory-detection association map; A feature-enhanced Transformer module, which is used to dynamically fuse and mine initial features to output richer feature information. It includes a multi-head self-attention mechanism, an enhanced multi-head cross-attention mechanism, and a feedforward network. The multi-head self-attention mechanism is used for interaction between different trajectory features, the enhanced multi-head cross-attention mechanism is used for interaction between different trajectory features and detection features, and the introduction of correlation features enhances the correlation between detection features and trajectory features. The feedforward network is used to improve the nonlinear capability of the network. A data association module, which calculates the similarity between the trajectory and the detection result based on the association features and performs data association; A speed estimation module, which estimates the speed according to the detected features and predicts the position of the trajectory in the next frame based on the speed; The trajectory management and update module uses trajectory features and detection features to manage and update the trajectory.

8. A three-dimensional multi-target tracking device based on feature-enhanced Transformer, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images