DETR-based multi-target detection and tracking method

Abstract

The invention provides a DETR-based 3D target detection and tracking method. The method comprises the steps of obtaining a point cloud data stream of continuous frames from an LIDER camera; original point cloud data of continuous frames are converted into regular voxel grids, and high-dimensional video features of the regular voxel grids are extracted; a group of 4D Tubelet Query is learned, and a time self-attention module and a space cross attention module are utilized to perform modeling on dynamic space-time characteristics of video clips so as to enhance the characterization capability of the dynamic space-time characteristics; and finally, performing bounding box prediction on each Tubelet Query by the model, and measuring the difference between a true value and a predicted value by using set-to-set loss to realize end-to-end learning of a multi-target tracking task. According to the method, the Tubelet is designed, the information in the space-time dimension is deeply mined and integrated, and a multi-target tracking task is simplified into a detection task, so that the unification of the detection task and the tracking task is realized, and the serious dependence of traditional multi-target tracking on a frame-level detector is reduced.

Description

Technical Field

[0001] The present invention relates to the fields of unmanned vehicle automatic driving and robot autonomous navigation, and in particular to a multi-target detection and tracking method based on DETR. Background Art

[0002] LiDAR is an important sensing and perception tool in autonomous driving because it can provide high-precision 3D point cloud data of the scanned environment. LiDAR-based Multiple Object Tracking (MOT) aims to stably detect and track the bounding boxes of objects in LiDAR point cloud sequences. However, the task of 3D MOT in point cloud sequences remains very challenging. Compared with image domain object detection, the scan points in LiDAR data can be sparse and irregularly spaced depending on the distance from the sensor. Most of the latest methods rely on discretizing the point cloud into volumes or projecting Bird's Eye View (BEV) feature maps to use 2D or 3D convolutional networks. Sometimes, it requires a second stage RCNN-style refinement network to compensate for the information loss in voxelization. However, the current two-stage networks lack context and global information learning and cannot be trained end-to-end. In addition, the environment of autonomous driving scenarios is not static, and local feature learning has more limitations when using scan sequences.

[0003] In the field of image processing, the Transformer architecture has proven its competitiveness in object detection and semantic segmentation tasks, especially through its ability to capture global contextual information. DETR, as a notable example, simplifies the detection process through end-to-end training, eliminating the manual post-processing steps in traditional methods. However, DETR and similar models face two major challenges: first, the computational cost grows quadratically with the input size, limiting its ability to process high-resolution feature maps, thereby affecting the detection accuracy of small objects; second, the query embedding learning process may lead to difficulties in training convergence.

[0004] Therefore, in order to solve the limitations of the DETR model in 3D MOT, it is necessary to provide a DETR-based multi-target detection and tracking method to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method for multi-target detection and tracking based on DETR, so as to solve the problem that the existing traditional multi-target detection and tracking methods cannot be trained end-to-end and over-rely on detectors, resulting in poor tracking effect.

[0006] To achieve the above object, the present invention provides a method for multi-target detection and tracking based on DETR, comprising the following steps:

[0007] S1: Obtain a point cloud sequence data stream of continuous frames from the LIDER camera and divide the entire sequence into multiple overlapping K-frame segments.

[0008] S2: Convert the original point cloud data of consecutive K frames into a regular voxel grid, use a 3D sparse convolutional network to extract voxel features from the voxelized data and compress them into K-frame BEV features, and then merge the K-frame BEV features into high-dimensional video features.

[0009] S3: Initialize a set of N Tubelet Queries, each query represents a high-dimensional vector for subsequent Tubelet prediction. Decode the center reference point of the 3D space from the Tubelet Queries, and use the camera transformation matrix to project the 3D center point onto the high-dimensional video features.

[0010] S4: Design a temporal attention mechanism to extract the features of the center position as the query embedding of the decoder, and use a linear layer to encode the center position as a position embedding. By using the center feature as the initial Tubelet Queries embedding, the training is guided to focus on features containing meaningful object information. And the longitudinal features in the video features are integrated to obtain feature information of different time frames to capture the temporal dynamics of the target. The specific implementation is to splice the center point features obtained in the video features and perform self-attention after splicing:

[0011]

[0012]

[0013] Among them, f li is the center point feature after splicing, f′ li It is the feature processed by temporal self-attention.

[0014] S5: Based on deformable attention, spatial cross attention is designed to sample key points, and each TubeletQueries only interacts with its interest area in the BEV feature of the current frame. The complexity of this cross attention is 0 (9SN), which is more efficient than ordinary attention. The cross attention based on variability can be expressed as:

[0015]

[0016] where q (l+1)i It is located in c li Tubelet Query, f liIt is a feature processed by temporal self-attention and adds residual connections to continuously update Tubelet Queries. This module enables each Tubelet Query to selectively extract relevant information from image features based on its current state, gradually refining its understanding of each potential object. (3)

[0017] S6: Tubelet Queries after a series of iterations: q Li , i∈{1,...,N} to predict the time series bounding boxes (Tubelets) of the object, and use these predictions to estimate the object's properties such as position, size, direction, and motion state:

[0018]

[0019] where b i Contains the center coordinates (x c ,y c , z c )∈R 3 , the offset from the center point to the end point (Δx, Δy, Δz)∈R 3 、Tubelet width, height, length (w, h, l) ∈ R 3 (The object moves in a uniform straight line in the K frame, assuming that it remains unchanged in the K frame), the orientation represented by the Euler angle of the Tubelet (θ x ,θ y ,θ z )∈R 3 (assuming it remains unchanged within the K frame), and the object's speed (v x ,v y , v z )∈R 3 .c i is the category prediction C cls is the number of categories.

[0020] S7: Similar to DETR, the present invention uses set-to-set loss to measure the difference between the true value and the predicted value to achieve end-to-end learning of multi-target tracking tasks. First, the optimal bipartite matching is generated between the predicted object and the true object, and then the object-specific loss is optimized.

[0021] Preferably, in step S2, the following steps are specifically included:

[0022] S2.1: Use VoxelNet to voxelize the point cloud to obtain voxelized data {V1, V1, .... V K},in represents the voxelization result of the i-th frame, C inis the number of input channels. After the point cloud data is voxelized, a 3D sparse convolutional network is used to extract high-dimensional features from the voxelized data. Specifically, for each frame V i Get 3D features:

[0023]

[0024] S2.2: Flatten these representations onto the BEV plane (compress the three-dimensional features generated by each point cloud frame along its height dimension to generate a BEV feature map) to obtain a two-dimensional pseudo feature volume:

[0025]

[0026] At the same time, temporal context information helps locate objects and reduce false positives. Therefore, we concatenate multiple frame feature maps along the channel dimension to obtain the video feature representation:

[0027]

[0028] Preferably, in step S3, the following steps are specifically included:

[0029] S3.1: First, initialize N Tubelet Queries, Q0 = {q 01 ,q 02 , ..., q 0N}∈R N×Cq , each query is a high-dimensional vector for subsequent Tubelet prediction. Tubelet Query is decoded into a central reference point in 3D space and the 3D center point is projected onto the video features.

[0030] S3.2: Use bilinear interpolation to sample features at corresponding positions on video features. Establish a connection between TubeletQuery in 3D space and video features to effectively utilize video feature information. The interaction process between Tubelet Queries and video features can be expressed as:

[0031] Each layer l∈{0,...,L-1} acts on a set of Tubelet Query: Generate a new set Q l+1 . Reference point c li Query q from an object li Decoded as follows:

[0032] c li =Φ ref (q li )∈R 3 (8)

[0033] where Φ ref is a neural network. Φ ref can be thought of as a hypothesis for the center of the ith box.

[0034] S3.3: Obtain Φ ref The corresponding video features are used to refine and predict the final bounding box.

[0035] c lki =Proj(c li , k)∈R 2 (9)

[0036] where c lki Represents the projection of the 3D center point on the video feature, and then collects the corresponding features:

[0037]

[0038] f lki is the feature of the i-th point in the k-th frame.

[0039] Preferably, in step S7, the following steps are specifically included:

[0040] S7.1: Given a set of predicted Tubelets {b i , c i} and a set of true target Tubelets {b′ j , c′ j}, the optimal matching solution between the prediction set and the true set is found through the matching function Hungarian algorithm, where σ is a mapping from the prediction set to the true set.

[0041] σ * =argmin σ ∑ i (-logp σ(i) (c i )+λL box (b i , b σ(i) )) (11)

[0042] Find a σ among all possible mappings σ * , so that the value of the entire expression is minimized. Here p σ(i) (c i ) indicates the predicted category c i The probability of belonging to the true category, and L box (b i , b σ(i) ) is the predicted bounding box b i and the matching ground-truth bounding box b σ(i) The regression loss between .

[0043] S7.2: Find the best match σ * After that, the total loss L is composed of the sum of the classification loss and regression loss of all matching items:

[0044]

[0045] Here, λ is a hyperparameter used to balance the contribution between classification loss and regression loss. By continuously optimizing the model through changing the loss function, the model can find the best match between the predicted Tubelets and the true target during the training process, thereby minimizing the overall loss.

[0046] Therefore, the present invention adopts the above-mentioned multi-target detection and tracking method based on DETR, which has the following beneficial effects:

[0047] (1) This paper proposes a 3D MOT based on dynamic point cloud. Compared with the TBD framework, this framework reduces the heavy reliance on frame-level detection results. It implements end-to-end learning, enabling the network to complete detection and tracking tasks simultaneously in a single forward propagation, overcoming the limitations of traditional step-by-step processing of detection and tracking.

[0048] (2) When performing 4D Tubelet detection, prediction is made directly on the Tubelets set, bypassing the limitations of post-processing in the traditional two-stage object detection network, further improving the efficiency and accuracy of detection, and avoiding the performance bottleneck introduced by the proxy regression problem.

[0049] (3) The present invention introduces a more simplified 4D Tubelet representation method, which represents the Tubelet as the center coordinates of the Tubelet in the center frame of the time series, the offset from the center point to the end point, the width, height, length of the Tubelet, the orientation represented by the Euler angle of the Tubelet, and the movement speed of the object, which simplifies the Tubelet modeling and improves the efficiency and robustness of the algorithm.

[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a multi-target detection and tracking method based on DTER provided in an embodiment of the present invention;

[0052] Figure 2 A temporal self-attention module provided by an embodiment of the present invention;

[0053] Figure 3 A spatial self-attention module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0055] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0056] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] like Figure 1 , Figure 2 , Figure 3 As shown, a multi-target detection and tracking method based on DETR disclosed in an embodiment of the present invention mainly includes the following steps:

[0058] S1: Obtain a point cloud sequence data stream of continuous frames from the LIDER camera and divide the entire sequence into multiple overlapping K-frame segments.

[0059] S2: Convert the original point cloud data of consecutive K frames into a regular voxel grid, extract voxel features from the voxelized data using a 3D sparse convolutional network, compress them into K-frame BEV features, and merge the K-frame BEV features into high-dimensional video features. Specifically, the following steps are included:

[0060] S2.1: Use VoxelNet to voxelize the point cloud to obtain voxelized data {V1, V1, .... V K},in represents the voxelization result of the i-th frame, C in is the number of input channels. After the point cloud data is voxelized, a 3D sparse convolutional network is used to extract high-dimensional features from the voxelized data. Specifically, for each frame V i Get 3D features:

[0061]

[0062] S2.2: Flatten these representations onto the BEV plane (compress the three-dimensional features generated by each point cloud frame along its height dimension to generate a BEV feature map) to obtain a two-dimensional pseudo feature volume:

[0063]

[0064] At the same time, temporal context information helps locate objects and reduce false positives. Therefore, we concatenate multiple frame feature maps along the channel dimension to obtain the video feature representation:

[0065]

[0066] S3: Initialize a set of N Tubelet Queries, each of which represents a high-dimensional vector for subsequent Tubelet prediction. Decode the center reference point of the 3D space from the Tubelet Queries, and use the camera transformation matrix to project the 3D center point onto the high-dimensional video features. The specific steps include:

[0067] S3.1: First, initialize N Tubelet Queries. Each query is a high-dimensional vector used for subsequent Tubelet prediction. The Tubelet Query is decoded into a central reference point in 3D space and the 3D center point is projected onto the video features.

[0068] S3.2: Use bilinear interpolation to sample features at corresponding positions on video features. Establish a connection between TubeletQuery in 3D space and video features to effectively utilize video feature information. The interaction process between Tubelet Queries and video features can be expressed as:

[0069] Each layer l∈{0,...,L-1} acts on a set of Tubelet Query: Generate a new set Q l+1 . Reference point c li Query q from an object li Decoded as follows:

[0070] c li =Φ ref (q li )∈R 3 (4)

[0071] where Φ ref is a neural network. Φ refcan be thought of as a hypothesis for the center of the ith box.

[0072] S3.3: Obtain Φ ref The corresponding video features are used to refine and predict the final bounding box.

[0073] C lki =Proj(c li , k)∈R 2 (5)

[0074] Among them C lki Represents the projection of the 3D center point on the video feature, and then collects the corresponding features:

[0075]

[0076] f lki is the feature of the i-th point in the k-th frame.

[0077] S4: Design a temporal attention mechanism to extract the features of the center position as the query embedding of the decoder, and use a linear layer to encode the center position as a position embedding. By using the center feature as the initial Tubelet Queries embedding, the training is guided to focus on features containing meaningful object information. And the longitudinal features in the video features are integrated to obtain feature information of different time frames to capture the temporal dynamics of the target. The specific implementation is to splice the center point features obtained in the video features and perform self-attention after splicing:

[0078]

[0079]

[0080] Among them, f li is the center point feature after splicing, f′ li It is a feature processed by temporal self-attention.

[0081] S5: Based on deformable attention, spatial cross attention is designed to sample key points, and each TubeletQueries only interacts with its interest area in the BEV feature of the current frame. The complexity of this cross attention is 0 (9SN), which is more efficient than ordinary attention. The cross attention based on variability can be expressed as:

[0082]

[0083] where q (l+1)i It is located in c li Tubelet Query, f′ liIt is a feature processed by temporal self-attention and adds residual connections to continuously update Tubelet Queries. This module enables each Tubelet Query to selectively extract relevant information from image features based on its current state, gradually refining its understanding of each potential object.

[0084] S6: Tubelet Queries after a series of iterations: q Li , i∈{1,...,N} to predict the time series bounding boxes (Tubelets) of the object, and use these predictions to estimate the object's properties such as position, size, direction, and motion state:

[0085]

[0086] where b i Contains the center coordinates (x c ,yc,z c )∈R 3 , the offset from the center point to the end point (Δx, Δy, Δz)∈R 3 、Tubelet width, height, length (w, h, l) ∈ R 3 (The object moves in a uniform straight line in the K frame, assuming that it remains unchanged in the K frame), the orientation represented by the Euler angle of the Tubelet (θ x ,θ y ,θ z )∈R 3 (assuming it remains unchanged within the K frame), and the object's speed (v x ,v y , v z )∈R 3 .c i is the category prediction C cls is the number of categories.

[0087] S7: Similar to DETR, the present invention uses set-to-set loss to measure the difference between the true value and the predicted value to achieve end-to-end learning of multi-target tracking tasks. First, the optimal two-part match is generated between the predicted object and the true object, and then the object-specific loss is optimized. Specifically, the following steps are included:

[0088] S7.1: Given a set of predicted Tubelets {b i , c i} and a set of true target Tubelets {b′ j , c′ j}, the optimal matching solution between the prediction set and the true set is found through the matching function Hungarian algorithm, where σ is a mapping from the prediction set to the true set.

[0089] σ * =argmin σ ∑ i (-logp σ(i) (c i )+λL box (b i , b σ(i) )) (11)

[0090] Find a σ among all possible mappings σ * , so that the value of the entire expression is minimized. Here p σ(i) (c i ) indicates the predicted category c i The probability of belonging to the true category, and L box (b i , b σ(i) ) is the predicted bounding box b i and the matching ground-truth bounding box b σ(i) The regression loss between .

[0091] S7.2: Find the best match σ * After that, the total loss L is composed of the sum of the classification loss and regression loss of all matching items:

[0092]

[0093] Here, λ is a hyperparameter used to balance the contribution between classification loss and regression loss. By continuously optimizing the model through changing the loss function, the model can find the best match between the predicted Tubelets and the true target during the training process, thereby minimizing the overall loss.

[0094] Embodiment 1

[0095] like Figure 1 As shown in the figure, given K frames of dynamic point cloud, the original point cloud data of consecutive K frames are converted into regular voxel grids, and voxel features are extracted from the voxelized data using a 3D sparse convolutional network and compressed into K-frame BEV features, and the K-frame BEV features are merged into high-dimensional video features. Then a set of Tubelet Queries are initialized, each query represents a high-dimensional vector for subsequent Tubelet prediction. The central reference point of the 3D space is decoded from the Tubelet Queries, and the 3D center point is projected onto the high-dimensional video features using the camera transformation matrix.

[0096] like Figure 2As shown in the figure, a temporal attention mechanism is designed to extract the features of the center position as the query embedding of the decoder, and use a linear layer to encode the center position as a position embedding. By using the center feature as the initial Tubelet Queries embedding, the training is guided to focus on features containing meaningful object information. The longitudinal features in the video features are fused to obtain feature information of different time frames to capture the temporal dynamics of the target. The specific formula is implemented by splicing the center point features obtained in the video features and performing self-attention after splicing:

[0097]

[0098]

[0099] Among them, f li is the center point feature after splicing, f′ li It is the feature processed by temporal self-attention.

[0100] like Figure 3 As shown in the figure, based on deformable attention, spatial cross attention is designed to sample key points, and each Tubelet Queries only interacts with its interest area in the BEV feature of the current frame. The complexity of this cross attention is 0 (9SN), which is more efficient than ordinary attention. The cross attention based on variability can be expressed as:

[0101]

[0102] where q (l+1)i It is located in c li Tubelet Query, f li It is a feature processed by temporal self-attention and adds residual connections to continuously update Tubelet Queries. This module enables each Tubelet Query to selectively extract relevant information from image features based on its current state, gradually refining its understanding of each potential object.

[0103] Then, through a series of iterations, the updated Tubelet Queries: Li , i∈{1,...,N} to predict the time series bounding boxes (Tubelets) of the object, and use these predictions to estimate the object's properties such as position, size, direction, and motion state:

[0104]

[0105] where b i Contains the center coordinates (x c ,y c, z c )∈R 3 , the offset from the center point to the end point (Δx, Δy, Δz)∈R 3 、Tubelet width, height, length (w, h, l) ∈ R 3 (The object moves in a uniform straight line in the K frame, assuming that it remains unchanged in the K frame), the orientation represented by the Euler angle of the Tubelet (θ x ,θ y ,θ z )∈R 3 (assuming it remains unchanged within the K frame), and the object's speed (v x , v y , v z )∈R 3 .c i is the category prediction C cls is the number of categories.

[0106] Finally, we use set-to-set loss to measure the difference between the true value and the predicted value to achieve end-to-end learning of the multi-target tracking task, first generating the best bipartite match between the predicted object and the true object, and then optimizing the object-specific loss. The specific steps include:

[0107] Given a set of predicted Tubelets {b i , c i} and a set of true target Tubelets {b′ j , c′ j}, the optimal matching solution between the prediction set and the true set is found through the matching function Hungarian algorithm, where σ is a mapping from the prediction set to the true set.

[0108] σ * =argmin σ ∑ i (-logp σ(i) (c i )+λL box (b i , b σ(i) )) (5)

[0109] Find a σ among all possible mappings σ * , so that the value of the entire expression is minimized. Here p σ(i) (c i ) indicates the predicted category c i The probability of belonging to the true category, and L box (b i , b σ(i) ) is the predicted bounding box b i and the matching ground-truth bounding box b σ(i)The regression loss between .

[0110] Find the best match σ * After that, the total loss L is composed of the sum of the classification loss and regression loss of all matching items:

[0111]

[0112] Here, λ is a hyperparameter used to balance the contribution between classification loss and regression loss. By continuously optimizing the model through changing the loss function, the model can find the best match between the predicted Tubelets and the true target during the training process, thereby minimizing the overall loss.

[0113] Therefore, the present invention adopts the above-mentioned DETR-based multi-target detection and tracking method. By designing Tubelet, it deeply mines and integrates information in the time and space dimensions, simplifies the multi-target tracking task into a detection task, so as to achieve the unification of detection and tracking tasks, and reduce the traditional multi-target tracking The heavy dependence on frame-level detectors.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-target detection and tracking method based on DETR, characterized in that: The following steps are involved: S1: Obtain a point cloud sequence data stream of continuous frames from the LIDER camera and divide the entire sequence into multiple overlapping K-frame segments. S2: Convert the original point cloud data of consecutive K frames into a regular voxel grid, use a 3D sparse convolutional network to extract voxel features from the voxelized data and compress them into K-frame BEV features, and then merge the K-frame BEV features into high-dimensional video features. S3: Initialize a set of N Tubelet Queries, each query represents a high-dimensional vector for subsequent Tubelet prediction. Decode the center reference point of the 3D space from the Tubelet Queries, and use the camera transformation matrix to project the 3D center point onto the high-dimensional video features. S4: Design a temporal attention mechanism to extract the features of the center position as the query embedding of the decoder, and use a linear layer to encode the center position as a position embedding. By using the center feature as the initial Tubelet Queries embedding, the training is guided to focus on features containing meaningful object information. And the longitudinal features in the video features are integrated to obtain feature information of different time frames to capture the temporal dynamics of the target. The specific implementation is to splice the center point features obtained in the video features and perform self-attention after splicing: Among them, f li is the center point feature after splicing, f′ li It is the feature processed by temporal self-attention. S5: Based on deformable attention, spatial cross attention is designed to sample key points, and each Tubelet Queries only interacts with its interest area in the BEV feature of the current frame. The complexity of this cross attention is 0 (9SN), which is more efficient than ordinary attention. The cross attention based on variability can be expressed as: where q (l+1)i It is located in c li Tubelet Query, f′ li It is a feature processed by temporal self-attention and adds residual connections to continuously update Tubelet Queries. This module enables each Tubelet Query to selectively extract relevant information from image features based on its current state, gradually refining its understanding of each potential object. (3) S6: Tubelet Queries after a series of iterations: q Li , i∈{1,...,N} to predict the time series bounding boxes (Tubelets) of the object, and use these predictions to estimate the object's properties such as position, size, direction, and motion state: where b i Contains the center coordinates (x c ,y c , z c )∈R 3 , the offset from the center point to the end point (Δx, Δy, Δz)∈R 3 、Tubelet width, height, length (w, h, l) ∈ R 3 (The object moves in a uniform straight line in the K frame, assuming that it remains unchanged in the K frame), the orientation represented by the Euler angle of the Tubelet (θ x ,θ y ,θ z )∈R 3 (assuming it remains unchanged within the K frame), and the object's speed (v x , v y , v z )∈R 3 .c i is the category prediction C cls is the number of categories. S7: Similar to DETR, the present invention uses set-to-set loss to measure the difference between the true value and the predicted value to achieve end-to-end learning of multi-target tracking tasks. First, the optimal bipartite matching is generated between the predicted object and the true object, and then the object-specific loss is optimized.

2. The method for multi-target detection and tracking based on DETR according to claim 1, characterized in that: The specific method of step S2 is as follows: S2.1: Use VoxelNet to voxelize the point cloud to obtain voxelized data {V1, V1, .... V K },in represents the voxelization result of the i-th frame, C in is the number of input channels. After the point cloud data is voxelized, a 3D sparse convolutional network is used to extract high-dimensional features from the voxelized data. Specifically, for each frame V i Get 3D features: S2.2: Flatten these representations onto the BEV plane (compress the three-dimensional features generated by each point cloud frame along its height dimension to generate a BEV feature map) to obtain a two-dimensional pseudo feature volume: At the same time, temporal context information helps locate objects and reduce false positives. Therefore, we concatenate multiple frame feature maps along the channel dimension to obtain the video feature representation:

3. The method of multi-target detection and tracking based on DETR according to claim 1, characterized in that: The specific method of step S3 is as follows: S3.1: First, initialize N Tubelet Queries, Q0 = {q 01 ,q 02 , ..., q 0N }∈R N×Cq , each query is a high-dimensional vector for subsequent Tubelet prediction. Tubelet Query is decoded into a central reference point in 3D space and the 3D center point is projected onto the video features. S3.2: Use bilinear interpolation to sample features at corresponding positions on video features. Establish a connection between TubeletQuery in 3D space and video features to effectively utilize video feature information. The interaction process between Tubelet Queries and video features can be expressed as: Each layer 1∈{0,...,L-1} acts on a set of Tubelet Query: Generate a new set Q l+1 . Reference point c li Query q from an object li Decoded as follows: c li =Φ ref (what li )∈R 3 (8) where Φ ref is a neural network. Φ ref can be thought of as a hypothesis for the center of the ith box. S3.3: Obtain Φ ref The corresponding video features are used to refine and predict the final bounding box. C lki =Proj(c li ,k)∈R 2 (9) where c lki Represents the projection of the 3D center point on the video feature, and then collects the corresponding features: f lki is the feature of the i-th point in the k-th frame.

4. The method for multi-target detection and tracking based on DETR according to claim 1, characterized in that: The specific method of step S7 is as follows: S7.1: Given a set of predicted Tubelets {b i , c i } and a set of true target Tubelets {b′ j , c′ j }, the optimal matching solution between the prediction set and the true set is found through the matching function Hungarian algorithm, where σ is a mapping from the prediction set to the true set. s * argmin σ S i (-log p σ(i) (c i )+λL box (b) i ,b σ(i) )) (11) Find a σ among all possible mappings σ * , so that the value of the entire expression is minimized. Here p σ(i) (c i ) indicates the predicted category c i The probability of belonging to the true category, and L box (b i , b σ(i) ) is the predicted bounding box b i and the matching ground-truth bounding box b σ(i) The regression loss between . S7.2: Find the best match σ * After that, the total loss L is composed of the sum of the classification loss and regression loss of all matching items: Here, λ is a hyperparameter used to balance the contribution between classification loss and regression loss. By continuously optimizing the model through changing the loss function, the model can find the best match between the predicted Tubelets and the true target during the training process, thereby minimizing the overall loss.

Images