A 3D multi-target tracking method based on multimodal collaboration

Abstract

The present invention discloses a 3D multi-target tracking method based on multimodal collaboration, the method comprising: inputting the currently acquired 3D point cloud sequence into a 3D target detector, and the image sequence into a 2D target detector, respectively obtaining a 3D detection frame and a 2D detection frame and inputting them into a multimodal position optimizer, adjusting the 3D detection frame of the corresponding object using a 2D detection frame based on image semantic information, obtaining an enhanced 3D coordinate frame and projecting it onto a corresponding camera plane to obtain a corresponding area, and obtaining the current appearance features corresponding to the object using an appearance feature extractor; using a trajectory management module to store and update the appearance features and motion features, specifically: performing similarity matching between the current appearance features, the enhanced 3D coordinate frame, and the appearance features and motion features of the historical trajectory; updating the appearance features and motion features according to the matching results; and outputting the tracking results from the trajectory management module after the update. The method of the present invention improves the 3D multi-target tracking effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-target tracking, and in particular relates to a 3D multi-target tracking method based on multi-modal collaboration. Background Art

[0002] 3D multi-target tracking has always been an important research direction in computer vision. Its goal is to establish the trajectory of objects of interest in a three-dimensional coordinate system. Its applications include autonomous driving, robotics, intelligent transportation, and virtual reality. For example, in the field of autonomous driving, 3D multi-target tracking technology can achieve real-time perception and prediction of surrounding vehicles, pedestrians and other traffic elements in three-dimensional space to ensure the safety and stability of autonomous vehicles. In the field of robotics, 3D multi-target tracking technology can establish the motion trajectory of the operating target in the workspace, thereby improving the accuracy and efficiency of robot operations.

[0003] The current 3D multi-target tracking methods are divided into two categories: detection-based tracking methods and joint detection and tracking methods. Among them, the detection-based tracking method first detects the target in the current frame through the target detection algorithm, and then inputs the detection results into the motion feature extraction algorithm or the appearance feature extraction algorithm to extract the corresponding features as the basis for association. This method makes full use of the detection accuracy of the target detection algorithm and the robust feature extraction ability of the feature extraction algorithm. The disadvantage is that the network parameters cannot be shared, resulting in excessive calculations, especially in crowded scenes, making it difficult to complete the task of real-time tracking; the joint detection and tracking method is to complete target detection and inter-frame association at the same time through a single network, with a small amount of calculation, and can achieve real-time tracking in crowded scenes. The disadvantage is that it is difficult to select a better feature extraction network for the target object, and there is a conflict between the detection and tracking tasks during the training process.

[0004] In the process of target detection, in order to make up for the performance limitations of 3D target detection algorithms, existing methods attempt to reduce missed detections and false detections by introducing 2D target detection algorithms. However, they simply use the intersection over union (IoU) in the image plane to correlate the 3D detection results with the 2D detection results, and do not fully integrate the advantages of different modal detection algorithms. For example, the positioning accuracy of existing methods still depends solely on the 3D detection algorithm. In the process of extracting motion features, the motion model constructed by the existing methods only uses the 3D detection results as observations, and the motion prediction has a large error. In the process of extracting appearance features, existing methods simply project the 3D detection box onto the corresponding image plane to obtain the 2D area corresponding to the specific object, and then use the feature extraction network to obtain the feature vector of the corresponding area as its appearance feature. Therefore, on the one hand, this type of method will introduce a lot of background information, which greatly damages the quality of appearance features; on the other hand, it only uses the feature extractor pre-trained on other data sets to extract appearance features, which is difficult to adapt to existing tasks, and the feature extraction network parameters are not updated in real time during the test phase, resulting in the appearance features not having strong robustness, which has an adverse effect on the long-term target tracking task. In the process of data association, spatial similarity is an important basis for performing association. However, existing methods only use similarity metrics in 3D space to measure the spatial similarity of objects between frames, which will produce a large amount of confusing spatial similarities in dense scenes and cannot guide the precise association between frames. Summary of the invention

[0005] In view of the above problems, the present invention provides a 3D multi-target tracking method based on multi-modal collaboration, aiming to improve the 3D multi-target tracking effect.

[0006] The technical solution of the present invention is: a 3D multi-target tracking method based on multi-modal collaboration, the method comprising the following steps:

[0007] Input the currently acquired 3D point cloud sequence into the 3D object detector and the image sequence into the 2D object detector to obtain the 3D detection frame and 2D detection frame respectively;

[0008] Inputting the 3D detection frame and the 2D detection frame results into a multimodal position optimizer, and adjusting the 3D detection frame of the corresponding object using the 2D detection frame based on image semantic information to obtain an enhanced 3D coordinate frame;

[0009] The enhanced 3D coordinate frame is projected onto the corresponding camera plane through the known external parameter matrix and internal parameter matrix of the camera to obtain the corresponding area, and the current appearance feature corresponding to the object is obtained by using the appearance feature extractor;

[0010] The trajectory management module is used to store and update appearance features and motion features, specifically:

[0011] Performing similarity matching on the current appearance features, the enhanced 3D coordinate frame, and the historical trajectory appearance features and motion features extracted from the trajectory management module;

[0012] The appearance features and motion features of the successfully matched enhanced 3D coordinate frame are updated, the unmatched enhanced 3D coordinate frame is initialized as a new track, and the unmatched historical track is judged for the number of lost frames. If it is greater than the maximum number of lost frames, the object corresponding to the historical track disappears and the historical track is removed;

[0013] After the update is completed, the trajectory management module outputs the 3D multi-target motion tracking result of the current frame.

[0014] In some embodiments, the 3D object detector adopts a LargeKernel-3D detection network, and the 2D object detector adopts a YoloX detection network.

[0015] In some embodiments, the YoloX detection network has a regression branch and a classification branch, and the specific expression of the regression branch loss is:

[0016] ;

[0017] The specific expression of the classification branch loss is: ;

[0018] The total loss function expression of the 2D target detector is: ;

[0019] in, represents the intersection-over-union loss, represents the intersection-over-union ratio, A represents the detection box output by the 2D target detector, and B represents the object label box. is the multi-classification cross entropy loss, used as the loss function for multi-classification; is the label of the target, Represents the number of object categories; is the probability that the object category is the i-th object, , are all hyperparameters, represents the loss of the regression branch, represents the loss of the classification branch.

[0020] In some embodiments, the LargeKernel-3D detection network has the regression branch, the classification branch, and the speed prediction branch, and the specific expression of the speed prediction branch loss is: , the total loss function expression of the 3D target detector is: ,in It represents the difference between the speed prediction value and the actual speed value, which is the sum of the differences between the speed prediction value in the x and y directions and the corresponding actual speed value. Otherwise, it is not satisfied Under this condition, is a hyperparameter, represents the loss of speed prediction branch.

[0021] In some embodiments, obtaining the 3D frame-by-frame detection result and the 2D frame-by-frame detection result respectively includes the following steps:

[0022] For 2D object detectors and 3D object detectors, the SimOTA label matching strategy is used to obtain positive and negative samples for input sequence detection, where the positive sample is the anchor point that matches the label. The error between the positive sample and the corresponding label is calculated according to the total loss function of the 2D object detector or the total loss function of the 3D object detector, and the parameters are updated through the back propagation process to reduce the error. Finally, the 3D detection boxes of all objects in the 3D point cloud sequence and the 2D detection boxes of all objects in the image sequence are obtained.

[0023] In some embodiments, obtaining an enhanced 3D coordinate frame specifically includes the following steps:

[0024] Associating the 3D detection frame with the 2D detection frame using an intersection-and-union ratio in a 2D space to obtain an object having both the 2D detection frame and the 3D detection frame;

[0025] For the object having both the 2D detection frame and the 3D detection frame, using a multimodal position optimizer to enhance the accuracy of the 3D detection frame to obtain the enhanced 3D coordinate frame specifically includes:

[0026] Newton gradient descent method is used for optimization, and the objective function is as follows:

[0027] ,

[0028] in, Indicates the center point of the optimized 3D detection box , respectively axis, axis, Axis coordinates, Represents the center point of the 2D detection box of the object, Indicates the center point of the 3D detection box of the object, represents the square of the Euclidean distance, , They represent the weights of the distance between the center point of the 2D detection box and the center point of the 3D detection box, respectively, and are obtained by the following formula:

[0029] ,

[0030] in, Represents the confidence score of the detection box, , The inputs correspond to the confidence scores of the 2D detection box and the 3D detection box respectively. , is a hyperparameter;

[0031] In the During round optimization, the optimization formula is as follows:

[0032] ,

[0033] in is the first-order derivative, is the Hessian matrix, and the calculation process is as follows:

[0034] ,

[0035] ,

[0036] When the difference between the two iteration results is less than the set threshold, the iteration is terminated and the final enhanced 3D coordinate frame is obtained. , Indicates the optimized 3D detection frame, that is, the center point coordinates of the enhanced 3D coordinate frame. Indicates the width, length, height, and orientation angle of the enhanced 3D coordinate box.

[0037] In some embodiments, the method includes predicting the motion state of the previous frame trajectory, i.e., the historical trajectory, and specifically includes the following steps:

[0038] A Kalman filter is established for each trajectory, where the state vector Set to:

[0039] ,

[0040] in, Indicates the optimized 3D detection frame, that is, the center point coordinates of the enhanced 3D coordinate frame. Indicates the width, length, height and orientation angle of the enhanced 3D coordinate box. , They are the speeds in the x and y directions of the center point of the object coordinate frame, , They are the accelerations in the x and y directions of the center point of the object coordinate frame; based on the assumption of uniformly accelerated linear motion, the corresponding state transfer matrix is ​​obtained , and predict the current frame position based on the following equation , Indicates the previous frame position;

[0041] After similarity matching between the motion state prediction value of the previous frame trajectory and the enhanced 3D coordinate frame of the current frame, the Kalman filter is updated using multimodal observations, including:

[0042] Set the observation vector for:

[0043] ,

[0044] in, Indicates the 2D coordinate center point of the object corresponding to the observation vector. When the corresponding object only has a 3D detection box, the projection of the center point of the 3D detection box on the image plane is used as its 2D coordinate center point;

[0045] Calculate predicted values With observed value Error ,in is the measurement matrix of the current frame:

[0046] ,

[0047] Then, the updated state vector is obtained by With the covariance matrix :

[0048] ,

[0049] ,

[0050] ,

[0051] ,

[0052] in, is the current frame covariance matrix, is the process noise matrix, is the measurement noise, is the Kalman gain, is the identity matrix.

[0053] In some embodiments, the appearance feature extractor is based on a transformer network, and when tracking the current frame, the previous The frames are used as training frames to train the transformer network, thereby achieving domain adaptation of appearance feature extraction.

[0054] In some embodiments, similarity matching is performed between the current appearance feature, the 2D detection frame, the enhanced 3D coordinate frame, and the historical trajectory appearance feature and motion feature extracted from the trajectory management module, specifically including:

[0055] Associating the 3D detection frame with the 2D detection frame using an intersection-and-union ratio in a 2D space to obtain an object having both the 2D detection frame and the 3D detection frame, an object having only the 3D detection frame, and an object having only the 2D detection frame;

[0056] For objects with both the 2D detection frame and the 3D detection frame, the 3D coordinate frame will be enhanced , 2D detection box Perform cross-modal motion cost matrix calculations with the Kalman filter prediction box respectively:

[0057] ,

[0058] ,

[0059] ,

[0060] in, Represents the i-th coordinate frame in the enhanced 3D coordinate frame set, represents the jth prediction box in the 3D Kalman filter prediction box set, represents the i-th detection box in the 2D detection box set, represents the jth projection frame in the projection frame set of the 3D Kalman filter prediction frame set on the 2D image plane, , Represent the calculation operations of volume and area respectively, , Represent the convex hull calculation operations of volume and area respectively, , They are and The corresponding weight of

[0061] Calculate the appearance features of the i-th historical trajectory and the current appearance features of the jth object Similarity ;

[0062] pass and Construct the cost matrix C, the elements of the cost matrix C The expression is:

[0063] ,

[0064] in, Indicates different values ​​of i and j , Represents the weights of motion features and appearance features in the cost matrix;

[0065] According to the constructed cost matrix C, Hungarian matching is used to associate the object detection of the current frame with the survival trajectory of the previous frame in the first stage, where the survival trajectory of the previous frame is the historical trajectory that has not been removed in the previous frame;

[0066] For objects with only 3D detection boxes, they are matched with unmatched tracks in the first stage of association, including:

[0067] The 3D coordinate frame will be enhanced The Kalman filter prediction box performs Calculation of motion cost matrix;

[0068] The ROI area obtained by projecting the 3D detection frame onto the 2D image plane is input into the appearance feature extraction module to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ;

[0069] pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the second stage;

[0070] For objects with only 3D detection frames and objects with only 2D detection frames, they are matched with unmatched tracks in the second stage association, including:

[0071] The 2D detection box The projection box of the Kalman filter prediction box on the 2D image plane is performed based on Calculation of motion cost matrix;

[0072] The area corresponding to the 2D detection frame is input into the appearance feature extraction module as the ROI area to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ;

[0073] pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the third stage.

[0074] In some embodiments, for the trajectory successfully matched in the first-stage association to the third-stage association, the trajectory is updated, that is, the Kalman filter is updated and the appearance feature is updated;

[0075] If the current frame object detection fails to successfully match the trajectory, it is considered that the current frame object has not appeared in the past frame, a new trajectory is created for it, and its current appearance features are stored in the trajectory management module;

[0076] If the trajectory does not match the object in the current frame, and if the number of lost frames of the trajectory is greater than the threshold number of frames, the trajectory is deleted and cleared in the trajectory management module;

[0077] The survival trajectory of the current frame is output as the 3D multi-target tracking result of the current frame.

[0078] The disclosed embodiment provides a 3D multi-target tracking method based on multimodal collaboration, which can be applied to fields including autonomous driving, robotics, intelligent transportation, and virtual reality. In the field of autonomous driving, 3D multi-target tracking technology can realize real-time perception and prediction of surrounding vehicles, pedestrians and other traffic elements in three-dimensional space to ensure the safety and stability of autonomous driving vehicles. In the field of robotics, 3D multi-target tracking technology can establish the motion trajectory of the operating target in the workspace, thereby improving the accuracy and efficiency of the robot's operation. The 3D multi-target tracking method based on multimodal collaboration of the present invention uses 2D detection results to provide important guidance for optimizing 3D detection results, and uses the potential benefits brought by the 2D detection algorithm to enhance the detection effect; at the same time, the motion trajectory of the target in 2D and 3D space is established by combining 2D detection results, so that the motion prediction error is reduced; in addition, the method of the present invention not only uses the similarity index in 3D space to measure the spatial similarity of objects between frames, but also comprehensively considers the 2D spatial similarity, reduces the confusion of spatial similarity in dense scenes, thereby guiding the accurate association between frames and improving the 3D multi-target tracking effect.

[0079] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0081] Figure 1 It is a flow chart of a 3D multi-target tracking method based on multi-modal collaboration in an embodiment of the present invention;

[0082] Figure 2 is a schematic diagram of the overall network structure in an embodiment of the present invention;

[0083] Figure 3 Schematic diagram of the structure of the online appearance feature module in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0085] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0086] The present invention provides a 3D multi-target tracking method based on multi-modal collaboration and provides the following embodiments:

[0087] like Figure 1As shown in the flowchart of the 3D multi-target tracking method based on multimodal collaboration, the 3D point cloud sequence and the video frame sequence are respectively passed through the 3D target detector and the 2D target detector to obtain the frame-by-frame detection results 3D detection frame and 2D detection frame; the detection results of the 3D detection frame and the 2D detection frame are input into the multimodal position optimizer, and the 3D detection frame of the corresponding object is adjusted by using the 2D detection frame based on the image semantic information, so as to obtain a more accurate object 3D coordinate frame, that is, an enhanced 3D coordinate frame. Subsequently, the 3D coordinate frame is projected onto the corresponding camera plane through the known external parameter matrix and internal parameter matrix of the camera to obtain the corresponding area, and the appearance feature extraction module is used to obtain the corresponding appearance features of the object. At the same time, the appearance features and motion features of the historical trajectory are extracted from the trajectory management module. Subsequently, the 3D coordinate frame of the current frame object is matched with the historical trajectory in the multi-stage association module, where the basis of the matching process is the similarity between the motion feature and the appearance feature. The motion feature matching is the motion state prediction value of the previous frame trajectory in the current frame, and the GIOU-based similarity calculation is performed with the 3D coordinate frame of the current frame. After the matching process is completed, the trajectory of the 3D coordinate frame of the object in the current frame is updated, including the update of the appearance features and the update of the motion features; and the 3D coordinate frame of the current frame object that has not been successfully matched is initialized as a new trajectory, and the historical trajectory that has not been successfully matched is judged for the number of lost frames. If it is greater than the maximum number of lost frames, it is considered that the object corresponding to the trajectory has disappeared, and the specific trajectory is removed. Finally, the trajectory management module is updated and the 3D multi-target motion tracking result of the current frame is output.

[0088] Specifically, Figure 2 As shown in the figure, it is a schematic diagram of the overall network structure in an embodiment of the present invention. The 3D target detector adopts the LargeKernel-3D network, and the 2D target detector adopts the YoloX network. Specifically, the YoloX detection network has two output branches, namely the regression branch and the classification branch, and the losses are as follows:

[0089] regress branch: ,

[0090] Class branch: ,

[0091] represents the intersection-over-union loss, represents the intersection-over-union ratio, A represents the detection box output by the 2D target detector, and B represents the object label box. is the multi-classification cross entropy loss, used as the loss function for multi-classification; is the label of the target. In the specific embodiment, y takes the value [1, 2, 3, 4, 5, 6, 7], representing pedestrians, cars, bicycles, buses, motorcycles, trucks, and trailers, respectively. Represents the number of object categories, the value is 7; is the probability that the object category is the i-th object, ranging from 0 to 1, and the sum of the probabilities of the seven categories of objects is 1.

[0092] The total loss of the 2D object detector is:

[0093] ,

[0094] in, , They are all hyperparameters, and their values ​​here are 5.0 and 1.0 respectively.

[0095] For the 3D object detector LargeKernel-3D network, it has three output branches, namely regression branch, classification branch, and velocity prediction branch. The regress branch and class branch are consistent with the above 2D detector, and the velocity branch loss is as follows:

[0096] Branches: ,

[0097] in, It represents the difference between the speed prediction value and the actual speed value, which is the sum of the differences between the speed prediction value and the corresponding actual speed value in the x and y directions; represents the loss function used for regression task network training, Otherwise, it is not satisfied This condition;

[0098] The total loss of the 3D object detector is:

[0099] ,

[0100] in, , , is a hyperparameter, and the preferred values ​​are 5.0, 1.0, and 1.0, respectively. represents the loss of speed prediction branch. represents the loss of the regress branch, represents the loss of the class branch, Represents the loss of the velocity branch.

[0101] The SimOTA label matching strategy is used to obtain positive and negative samples for detection, where the positive sample is the anchor that matches the label. Then, the error between the positive sample and the corresponding label is calculated based on the total loss function of the designed 2D target detector or the total loss function of the 3D target detector, and the parameters are updated through the back propagation process to reduce the error. The entire 3D point cloud sequence and 2D image sequence are processed by the 3D and 2D target detectors, and finally the 3D detection boxes and 2D detection boxes of all objects in the video sequence are obtained.

[0102] The cross-modal target association module is used to associate the 3D detection box of the target with the 2D detection box. The successfully associated 2D detection box and 3D detection box represent the position of the same object in different coordinate systems. Specifically, the intersection-over-union ratio (2DIoU) in 2D space is used for matching, and the formula is as follows:

[0103] ,

[0104] in, , Respectively represent the 3D detection frame and 2D detection frame of the object, represents the projection function of the 3D detection frame to the 2D image plane, which is determined by the camera's external parameter matrix and internal parameter matrix. Subsequently, the cost matrix of the 3D detection frame and the 2D detection frame can be calculated , where the cost matrix Each of Represents the relationship between the i-th 3D detection frame and the j-th 2D detection frame Then, according to the Hungarian matching strategy, the 3D-2D detection box pair is obtained. When the value is greater than 0.5, the match is successful and an object with both a 2D detection frame and a 3D detection frame is obtained. If the match fails, the object with only a 3D detection box is obtained. and objects with only 2D detection boxes .

[0105] For objects with both 2D and 3D detection frames , the multimodal position optimizer is used to enhance the accuracy of the 3D detection frame. Specifically, the Newton gradient descent method is used for optimization, and the objective function is as follows:

[0106] ,

[0107] in, Indicates the center point of the optimized 3D detection box , respectively axis, axis, Axis coordinates, Represents the center point of the 2D detection box of the object, Represents the center point of the 3D detection box of the object, represents the square of the Euclidean distance, , Represent the weights of the 2D center point distance and the 3D center point distance, respectively, which are obtained by the following formula: ,

[0108] in, Represents the confidence score of the detection box, , The inputs correspond to the confidence scores of the 2D detection box and the 3D detection box respectively. , are hyperparameters, and their preferred values ​​are 10 and 0.5 respectively.

[0109] No. During round optimization, the optimization formula is as follows:

[0110] ,

[0111] in is the first-order derivative, is the Hessian matrix, and the calculation process is as follows:

[0112] ,

[0113] ,

[0114] When the difference between the two iteration results is less than the set threshold (preferably ), the iteration is terminated and the final 3D coordinate frame is obtained , represents the optimized center point coordinates, Represents a 3D detection box Width, length, height and orientation angle.

[0115] like Figure 2 As shown in , for the previous frame trajectory, the motion state prediction is performed. Specifically, a Kalman filter model is established for each trajectory, where the state vector Set to:

[0116] ,

[0117] in , They are the x- and y-direction velocities of the center point of the object coordinate frame, , They are the acceleration in the x and y directions of the center point of the object coordinate frame, Represents the angular velocity of the current trajectory. Based on the assumption of uniformly accelerated linear motion, the corresponding state transfer matrix can be obtained , and predict the current frame position based on the following equation :

[0118] ,

[0119] ,

[0120] in, is the covariance matrix, is the process noise matrix.

[0121] After the motion matching with the current frame object is completed, the Kalman filter is updated using multimodal observations. Specifically, the observation vector is set for:

[0122] ,

[0123] in, Indicates the 2D coordinate center point of the object corresponding to the observation vector. When the corresponding object only has a 3D detection box, the projection of the center point of the 3D detection box on the image plane is used as its 2D coordinate center point.

[0124] Calculate predicted values With observed value Error ,in is the measurement matrix of the current frame:

[0125] ,

[0126] Then, the updated state vector is obtained by With the covariance matrix :

[0127] ,

[0128] ,

[0129] ,

[0130] in, is the measurement noise, is the Kalman gain, is the identity matrix. In addition, the measurement matrix As shown below:

[0131] ,

[0132] like Figure 3As shown in Figure 1, the online ReID module structure diagram is shown in Figure 1. Frame time, front The frames are used as training frames to train the transformer network-based appearance feature (ReID) extractor, thereby achieving domain adaptation of ReID feature extraction. Specifically, for objects with both 2D detection boxes and 3D detection boxes , the optimized 3D coordinate frame Project it onto the 2D image plane to obtain the corresponding area, recorded as the ROI area, for objects with only 3D detection frames Directly project the 3D detection frame onto the 2D image plane to obtain the ROI area. For objects with only a 2D detection frame Directly use the 2D detection box as the ROI area. For the above ROI area, input it into the network model pre-trained in the ReID dataset to extract the corresponding appearance features. , where i represents the identity tag, i.e. the i-th object, is a 256-dimensional feature vector. Then, it is mapped to the class distribution vector through the fully connected layer and SoftMax operation , the losses used are as follows:

[0133] ,

[0134] Based on the historical trajectory information, all object detections belonging to the same trajectory are assumed to be the same object in the training label, and a pseudo label can be obtained. In the above formula, Indicates the one-hot encoding form of the pseudo label, that is, when it is the object, the value is 1, and the rest are 0. K represents the number of all objects in the trajectory data, M represents the number of objects that appeared in the previous frame, and P represents the appearance characteristics of the object The probability of belonging to the kth object.

[0135] Furthermore, the cosine annealing strategy is used for model training, and the initial learning rate of the model is set to .

[0136] Furthermore, in the inference process of the current frame, the ReID model trained on the previous frame is used to extract the appearance features, thereby obtaining a robust appearance feature set of the current frame after domain adaptation. .

[0137] like Figure 2 As shown, the 3D Kalman predicted position set of the trajectory is obtained With historical appearance feature collection Finally, a three-stage association-based strategy is designed to perform detection and trajectory matching. Specifically, for objects with both 2D detection boxes and 3D detection boxes, , firstly, the optimized 3D coordinate frame With 2D detection box Perform cross-modal GIOU-based motion cost matrix calculation with 3D Kalman predicted position:

[0138] ,

[0139] ,

[0140] ,

[0141] in, Represents the i-th coordinate frame in the enhanced 3D coordinate frame set, represents the jth prediction box in the 3D Kalman filter prediction box set, represents the i-th detection box in the 2D detection box set, represents the jth projection frame in the projection frame set of the 3D Kalman filter prediction frame set on the 2D image plane, , Represent the calculation operations of volume and area respectively, , Represent the convex hull calculation operations of volume and area respectively, , They are and The corresponding weight of is set to 1 in the preferred embodiment.

[0142] Calculate the appearance features of the i-th historical trajectory and the current appearance features of the jth object Similarity ;

[0143] Furthermore, the appearance feature vector of the object in the current frame output by the ReID model is calculated The appearance feature vector with the historical trajectory Similarity:

[0144] ,

[0145] in, Represents the appearance feature vector of the jth object in the current frame, Represents the center vector of the appearance feature of the i-th track in the previous frame.

[0146] The cost matrix C is constructed by GIOU and appearance similarity, where the element value at position (i, j) is:

[0147] ,

[0148] Here, α represents the weight of the motion module and the ReID module in the cost matrix, which is set to 0.5 in the preferred embodiment.

[0149] According to the constructed cost matrix C, Hungarian matching is used to associate the current frame object detection, i.e., the enhanced 3D coordinate frame (with both 2D and 3D detection), with the previous frame survival trajectory in the first stage. The previous frame survival trajectory is the historical trajectory that has not been removed in the previous frame;

[0150] For objects with only 3D detection boxes, they are matched with unmatched tracks in the first stage of association, including:

[0151] The 3D coordinate frame will be enhanced The Kalman filter prediction box performs The ROI area obtained by projecting the 3D detection frame onto the 2D image plane is input into the appearance feature extraction module to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ;

[0152] pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the second stage;

[0153] For objects with only 2D detection boxes, they are matched with unmatched tracks in the second stage association, including:

[0154] The 2D detection box The projection box of the Kalman filter prediction box on the 2D image plane is performed based on Calculation of motion cost matrix;

[0155] The area corresponding to the 2D detection frame is input into the appearance feature extraction module as the ROI area to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ;

[0156] pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the third stage.

[0157] like Figure 2As shown, for the trajectory successfully matched in the three-stage association, the trajectory update module, namely the Kalman filter update and the appearance feature vector momentum update, is used to update the motion and appearance state of the trajectory, where the appearance feature vector momentum update is shown in the following formula:

[0158] ,

[0159] in, is the momentum term, which is set to 0.5 in the preferred embodiment.

[0160] Hungarian matching is explained by a specific example. If there are m historical trajectories and n current frame detections, an m*n matrix can be established, where the element in the i-th row and j-th column represents the feature distance (also called matching cost) between the i-th trajectory and the j-th detection. The formula is Then, a greedy algorithm is used to find a matching method that minimizes the global cost, and the (trajectory, detection) pair is obtained. The global cost threshold is screened for these (trajectory, detection) pairs. If it is greater than 0.5, the cost is too high and the matching fails. If the global cost is less than 0.5 (including 0.5), the (trajectory, detection) pair is matched successfully.

[0161] In the first stage, this cost is based on For unmatched trajectories, a second-stage association is performed with objects that only have 3D detection frames, and a matrix is ​​also established. The cost is based on The distance from the appearance feature is preferably 0.5. For the track that has not been successfully matched, the third stage association is performed with the object with only a 2D detection box, and a matrix is ​​also established. The cost is based on The preferred threshold for the distance to the appearance feature is also 0.5.

[0162] When the object detection in the current frame fails to successfully match the trajectory, it is considered that it has not appeared in the past frame, a new trajectory is created for it, and its current appearance features are stored in the trajectory management module.

[0163] When the trajectory does not match the current frame detection, if the number of lost frames is greater than 30 frames, the trajectory is deleted and cleared in the trajectory management module.

[0164] The survival trajectory of the current frame is output as the 3D multi-target tracking result of the current frame.

[0165] Based on the technical solutions provided by the above embodiments, a 3D multi-target tracking method based on multimodal collaboration can be applied to fields including autonomous driving, robots, intelligent transportation and virtual reality. In the field of autonomous driving, 3D multi-target tracking technology can realize real-time perception and prediction of surrounding vehicles, pedestrians and other traffic elements in three-dimensional space to ensure the safety and stability of autonomous driving vehicles. In the field of robotics, 3D multi-target tracking technology can establish the motion trajectory of the operating target in the workspace, thereby improving the accuracy and efficiency of the robot operation. The 3D multi-target tracking method based on multimodal collaboration of the present invention uses 2D detection results to provide important guidance for optimizing 3D detection results, and uses the potential benefits brought by 2D detection algorithms to enhance the detection effect; at the same time, the motion trajectory of the target in 2D and 3D space is established by combining 2D detection results, so that the motion prediction error is reduced; in addition, the method of the present invention not only uses the similarity index in 3D space to measure the spatial similarity of objects between frames, but also comprehensively considers the 2D spatial similarity, reduces the confusion of spatial similarity in dense scenes, thereby guiding the accurate association between frames and improving the 3D multi-target tracking effect.

[0166] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0167] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A 3D multi-target tracking method based on multimodal collaboration, characterized in that: The method comprises the following steps: Input the currently acquired 3D point cloud sequence into the 3D object detector and the image sequence into the 2D object detector to obtain the 3D detection frame and 2D detection frame respectively; The 3D detection frame and the 2D detection frame results are input into a multimodal position optimizer, and the 3D detection frame of the corresponding object is adjusted using the 2D detection frame based on the image semantic information to obtain an enhanced 3D coordinate frame; Projecting the enhanced 3D coordinate frame onto the corresponding camera plane to obtain a corresponding area, and using an appearance feature extractor to obtain a current appearance feature corresponding to the object; The trajectory management module is used to store and update the appearance features and motion features, specifically: Performing similarity matching on the current appearance features, the enhanced 3D coordinate frame, and the historical trajectory appearance features and motion features extracted from the trajectory management module; The appearance features and motion features of the successfully matched enhanced 3D coordinate frame are updated, the unmatched enhanced 3D coordinate frame is initialized as a new track, and the unmatched historical track is judged for the number of lost frames. If it is greater than the maximum number of lost frames, the object corresponding to the historical track disappears and the historical track is removed; After the update is completed, the trajectory management module outputs the 3D multi-target motion tracking result of the current frame; The enhanced 3D coordinate frame is obtained, which specifically includes the following steps: Associating the 3D detection frame with the 2D detection frame using an intersection-and-union ratio in a 2D space to obtain an object having both the 2D detection frame and the 3D detection frame; For the object having both the 2D detection frame and the 3D detection frame, using a multimodal position optimizer to enhance the accuracy of the 3D detection frame, to obtain the enhanced 3D coordinate frame; The multimodal position optimizer uses the Newton optimization method to construct a weighted distance optimization objective function in the 2D space and the 3D space to find a 3D space position, which is the enhanced 3D coordinate frame; The current appearance feature, the enhanced 3D coordinate frame, and the historical trajectory appearance feature and motion feature extracted from the trajectory management module are matched by similarity, specifically including: Associating the 3D detection frame with the 2D detection frame using an intersection-and-union ratio in a 2D space to obtain an object having both the 2D detection frame and the 3D detection frame, an object having only the 3D detection frame, and an object having only the 2D detection frame; For objects with both the 2D detection frame and the 3D detection frame, the 3D coordinate frame will be enhanced , 2D detection box Perform cross-modal motion cost matrix calculations with the Kalman filter prediction box respectively: ， ， ， in, Represents the i-th coordinate frame in the enhanced 3D coordinate frame set, represents the jth prediction box in the 3D Kalman filter prediction box set, represents the i-th detection box in the 2D detection box set, represents the jth projection frame in the projection frame set of the 3D Kalman filter prediction frame set on the 2D image plane, , Represent the calculation operations of volume and area respectively, , Represent the convex hull calculation operations of volume and area respectively, , They are and The corresponding weight of Calculate the appearance features of the i-th historical trajectory and the current appearance features of the jth object Similarity ; pass and Construct the cost matrix C, the elements of the cost matrix C The expression is: ， in, Indicates different values ​​of i and j , Represents the weights of motion features and appearance features in the cost matrix; According to the constructed cost matrix C, Hungarian matching is used to associate the object detection of the current frame with the survival trajectory of the previous frame in the first stage, where the survival trajectory of the previous frame is the historical trajectory that has not been removed in the previous frame; For objects with only 3D detection boxes, they are matched with unmatched tracks in the first stage of association, including: The 3D coordinate frame will be enhanced The Kalman filter prediction box performs Calculation of motion cost matrix; The ROI area obtained by projecting the 3D detection frame onto the 2D image plane is input into the appearance feature extraction module to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ; pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the second stage; For objects with only 2D detection boxes, they are matched with unmatched tracks in the second stage association, including: The 2D detection box The projection box of the Kalman filter prediction box on the 2D image plane is performed based on Calculation of motion cost matrix; The area corresponding to the 2D detection frame is input into the appearance feature extraction module as the ROI area to obtain the appearance feature vector of the object in the current frame , and calculate the appearance feature vector with the historical trajectory Similarity ; pass Similarity to appearance Constructing the cost matrix , and use Hungarian matching to associate the object detection in the current frame with the survival trajectory in the previous frame in the third stage.

2. The 3D multi-target tracking method based on multimodal collaboration according to claim 1, characterized in that: The 3D object detector adopts the LargeKernel-3D detection network, and the 2D object detector adopts the YoloX detection network.

3. The 3D multi-target tracking method based on multimodal collaboration according to claim 2, characterized in that: The YoloX detection network has a regression branch and a classification branch, and the specific expression of the regression branch loss is: ； The specific expression of the classification branch loss is: ; The total loss function expression of the 2D target detector is: ; in, represents the intersection-over-union loss, represents the intersection-over-union ratio, A represents the detection box output by the 2D target detector, and B represents the object label box. is the multi-classification cross entropy loss, used as the loss function for multi-classification; is the label of the i-th object, Represents the number of object categories; is the probability that the object category is the i-th object, , are all hyperparameters, represents the loss of the regression branch, represents the loss of the classification branch.

4. The 3D multi-target tracking method based on multimodal collaboration according to claim 3 is characterized in that: The LargeKernel-3D detection network has the regression branch, the classification branch and the speed prediction branch. The specific expression of the speed prediction branch loss is: , the total loss function expression of the 3D target detector is: ,in, It represents the difference between the predicted speed and the actual speed. , The sum of the differences between the predicted speed in a direction and the corresponding actual speed; represents the loss function used for regression task network training, Otherwise, it is not satisfied Under this condition, , , is a hyperparameter, represents the loss of speed prediction branch.

5. The 3D multi-target tracking method based on multi-modal collaboration according to claim 4 is characterized in that: Obtaining 3D frame-by-frame detection results and 2D frame-by-frame detection results respectively includes the following steps: For 2D object detectors and 3D object detectors, the SimOTA label matching strategy is used to obtain positive and negative samples for input sequence detection, where the positive sample is the anchor point that matches the label. The error between the positive sample and the corresponding label is calculated according to the total loss function of the 2D object detector or the total loss function of the 3D object detector, and the parameters are updated through the back propagation process to reduce the error. Finally, the 3D detection boxes of all objects in the 3D point cloud sequence and the 2D detection boxes of all objects in the image sequence are obtained.

6. The 3D multi-target tracking method based on multimodal collaboration according to claim 1, characterized in that: The multimodal position optimizer uses the Newton optimization method to construct a weighted distance optimization objective function in 2D space and 3D space to find a 3D space position, which is the enhanced 3D coordinate frame, specifically including: Newton gradient descent method is used for optimization, and the objective function is as follows: ， in, represents the projection function of the 3D detection box to the 2D image plane, Indicates the center point of the optimized 3D detection box , respectively axis, axis, Axis coordinates, Represents the center point of the 2D detection box of the object, Indicates the center point of the 3D detection box of the object, represents the square of the Euclidean distance, , They represent the weights of the distance between the center point of the 2D detection box and the center point of the 3D detection box, respectively, and are obtained by the following formula: ， in, Represents the confidence score of the detection box, , The inputs correspond to the confidence scores of the 2D detection box and the 3D detection box respectively. , is a hyperparameter; In the During round optimization, the optimization formula is as follows: ， in represents the objective function, is the kth round of optimization update of x, yes The first derivative of yes The Hessian matrix of is calculated as follows: ， ， When the difference between the two iteration results is less than the set threshold, the iteration is terminated and the final enhanced 3D coordinate frame is obtained. , Indicates the optimized 3D detection frame, that is, the center point coordinates of the enhanced 3D coordinate frame. Indicates the width, length, height, and orientation angle of the enhanced 3D coordinate box.

7. The 3D multi-target tracking method based on multi-modal collaboration according to claim 1, characterized in that: The method includes predicting the motion state of the previous frame trajectory, i.e., the historical trajectory, and specifically includes the following steps: A Kalman filter is established for each trajectory, where the state vector Set to: ， in, Indicates the optimized 3D detection frame, that is, the center point coordinates of the enhanced 3D coordinate frame. Indicates the width, length, height and orientation angle of the enhanced 3D coordinate box. , They are the center point of the object coordinate frame. direction, The speed of the direction, , They are the center point of the object coordinate frame. direction, The acceleration in the direction, Indicates the angular velocity of the current trajectory; Based on the assumption of uniformly accelerated linear motion, the corresponding state transfer matrix is ​​obtained , and predict the current frame position based on the following equation , Indicates the previous frame position; After similarity matching between the motion state prediction value of the previous frame trajectory and the enhanced 3D coordinate frame of the current frame, the Kalman filter is updated using multimodal observations, including: Set the observation vector for: ， Where (u, v) represents the 2D coordinate center point of the object corresponding to the observation vector. When the corresponding object only has a 3D detection frame, the projection of the center point of the 3D detection frame on the image plane is used as its 2D coordinate center point. Calculate predicted values With observed value Error ,in is the measurement matrix of the current frame: ， Then, the updated state vector is obtained by With the covariance matrix : ， ， ， ， in, is the current frame covariance matrix, is the covariance matrix of the previous frame, yes The transposed matrix of is the process noise matrix, is the measurement noise, is the Kalman gain, is the identity matrix.

8. The 3D multi-target tracking method based on multimodal collaboration according to claim 1, characterized in that: The appearance feature extractor is based on the transformer network. When tracking the current frame, The frames are used as training frames to train the transformer network, thereby achieving domain adaptation of appearance feature extraction.

9. The 3D multi-target tracking method based on multi-modal collaboration according to claim 7, characterized in that: For the trajectories successfully matched in the first-stage association, the second-stage association, and the third-stage association, trajectory update is performed, i.e., Kalman filter update and appearance feature update; If the current frame object detection fails to successfully match the trajectory, it is considered that the current frame object has not appeared in the past frame, a new trajectory is created for it, and its current appearance features are stored in the trajectory management module; If the track does not match the object in the current frame, and the number of lost frames of the track is greater than the threshold number of frames, the track is deleted and cleared in the track management module; The survival trajectory of the current frame is output as the 3D multi-target tracking result of the current frame.

Images