Joint target detection and tracking method based on spatial-temporal feature aggregation of multi-sensor data

Abstract

The invention belongs to the technical field of artificial intelligence and computer vision, and relates to a combined target detection and tracking method based on spatial-temporal feature aggregation of multi-sensor data, which comprises the following steps: (1) constructing an experimental data set, (2) carrying out data preprocessing to generate false point cloud data, and (3) carrying out target tracking; (3) constructing a combined target detection and tracking network based on multi-sensor data spatial-temporal feature aggregation, (4) training the combined target detection and tracking network based on multi-sensor data spatial-temporal feature aggregation, and (5) visualizing a vehicle target tracking result. The method has the advantages that detection and tracking of vehicle motion in a motion scene are realized, three-dimensional evaluation indexes and two-dimensional evaluation indexes are used for evaluation, the three-dimensional evaluation indexes comprise tracking accuracy and tracking precision, the two-dimensional evaluation indexes comprise high-order tracking precision, a detection precision score and a correlation precision score, and the evaluation accuracy is improved. The system can well adapt to an unmanned motion environment.

Description

Technical Field

[0001] The invention relates to a joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data, belonging to the technical field of artificial intelligence and computer vision. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, intelligent driverless cars equipped with assistance systems have gradually become an important development direction in the future automotive field. Faced with complex and changing traffic environments, driverless cars need to use environmental perception assistance systems to continuously improve their ability to perceive the environment outside the car. The environmental perception assistance system uses cameras, lidar and other sensors to collect data during the driving process of intelligent driverless cars, and then detects and tracks targets to understand the surrounding environment information of the driverless vehicle, and ultimately achieve goals such as assisted risk avoidance and adaptive cruise control.

[0003] The advanced decision-making of the environmental perception assistance system mainly depends on environmental perception technology, among which target detection and tracking are important technologies that determine the efficiency and robustness of environmental perception. The target detection module determines the number and bounding box of the target of interest by fusing the color image and laser point cloud data collected by the sensor, using deep learning, image processing and other technologies, and inputs the detection results into the target tracking module. The goal of the target tracking module is to determine the position relationship of the tracked object in continuous frames to obtain the motion state and complete motion trajectory of the target object.

[0004] Each data acquisition method has its own unique advantages and limitations. For example, LiDAR can provide high-precision distance measurement, while color cameras can capture rich visual details. In order to fully utilize the advantages of various sensors and make up for their shortcomings, multi-sensor data fusion has become a core research direction in the field of multi-target detection and tracking, with the goal of obtaining more comprehensive and accurate target information and improving the accuracy and robustness of detection and tracking. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data. The method of the present invention adopts a deep learning model, uses color image guidance, completes the laser point cloud into a pseudo point cloud through depth completion, and extracts and fuses multi-scale features of the laser point cloud and the pseudo point cloud respectively, generating RoI features and multimodal fusion RoI features. Secondly, the BiGRU network is introduced to aggregate the features in time sequence to train a joint detection association network to obtain the detection target and its appearance correlation. Thirdly, the interactive multi-model Kalman filter is introduced to make full use of the advantages of the constant velocity model (CV model) and the constant acceleration model (CA model), better adapt to the uncertainty of vehicle speed changes in dynamic scenes, and improve the robustness of the system. Finally, the correlation matrix is ​​calculated using appearance information and motion information, and the matching problem in different situations is solved based on the multi-level association algorithm, which effectively eliminates the interference caused by false detection and realizes correct multi-target tracking.

[0006] In order to achieve the above-mentioned invention purpose and solve the problems existing in the prior art, the technical solution adopted by the present invention is A joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data comprises the following steps: Step 1: Construct an experimental dataset. Use the color image data and laser point cloud data in the KITTI public dataset to construct an experimental dataset, including 3999 frames for training, 4005 frames for validation, and 11095 frames for testing. In the laser point cloud coordinate system, the first frame in a frame of laser point cloud data is The laser points are , whose homogeneous coordinates are expressed as ; Step 2: Perform data preprocessing to generate pseudo point cloud data, which specifically includes the following sub-steps: (a) Record the first The laser points are projected into the camera coordinate system to obtain the projection point , whose homogeneous coordinates are expressed as , the pixel point obtained in the pixel coordinate system , the width and height of the camera CCD photosensitive element are and , the depth value corresponding to each pixel is defined as , projection coordinate transformation, through the formula and Describe,

[0007]

[0008] in, is the internal parameter matrix of the camera, obtained by calibrating the color camera, are the rotation matrix and translation vector from the laser radar coordinate system to the camera coordinate system, respectively. or or or When the laser point There is no corresponding projection point in the pixel coordinate system, and no subsequent operations are performed on this point. and , then the projection point and its depth value are used together to generate the corresponding sparse depth map; (b) The color image data and the sparse depth map are used as input data, and the hyperbolic extrapolation network is used to complete the depth to generate a dense depth map. The dense depth map and the pixels in the color image are then projected into the laser point cloud coordinate system to generate a pseudo point cloud. The first pixel in a frame of the pseudo point cloud is recorded. The pseudo point cloud is ,in, is the three-dimensional coordinate of the pseudo point cloud in the laser point cloud coordinate system, ( ) is the RGB information of the corresponding pixel in the color image, ( ) represents the two-dimensional coordinates in the color image, and the three-dimensional coordinate transformation of the pseudo point cloud is described by formulas (3), (4) and (5).

[0009]

[0010]

[0011] in,( ) is the center point coordinate of the color image, ( ) is the focal length of the camera in the horizontal and vertical directions, ( ) represent the horizontal and vertical offsets respectively; Step 3: Construct a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data, including a sparse dense fusion backbone network based on laser point cloud and pseudo point cloud and a joint detection and association network based on spatiotemporal feature aggregation of multi-sensor. The sparse dense fusion backbone network based on laser point cloud and pseudo point cloud is used to extract laser point cloud RoI features and pseudo point cloud RoI features respectively, and weighted fusion of the two features is used to obtain multi-sensor fusion RoI features. The joint detection and association network based on spatiotemporal feature aggregation of multi-sensor mainly includes an RoI optimization detection network, a spatiotemporal feature aggregation network and an association network. Finally, tracking is achieved using an interactive multi-model Kalman filter and a multi-level association algorithm, which specifically includes the following sub-steps: (a) Laser point cloud feature extraction. The laser point cloud data of 5 consecutive frames are used as input to perform laser point cloud feature extraction. The laser point cloud feature extraction includes five parts: point cloud voxel feature encoding, 3D spatial feature extraction network, 2D bird's-eye view feature extraction network, region candidate network and RoI pooling. Since point cloud data has the characteristics of irregularity and high dimensionality, in order to reduce the amount of calculation and improve the calculation efficiency, the point cloud data is firstly encoded with point cloud voxel features, and the voxel size is set to , since the coordinate ranges of the point cloud data used are , and , then the entire three-dimensional point cloud space is generated voxels, load the point cloud data into the corresponding voxels, complete the point cloud grouping, and set the maximum number of points in each voxel block , for the number of points exceeding The voxels of the point cloud are randomly downsampled; after preprocessing, all point cloud voxels are processed by average encoding to obtain the point cloud voxel encoding features , the dimension is , the voxel features The input 3D spatial feature extraction network consists of six convolution modules: ConvIn, Conv1-Conv4, and ConvOut. Each convolution block consists of sparse convolution layers (SpConv) and sub-stream sparse convolution layers (SubSpConv) with different numbers of layers. The voxels are downsampled eightfold to obtain the features. , voxel features The final output feature of the three-dimensional spatial feature extraction network is ; Voxel features along Axis stacking to obtain compressed 2D bird's-eye view features , the feature dimension is , input to the 2D bird's-eye view feature extraction network, the entire network is divided into two branches, including two downsampling layers and two upsampling layers . We design two-dimensional omnidirectional dynamic convolution (ODConv2D) and two-dimensional transposed omnidirectional dynamic convolution (ODTransposeConv2D), increase the attention to the input channel dimension, output channel dimension and convolution kernel space dimension, and apply them to the downsampling layer and upsampling layer respectively. Each downsampling layer includes a zero padding layer (Zeropad) and five ODConv2D, and each upsampling layer includes an ODTransposeConv2D. Branch one consists of a downsampling layer and upsampling layers constitute, will Input to the downsampling layer Get downsampled features , the dimension is ,Will Input to the upsampling layer , get the upsampled features , the dimension is ,Will Input to the downsampling layer Downsample again to get the downsampled features , the dimension is , and then through the upsampling layer Upsample to get the dimension ,Will and Cascade to get the 2D bird's-eye view feature , the dimension is , the region proposal network is used to extract the candidate box of the target from the feature map; the two-dimensional bird's-eye view feature Treated as 256 channels The feature image is input into the region candidate network. Since the region feature network is implemented based on the anchor frame, two RoI region anchor frames are generated for each pixel of the feature image, corresponding to and In two directions, the entire feature map contains anchor boxes The region candidate network contains three convolution branches: category classification branch, location regression branch and direction classification branch, using convolution kernel sizes of The one-dimensional omnidirectional dynamic convolution (ODConv1D) is used for prediction to obtain category prediction features, bounding box parameter prediction features, and direction classification prediction features, with dimensions of ; For the candidate RoI regions obtained by the region candidate network, the non-maximum suppression algorithm is used to screen out the candidate RoI regions with high confidence and consistent with the actual target object, and each RoI region is divided into The grid is input into the RoI grid pooling layer to obtain the RoI grid features of the laser point cloud , the dimension is ; (b) Pseudo point cloud feature extraction: use the laser point cloud RoI area to perform RoI cropping on the pseudo point cloud, and then extract features from the cropped pseudo point cloud in the RoI area. The pseudo point cloud feature extraction includes three parts: point feature extraction network, RoI pooling, and voxel feature extraction network. Pseudo-point The characteristics are ,in, is a three-dimensional geometric feature, is the color information feature, for pseudo points Perform neighborhood point search and set the number of neighborhood points to For pseudo points with less than 9 neighboring points, use pseudo points The feature information of the point is completed to ensure that the number of neighborhood points of each pseudo point is 9. The point feature extraction network consists of three color point convolution layers. Each color point convolution layer contains a feature extraction FEA layer, a weight calculation WGT layer and a feature aggregation FUS layer. The feature extraction layer FEA layer is a fully connected layer, which represents the features of all pseudo points as a dimension of The tensor is used as the input of the feature extraction FEA layer, where is the number of pseudo points, for three color point convolution layers, They are 6, 12 and 24 respectively. The output features are obtained through the fully connected layer , the dimension is , for the three color point convolution layers, The weight calculation layer WGT layer is a fully connected layer. The weight calculation layer WGT layer uses the position residuals of pseudo points and neighboring pseudo points to calculate weights, build the relationship between each pseudo point and its neighboring points, and convert the pseudo points The position is represented by ,in, is a three-dimensional geometric feature, is a two-dimensional information feature, the position residual of all pseudo points and their neighboring points Represented as a dimension A tensor of is the number of pseudo points in the pseudo point cloud, which is used as the input of the weight layer WGT layer to obtain the neighborhood point weight , the dimension is ; For neighborhood point features and neighborhood weights , multiply each neighborhood point feature by its corresponding weight to obtain the weighted neighborhood point feature, expressed as a dimension of The tensor of , which is cascaded to get a dimension of The two-dimensional tensor of is used as the input of the feature aggregation FUS layer to obtain the output feature dimension , the dimension is , and use it as the input of the next layer of color point convolution feature extraction FEA layer; voxelize the pseudo point cloud RoI feature area and divide each RoI area into Each voxel corresponds to an output feature. After the feature is upgraded using a multi-layer perceptron, the maximum pooling is performed to splice the features of all voxels together to generate a pseudo point cloud RoI grid feature. , the dimension is ; The pseudo point cloud voxel feature extraction network consists of a sparse convolutional layer and two sub-flow sparse convolutional layers. Perform voxel feature extraction and finally output features The dimension is ; (c) Construct the laser point cloud and pseudo point cloud RoI grid feature fusion network, and respectively integrate the laser point cloud RoI grid feature and pseudo point cloud RoI grid features Input to the fully connected layer and , the output features are cascaded and input into the fully connected layer FC3 to obtain the grid feature weights and ; respectively and Input 1D convolutional block and After feature extraction, it is multiplied point by point with the corresponding grid feature weight to obtain the weighted feature, and the weighted feature is cascaded and input into the one-dimensional convolution block. The fused RoI grid feature is , the dimension is ; (d) Classification of candidate RoI region features and bounding box improvement. The RoI optimization detection network is used to classify the candidate RoI region features and improve their bounding boxes. First, the fully connected layer is used to integrate the RoI grid features. The feature channel dimension of Reduce to dimensional, and then the reduced-dimensional RoI grid feature features The input is sent to the RoI optimization detection network, which consists of two fully connected layer branches, one for category classification and one for position regression branch tasks, and the output feature dimensions are and ; (e) Construct a spatiotemporal feature aggregation network, using a bidirectional gated recurrent unit (BiGRU) to enhance features in the time dimension, and further aggregate the RoI features of the same target at consecutive moments, thereby strengthening the relationship between them and realizing the comprehensive utilization of the spatiotemporal dimension of features; the gated recurrent unit determines how much previous memory to retain at the current moment by controlling the reset gate and the update gate, and combines the current input information to update and predict the state. The bidirectional gated recurrent unit consists of two gated recurrent units in opposite directions. Considering the bidirectional information of the historical state of the object, the forward and backward input sequences are calculated respectively to obtain the corresponding forward and backward hidden states, and then these hidden states are weighted and combined according to specific weights to obtain the hidden state at the current moment; the input of the spatiotemporal feature aggregation network is the RoI feature at the current moment and the spatiotemporal aggregation feature at the previous moment, and the output is the spatiotemporal aggregation feature at the current moment. At time 𝑖̂, the RoI features of the same target object are aggregated, and the spatiotemporal aggregation feature calculation of each target is described by formula (6), (6) in, , Indicates that the target is RoI features at each moment, Indicates that the target is The spatiotemporal aggregation characteristics of each moment, Initialized to 0; (f) Construct an association network and design an association network to calculate the association between targets at two consecutive moments. Its input is the RoI feature at moment 𝑡. and Spatiotemporal aggregation characteristics of moments , after copying and expanding the input features, subtract them point by point to get the absolute value, and obtain the associated network input matrix , described by formula (7), (7) in, for The number of RoI targets at the moment, for The number of RoI targets at the moment, for Moment The RoI features of the target, for Moment The spatiotemporal aggregation features of the target are input into the associated network matrix After being passed into the fully connected layer FC, the Softmax operation is performed on the rows and columns of the output matrix of the fully connected layer respectively, and then the average value of the two is taken to obtain the appearance correlation matrix; (g) Target motion state prediction: The interactive multi-model Kalman filter is used to realize the target motion state prediction, which combines the constant velocity model (CV model) and the constant acceleration model (CA model), that is, To adapt to different sports modes, The motion state of a tracking target at time is described by formula (8):

[0012] in, represent Always track the center coordinates of the target 3D bounding box. Represents the heading angle of the 3D bounding box of the tracked target, ( ) represents the length, width and height of the three-dimensional bounding box of the tracking target, ( ) represent the tracking target edge The speed of the axis, ( ) represent the tracking target edge The acceleration of the axis is set in the CV model ( ) are both 0; the CV model and CA model are used to predict and update the target motion state respectively. After the prediction and update states of the two motion types are calculated, the correct probabilities of the two models are updated according to the likelihood, which is described by formula (9): in, Indicated in the model Generate observations The likelihood of Indicated in Before the update, the model is the prior probability of the correct model, Indicated in Moment Model is the probability of the correct model, The weighted prediction state is calculated using the probability that the model is correct as the weight, as The final motion state prediction result at time t is described by formula (10):

[0013] in, represents the weighted state prediction result, Indicated in the model Prediction results of the motion state; (h) To achieve target association, a multi-level association algorithm is proposed, and the Kuhn-Munkres algorithm is used to solve the assignment problem. At this moment, you need to Tracking target at all times and Detection target at any time Specifically, the following situations are involved: (a) For All detection targets at all times , according to its detection confidence, it is divided into high confidence detection targets and low confidence detection targets , set the confidence threshold =0.75, when the confidence level is greater than or equal to When , the confidence level is high, otherwise, the confidence level is low; (b) First-level association, using the Kuhn-Munkres algorithm to associate high-confidence detection targets Tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (c) Secondary association, using the Kuhn-Munkres algorithm to associate low-confidence detection targets Unmatched tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (d) For a tracking-detection target pair, the state of the detection target is used to update the state of the tracking target to obtain , used for matching at the next moment; (e) For unmatched tracking targets , it is considered as a temporarily disappeared target, and it is retained to continue the target state prediction until it is re-matched at a later time. If it is not re-matched for more than 50 frames, it is considered that the target has completely disappeared; (f) For unmatched detection targets with high confidence , considered as a newly appeared target, the bounding box and RoI features of the detected target are initialized as a new tracking target; (g) For unmatched detection targets with low confidence , which is considered to be the wrong detection target.

[0014] Step 4: Train a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data. During the training process, the Adam optimizer is used, where the first decay exponent and the second decay exponential Set to and ; The batch size during training is set to 2, and the training is set to 40 epochs.

[0015] Step 5: Visualization of vehicle target tracking results. After the network training is completed, the final vehicle target detection and tracking results are obtained, and the tracking results are visualized on the color image data and point cloud image data respectively.

[0016] The beneficial effects of the present invention are as follows: a joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data, comprising the following steps: (1) constructing an experimental data set, (2) performing data preprocessing to generate pseudo point cloud data, (3) constructing a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data, (4) training the joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data, and (5) visualizing vehicle target tracking results. The advantages of the present invention are that the detection and tracking of vehicle motion in a motion scene are realized, and three-dimensional evaluation indicators and two-dimensional evaluation indicators are used for evaluation, wherein the three-dimensional evaluation indicators include tracking accuracy (MOTA) and tracking precision (MOTP), and the two-dimensional evaluation indicators include high-order tracking accuracy (HOTA), detection accuracy score (DetA) and association accuracy score (AssA). For the training set data, the experimental results of the present method are that MOTA reaches 90.06%, MOTP reaches 82.36%, HOTA reaches 80.05%, DetA reaches 77.29%, and AssA reaches 83.12%; for the test set data, the experimental results of the present method are that MOTA reaches 88.47%, MOTP reaches 86.78%, HOTA reaches 79.68%, DetA reaches 76.87%, and AssA reaches 83.08%, and the inference speed reaches 237FPS on the test set, which can well adapt to the unmanned driving motion environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall network structure diagram of the present invention.

[0018] Figure 2 It is a diagram of the tracking implementation process of the present invention.

[0019] Figure 3 It is a schematic diagram of the three-dimensional space feature extraction network adopted by the present invention.

[0020] Figure 4 It is a schematic diagram of the two-dimensional bird's-eye view feature extraction network adopted by the present invention.

[0021] Figure 5This is a schematic diagram of the point feature extraction network used in the present invention.

[0022] Figure 6 This is a schematic diagram of the laser point cloud and pseudo point cloud RoI grid feature fusion network used in the present invention.

[0023] Figure 7 It is a flow chart of the multi-level association algorithm of the present invention.

[0024] Figure 8 This is a tracking result diagram of the KITTI car test data set used in the present invention in the point cloud space.

[0025] Fig. 9 This is a tracking result diagram of the KITTI car test data set used in the present invention in a color image.

[0026] Fig.10 This is a comparison diagram of the trajectory tracking results of the KITTI car test data set used in the present invention, the trajectory true value, and the AB3DMOT method.

[0027] Among them: (a) is the true value image of the trajectory of the KITTI test set 0002 sequence data, (b) is the trajectory image after detection and tracking of the 0002 sequence data using this method, (c) is the trajectory image of the 0002 sequence data using the AB3DMOT method, (d) is the true value image of the trajectory of the KITTI test set 0014 sequence data, (e) is the trajectory image after detection and tracking of the 0014 sequence data using this method, and (f) is the trajectory image of the 0014 sequence data using the AB3DMOT method.

[0028] Fig.11 It is a flow chart of the method steps of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings.

[0030] like Fig.11 As shown, a joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data includes the following steps: Step 1: Construct an experimental dataset. Use the color image data and laser point cloud data in the KITTI public dataset to construct an experimental dataset, including 3999 frames for training, 4005 frames for validation, and 11095 frames for testing. In the laser point cloud coordinate system, the first frame in a frame of laser point cloud data is The laser points are , whose homogeneous coordinates are expressed as ; Step 2: Perform data preprocessing to generate pseudo point cloud data, which specifically includes the following sub-steps: (a) Record the first The laser points are projected into the camera coordinate system to obtain the projection point , whose homogeneous coordinates are expressed as , the pixel point obtained in the pixel coordinate system , the width and height of the camera CCD photosensitive element are and , the depth value corresponding to each pixel is defined as , projection coordinate transformation, through the formula and Describe,

[0031]

[0032] in, is the internal parameter matrix of the camera, obtained by calibrating the color camera, are the rotation matrix and translation vector from the laser radar coordinate system to the camera coordinate system, respectively. or or or When the laser point There is no corresponding projection point in the pixel coordinate system, and no subsequent operations are performed on this point. and , then the projection point and its depth value are used together to generate the corresponding sparse depth map; (b) The color image data and the sparse depth map are used as input data, and the hyperbolic extrapolation network is used to complete the depth to generate a dense depth map. The dense depth map and the pixels in the color image are then projected into the laser point cloud coordinate system to generate a pseudo point cloud. The first pixel in a frame of the pseudo point cloud is recorded. The pseudo point cloud is ,in, is the three-dimensional coordinate of the pseudo point cloud in the laser point cloud coordinate system, ( ) is the RGB information of the corresponding pixel in the color image, ( ) represents the two-dimensional coordinates in the color image, and the three-dimensional coordinate transformation of the pseudo point cloud is described by formulas (3), (4) and (5).

[0033]

[0034]

[0035] in,( ) is the center point coordinate of the color image, ( ) is the focal length of the camera in the horizontal and vertical directions, ( ) represent the horizontal and vertical offsets respectively; Step 3, construct a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data, including a sparse dense fusion backbone network based on laser point cloud and pseudo point cloud and a joint detection and association network based on spatiotemporal feature aggregation of multi-sensor. The sparse dense fusion backbone network based on laser point cloud and pseudo point cloud is used to extract laser point cloud RoI features and pseudo point cloud RoI features respectively, and weighted fusion of the two features is used to obtain multi-sensor fusion RoI features. The joint detection and association network based on spatiotemporal feature aggregation of multi-sensor mainly includes a RoI optimization detection network, a spatiotemporal feature aggregation network and an association network. Finally, tracking is achieved using an interactive multi-model Kalman filter and a multi-level association algorithm. The overall network framework is as follows: Figure 1 As shown, track the implementation process, such as Figure 2 As shown, it specifically includes the following sub-steps: (a) Laser point cloud feature extraction. The laser point cloud data of 5 consecutive frames are used as input to perform laser point cloud feature extraction. The laser point cloud feature extraction includes five parts: point cloud voxel feature encoding, 3D spatial feature extraction network, 2D bird's-eye view feature extraction network, region candidate network and RoI pooling. Since point cloud data has the characteristics of irregularity and high dimensionality, in order to reduce the amount of calculation and improve the calculation efficiency, the point cloud data is firstly encoded with point cloud voxel features, and the voxel size is set to , since the coordinate ranges of the point cloud data used are , and , then the entire three-dimensional point cloud space is generated voxels, load the point cloud data into the corresponding voxels, complete the point cloud grouping, and set the maximum number of points in each voxel block , for the number of points exceeding The voxels of the point cloud are randomly downsampled; after preprocessing, all point cloud voxels are processed by average encoding to obtain the point cloud voxel encoding features , the dimension is , the voxel features Input the three-dimensional spatial feature extraction network, the network structure is as follows Figure 3 As shown in Figure 2, the entire network consists of six convolutional modules: ConvIn, Conv1-Conv4, and ConvOut. Each convolutional block consists of sparse convolutional layers (SpConv) and sub-stream sparse convolutional layers (SubSpConv) with different numbers of layers. The voxels are downsampled eightfold to obtain the features. , voxel features The final output feature of the three-dimensional spatial feature extraction network is ; Voxel features along Axis stacking to obtain compressed 2D bird's-eye view features , the feature dimension is , input to the two-dimensional bird's-eye view feature extraction network, the two-dimensional bird's-eye view feature extraction network structure is as follows Figure 4 As shown. The entire network is divided into two branches, including two downsampling layers and two upsampling layers . We design two-dimensional omnidirectional dynamic convolution (ODConv2D) and two-dimensional transposed omnidirectional dynamic convolution (ODTransposeConv2D), increase the attention to the input channel dimension, output channel dimension and convolution kernel space dimension, and apply them to the downsampling layer and upsampling layer respectively. Each downsampling layer includes a zero padding layer (Zeropad) and five ODConv2D, and each upsampling layer includes an ODTransposeConv2D. Branch one consists of a downsampling layer and upsampling layers constitute, will Input to the downsampling layer Get downsampled features , the dimension is ,Will Input to the upsampling layer , get the upsampled features , the dimension is ,Will Input to the downsampling layer Downsample again to get the downsampled features , the dimension is , and then through the upsampling layer Upsample to get the dimension ,Will and Cascade to get the 2D bird's-eye view feature , the dimension is , the region proposal network is used to extract the candidate box of the target from the feature map; the two-dimensional bird's-eye view feature Treated as 256 channels The feature image is input into the region candidate network. Since the region feature network is implemented based on the anchor frame, two RoI region anchor frames are generated for each pixel of the feature image, corresponding to and In two directions, the entire feature map contains anchor boxes The region candidate network contains three convolution branches: category classification branch, location regression branch and direction classification branch, using convolution kernel sizes of The one-dimensional omnidirectional dynamic convolution (ODConv1D) is used for prediction to obtain category prediction features, bounding box parameter prediction features, and direction classification prediction features, with dimensions of ; For the candidate RoI regions obtained by the region candidate network, the non-maximum suppression algorithm is used to screen out the candidate RoI regions with high confidence and consistent with the actual target object, and each RoI region is divided into The grid is input into the RoI grid pooling layer to obtain the RoI grid features of the laser point cloud , the dimension is ; (b) Pseudo point cloud feature extraction: use the laser point cloud RoI area to perform RoI cropping on the pseudo point cloud, and then extract features from the cropped pseudo point cloud in the RoI area. The pseudo point cloud feature extraction includes three parts: point feature extraction network, RoI pooling, and voxel feature extraction network. Pseudo-point The characteristics are ,in, is a three-dimensional geometric feature, is the color information feature, for pseudo points Perform neighborhood point search and set the number of neighborhood points to For pseudo points with less than 9 neighboring points, use pseudo points The feature information of the point is completed to ensure that the number of neighborhood points of each pseudo point is 9. The point feature extraction network consists of three color point convolution layers. The network structure is as follows: Figure 5 As shown in the figure, each color point convolution layer contains a feature extraction FEA layer, a weight calculation WGT layer and a feature aggregation FUS layer; the feature extraction layer FEA layer is a fully connected layer, which represents the features of all pseudo points as a dimension of The tensor is used as the input of the feature extraction FEA layer, where is the number of pseudo points, for three color point convolution layers, They are 6, 12 and 24 respectively. The output features are obtained through the fully connected layer , the dimension is , for the three color point convolution layers, The weight calculation layer WGT layer is a fully connected layer. The weight calculation layer WGT layer uses the position residuals of pseudo points and neighboring pseudo points to calculate weights, build the relationship between each pseudo point and its neighboring points, and convert the pseudo points The position is represented by ,in, is a three-dimensional geometric feature, is a two-dimensional information feature, the position residual of all pseudo points and their neighboring points Represented as a dimension A tensor of is the number of pseudo points in the pseudo point cloud, which is used as the input of the weight layer WGT layer to obtain the neighborhood point weight , the dimension is ; For neighborhood point features and neighborhood weights , multiply each neighborhood point feature by its corresponding weight to obtain the weighted neighborhood point feature, expressed as a dimension of The tensor of , which is cascaded to get a dimension of The two-dimensional tensor of is used as the input of the feature aggregation FUS layer to obtain the output feature dimension , the dimension is , and use it as the input of the next layer of color point convolution feature extraction FEA layer; voxelize the pseudo point cloud RoI feature area and divide each RoI area into Each voxel corresponds to an output feature. After the feature is upgraded using a multi-layer perceptron, the maximum pooling is performed to splice the features of all voxels together to generate a pseudo point cloud RoI grid feature. , the dimension is ; The pseudo point cloud voxel feature extraction network consists of a sparse convolution layer and two sub-flow sparse convolution layers. Perform voxel feature extraction and finally output features The dimension is ; (c) Construct a laser point cloud and pseudo point cloud RoI grid feature fusion network. The network structure is as follows: Figure 6 As shown, the laser point cloud RoI grid features are and pseudo point cloud RoI grid features Input to the fully connected layer and , the output features are cascaded and input into the fully connected layer FC3 to obtain the grid feature weights and ; respectively and Input 1D convolutional block and After feature extraction, it is multiplied point by point with the corresponding grid feature weight to obtain the weighted feature, and the weighted feature is cascaded and input into the one-dimensional convolution block. The fused RoI grid feature is , the dimension is ; (d) Classification of candidate RoI region features and bounding box improvement. The RoI optimization detection network is used to classify the candidate RoI region features and improve their bounding boxes. First, the fully connected layer is used to integrate the fused RoI grid features. The feature channel dimension of Reduce to dimensional, and then the reduced-dimensional RoI grid feature features The input is sent to the RoI optimization detection network, which consists of two fully connected layer branches, one for category classification and one for position regression branch tasks, and the output feature dimensions are and ; (e) Construct a spatiotemporal feature aggregation network, using a bidirectional gated recurrent unit (BiGRU) to enhance features in the time dimension, and further aggregate the RoI features of the same target at consecutive moments, thereby strengthening the relationship between them and realizing the comprehensive utilization of the spatiotemporal dimension of features; the gated recurrent unit determines how much previous memory to retain at the current moment by controlling the reset gate and the update gate, and combines the current input information to update and predict the state. The bidirectional gated recurrent unit consists of two gated recurrent units in opposite directions. Considering the bidirectional information of the historical state of the object, the forward and backward input sequences are calculated respectively to obtain the corresponding forward and backward hidden states, and then these hidden states are weighted and combined according to specific weights to obtain the hidden state at the current moment; the input of the spatiotemporal feature aggregation network is the RoI feature at the current moment and the spatiotemporal aggregation feature at the previous moment, and the output is the spatiotemporal aggregation feature at the current moment. At time 𝑖̂, the RoI features of the same target object are aggregated, and the spatiotemporal aggregation feature calculation of each target is described by formula (6), (6) in, , Indicates that the target is RoI features at each moment, Indicates that the target is The spatiotemporal aggregation characteristics of each moment, Initialized to 0; (f) Construct an association network and design an association network to calculate the association between targets at two consecutive moments. Its input is the RoI feature at moment 𝑡. and Spatiotemporal aggregation characteristics of moments , after copying and expanding the input features, subtract them point by point to get the absolute value, and obtain the associated network input matrix , described by formula (7), (7) in, for The number of RoI targets at the moment, for The number of RoI targets at the moment, for Moment The RoI features of the target, for Moment The spatiotemporal aggregation features of the target are input into the associated network matrix After being passed into the fully connected layer FC, the Softmax operation is performed on the rows and columns of the output matrix of the fully connected layer respectively, and then the average value of the two is taken to obtain the appearance correlation matrix; (g) Target motion state prediction: The interactive multi-model Kalman filter is used to realize the target motion state prediction, which combines the constant velocity model (CV model) and the constant acceleration model (CA model), that is, To adapt to different sports modes, The motion state of a tracking target at time is described by formula (8):

[0036] in, represent Always track the center coordinates of the target 3D bounding box. Represents the heading angle of the 3D bounding box of the tracked target, ( ) represents the length, width and height of the three-dimensional bounding box of the tracking target, ( ) represent the tracking target edge The speed of the axis, ( ) represent the tracking target edge The acceleration of the axis is set in the CV model ( ) are both 0; the CV model and CA model are used to predict and update the target motion state respectively. After the prediction and update states of the two motion types are calculated, the correct probabilities of the two models are updated according to the likelihood, which is described by formula (9): in, Indicated in the model Generate observations The likelihood of Indicated in Before the update, the model is the prior probability of the correct model, Indicated in Moment Model is the probability of the correct model, The weighted prediction state is calculated using the probability that the model is correct as the weight, as The final motion state prediction result at time t is described by formula (10):

[0037] in, represents the weighted state prediction result, Indicated in the model Prediction results of the motion state; (h) To achieve target association, a multi-level association algorithm is proposed, and the Kuhn-Munkres algorithm is used to solve the assignment problem. At this moment, you need to Tracking target at all times and Detection target at any time The multi-level association algorithm process is as follows: Figure 7 As shown, it specifically includes the following situations: (1) For All detection targets at all times , according to its detection confidence, it is divided into high confidence detection targets and low confidence detection targets , set the confidence threshold =0.75, when the confidence level is greater than or equal to When , the confidence level is high, otherwise, the confidence level is low; (2) First-level association: using the Kuhn-Munkres algorithm to associate high-confidence detection targets Tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (3) Secondary association: using the Kuhn-Munkres algorithm to associate low-confidence detection targets Unmatched tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (4) For the tracking-detection target matching pair, the state of the detection target is used to update the state of the tracking target to obtain , used for matching at the next moment; (5) For unmatched tracking targets , it is considered as a temporarily disappeared target, and it is retained to continue the target state prediction until it is re-matched at a later time. If it is not re-matched for more than 50 frames, it is considered that the target has completely disappeared; (6) For unmatched detection targets with high confidence , considered as a newly appeared target, the bounding box and RoI features of the detected target are initialized as a new tracking target; (7) For unmatched detection targets with low confidence , which is considered to be the wrong detection target.

[0038] Step 4: Train a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data. During the training process, the Adam optimizer is used, where the first decay exponent and the second decay exponential Set to and ; The batch size during training is set to 2, and the training is set to 40 epochs.

[0039] Step 5: Visualization of vehicle target tracking results. After the network training is completed, the final vehicle target detection and tracking results are obtained, and the tracking results are visualized on the color image data and the point cloud image data, respectively, to obtain the tracking results of this method in the point cloud space and the color image, as shown in Figure 5. Figure 8 and Fig. 9 As shown in Figure 2, the tracking trajectory of this method is compared with the true value trajectory and the milestone tracking method AB3DMOT. The result is shown in Figure 2. Fig.10 shown.

Claims

1. A joint target detection and tracking method based on spatiotemporal feature aggregation of multi-sensor data, characterized in that The following steps are involved: Step 1: Construct an experimental dataset. Use the color image data and laser point cloud data in the KITTI public dataset to construct an experimental dataset, including 3999 frames for training, 4005 frames for validation, and 11095 frames for testing. In the laser point cloud coordinate system, the first frame in a frame of laser point cloud data is The laser points are , whose homogeneous coordinates are expressed as ; Step 2: Perform data preprocessing to generate pseudo point cloud data, which specifically includes the following sub-steps: (a) Record the first The laser points are projected into the camera coordinate system to obtain the projection point , whose homogeneous coordinates are expressed as , the pixel point obtained in the pixel coordinate system , the width and height of the camera CCD photosensitive element are and , the depth value corresponding to each pixel is defined as , projection coordinate transformation, through the formula and Describe, ； ； in, is the internal parameter matrix of the camera, obtained by calibrating the color camera, are the rotation matrix and translation vector from the laser radar coordinate system to the camera coordinate system, respectively. or or or When the laser point There is no corresponding projection point in the pixel coordinate system, and no subsequent operations are performed on this point. and , then the projection point and its depth value are used together to generate the corresponding sparse depth map; (b) The color image data and the sparse depth map are used as input data, and the hyperbolic extrapolation network is used to complete the depth to generate a dense depth map. The dense depth map and the pixels in the color image are then projected into the laser point cloud coordinate system to generate a pseudo point cloud. The first pixel in a frame of the pseudo point cloud is recorded. The pseudo point cloud is ,in, is the three-dimensional coordinate of the pseudo point cloud in the laser point cloud coordinate system, ( ) is the RGB information of the corresponding pixel in the color image, ( ) represents the two-dimensional coordinates in the color image, and the three-dimensional coordinate transformation of the pseudo point cloud is described by formulas (3), (4) and (5). ； ； ； in,( ) is the center point coordinate of the color image, ( ) is the focal length of the camera in the horizontal and vertical directions, ( ) represent the horizontal and vertical offsets respectively; Step 3: Build a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data. It includes a sparse dense fusion backbone network based on laser point cloud and pseudo point cloud and a joint detection and association network based on multi-sensor spatiotemporal feature aggregation. The sparse dense fusion backbone network based on laser point cloud and pseudo point cloud is used to extract laser point cloud RoI features and pseudo point cloud RoI features respectively, and weighted fusion of the two features obtains multi-sensor fusion RoI features. The joint detection and association network based on multi-sensor spatiotemporal feature aggregation mainly includes a RoI optimization detection network, a spatiotemporal feature aggregation network and an association network. Finally, tracking is achieved using an interactive multi-model Kalman filter and a multi-level association algorithm, which specifically includes the following sub-steps: (a) Laser point cloud feature extraction. The laser point cloud data of 5 consecutive frames are used as input to perform laser point cloud feature extraction. The laser point cloud feature extraction includes five parts: point cloud voxel feature encoding, 3D spatial feature extraction network, 2D bird's-eye view feature extraction network, region candidate network and RoI pooling. Since point cloud data has the characteristics of irregularity and high dimensionality, in order to reduce the amount of calculation and improve the calculation efficiency, the point cloud data is firstly encoded with point cloud voxel features, and the voxel size is set to , since the coordinate ranges of the point cloud data used are , and , then the entire three-dimensional point cloud space is generated voxels, load the point cloud data into the corresponding voxels, complete the point cloud grouping, and set the maximum number of points in each voxel block , for the number of points exceeding The voxels of the point cloud are randomly downsampled; after preprocessing, all point cloud voxels are processed by average encoding to obtain the point cloud voxel encoding features , the dimension is , the voxel features The input 3D spatial feature extraction network consists of six convolution modules: ConvIn, Conv1-Conv4, and ConvOut. Each convolution block consists of sparse convolution layers (SpConv) and sub-stream sparse convolution layers (SubSpConv) with different numbers of layers. The voxels are downsampled eightfold to obtain the features. , voxel features The final output feature of the three-dimensional spatial feature extraction network is ; Voxel features along Axis stacking to obtain compressed 2D bird's-eye view features , the feature dimension is , input to the 2D bird's-eye view feature extraction network, the entire network is divided into two branches, including two downsampling layers and two upsampling layers . We design two-dimensional omnidirectional dynamic convolution (ODConv2D) and two-dimensional transposed omnidirectional dynamic convolution (ODTransposeConv2D), increase the attention to the input channel dimension, output channel dimension and convolution kernel space dimension, and apply them to the downsampling layer and upsampling layer respectively. Each downsampling layer includes a zero padding layer (Zeropad) and five ODConv2D, and each upsampling layer includes an ODTransposeConv2D. Branch one consists of a downsampling layer and upsampling layers constitute, will Input to the downsampling layer Get downsampled features , the dimension is ,Will Input to the upsampling layer , get the upsampled features , the dimension is ,Will Input to the downsampling layer Downsample again to get the downsampled features , the dimension is , and then through the upsampling layer Upsample and get the dimension ,Will and Cascade to get the 2D bird's-eye view feature , the dimension is , the region proposal network is used to extract the candidate box of the target from the feature map; the two-dimensional bird's-eye view feature Treated as 256 channels The feature image is input into the region candidate network. Since the region feature network is implemented based on anchor frames, two RoI region anchor frames are generated for each pixel of the feature image, corresponding to and In two directions, the entire feature map contains anchor boxes The region candidate network contains three convolution branches: category classification branch, location regression branch and direction classification branch, using convolution kernel sizes of The one-dimensional omnidirectional dynamic convolution (ODConv1D) is used for prediction to obtain category prediction features, bounding box parameter prediction features, and direction classification prediction features, with dimensions of ; For the candidate RoI regions obtained by the region candidate network, the non-maximum suppression algorithm is used to screen out the candidate RoI regions with high confidence and consistent with the actual target object, and each RoI region is divided into The grid is input into the RoI grid pooling layer to obtain the RoI grid features of the laser point cloud , the dimension is ; (b) Pseudo point cloud feature extraction: use the laser point cloud RoI area to perform RoI cropping on the pseudo point cloud, and then extract features from the cropped pseudo point cloud in the RoI area. The pseudo point cloud feature extraction includes three parts: point feature extraction network, RoI pooling, and voxel feature extraction network. Pseudo-point The characteristics are ,in, is a three-dimensional geometric feature, is the color information feature, for pseudo points Perform neighborhood point search and set the number of neighborhood points to For pseudo points with less than 9 neighboring points, use pseudo points The feature information of the point is completed to ensure that the number of neighborhood points of each pseudo point is 9. The point feature extraction network consists of three color point convolution layers. Each color point convolution layer contains a feature extraction FEA layer, a weight calculation WGT layer and a feature aggregation FUS layer. The feature extraction layer FEA layer is a fully connected layer, which represents the features of all pseudo points as a dimension of The tensor is used as the input of the feature extraction FEA layer, where is the number of pseudo points, for three color point convolution layers, They are 6, 12 and 24 respectively. The output features are obtained through the fully connected layer , the dimension is , for the three color point convolution layers, The weight calculation layer WGT layer is a fully connected layer. The weight calculation layer WGT layer uses the position residuals of pseudo points and neighboring pseudo points to calculate weights, build the relationship between each pseudo point and its neighboring points, and convert the pseudo points The position is represented by ,in, is a three-dimensional geometric feature, is a two-dimensional information feature, the position residual of all pseudo points and their neighboring points Represented as a dimension A tensor of is the number of pseudo points in the pseudo point cloud, which is used as the input of the weight layer WGT layer to obtain the neighborhood point weight , the dimension is ; For neighborhood point features and neighborhood weights , multiply each neighborhood point feature by its corresponding weight to obtain the weighted neighborhood point feature, expressed as a dimension of The tensor of , which is cascaded to get a dimension of The two-dimensional tensor of is used as the input of the feature aggregation FUS layer to obtain the output feature dimension , the dimension is , and use it as the input of the next layer of color point convolution feature extraction FEA layer; voxelize the pseudo point cloud RoI feature area and divide each RoI area into Each voxel corresponds to an output feature. After the feature is upgraded using a multi-layer perceptron, the maximum pooling is performed to splice the features of all voxels together to generate a pseudo point cloud RoI grid feature. , the dimension is ; The pseudo point cloud voxel feature extraction network consists of a sparse convolutional layer and two sub-flow sparse convolutional layers. Perform voxel feature extraction and finally output features The dimension is ; (c) Construct the laser point cloud and pseudo point cloud RoI grid feature fusion network, and respectively integrate the laser point cloud RoI grid feature and pseudo point cloud RoI grid features Input to the fully connected layer and , the output features are cascaded and input into the fully connected layer FC3 to obtain the grid feature weights and ; respectively and Input 1D convolutional block and After feature extraction, it is multiplied point by point with the corresponding grid feature weight to obtain the weighted feature, and the weighted feature is cascaded and input into the one-dimensional convolution block. The fused RoI grid feature is , the dimension is ; (d) Classification of candidate RoI region features and bounding box improvement. The RoI optimization detection network is used to classify the candidate RoI region features and improve their bounding boxes. First, the fully connected layer is used to integrate the fused RoI grid features. The feature channel dimension of Reduce to dimensional, and then the reduced-dimensional RoI grid feature features The input is sent to the RoI optimization detection network, which consists of two fully connected layer branches, one for category classification and one for position regression branch tasks, and the output feature dimensions are and ; (e) Construct a spatiotemporal feature aggregation network, using a bidirectional gated recurrent unit (BiGRU) to enhance features in the time dimension, and further aggregate the RoI features of the same target at consecutive moments, thereby strengthening the relationship between them and realizing the comprehensive utilization of the spatiotemporal dimension of features; the gated recurrent unit determines how much previous memory to retain at the current moment by controlling the reset gate and the update gate, and combines the current input information to update and predict the state. The bidirectional gated recurrent unit consists of two gated recurrent units in opposite directions. Considering the bidirectional information of the historical state of the object, the forward and backward input sequences are calculated respectively to obtain the corresponding forward and backward hidden states, and then these hidden states are weighted and combined according to specific weights to obtain the hidden state at the current moment; the input of the spatiotemporal feature aggregation network is the RoI feature at the current moment and the spatiotemporal aggregation feature at the previous moment, and the output is the spatiotemporal aggregation feature at the current moment. At time 𝑖̂, the RoI features of the same target object are aggregated, and the spatiotemporal aggregation feature calculation of each target is described by formula (6), (6) in, , Indicates that the target is RoI features at each moment, Indicates that the target is The spatiotemporal aggregation characteristics of each moment, Initialized to 0; (f) Construct an association network and design an association network to calculate the association between targets at two consecutive moments. Its input is the RoI feature at moment 𝑡. and Spatiotemporal aggregation characteristics of moments , after copying and expanding the input features, subtract them point by point to get the absolute value, and obtain the associated network input matrix , described by formula (7), (7) in, for The number of RoI targets at the moment, for The number of RoI targets at the moment, for Moment The RoI features of the target, for Moment The spatiotemporal aggregation features of the target are input into the associated network matrix After being passed into the fully connected layer FC, the Softmax operation is performed on the rows and columns of the output matrix of the fully connected layer respectively, and then the average value of the two is taken to obtain the appearance correlation matrix; (g) Target motion state prediction: The interactive multi-model Kalman filter is used to realize the target motion state prediction, which combines the constant velocity model (CV model) and the constant acceleration model (CA model), that is, To adapt to different sports modes, The motion state of a tracking target at time is described by formula (8): ； in, represent Always track the center coordinates of the target 3D bounding box. Represents the heading angle of the 3D bounding box of the tracked target, ( ) represents the length, width and height of the three-dimensional bounding box of the tracking target, ( ) represent the tracking target edge The speed of the axis, ( ) represent the tracking target edge The acceleration of the axis is set in the CV model ( ) are both 0; the CV model and CA model are used to predict and update the target motion state respectively. After the prediction and update states of the two motion types are calculated, the correct probabilities of the two models are updated according to the likelihood, which is described by formula (9): in, Indicated in the model Generate observations The likelihood of Indicated in Before the update, the model is the prior probability of the correct model, Indicated in Moment Model is the probability of the correct model, The weighted prediction state is calculated using the probability that the model is correct as the weight, as The final motion state prediction result at time t is described by formula (10): in, represents the weighted state prediction result, Indicated in the model Prediction results of the middle motion state; (h) To achieve target association, a multi-level association algorithm is proposed, and the Kuhn-Munkres algorithm is used to solve the assignment problem. At this moment, you need to Tracking target at all times and Detection target at any time Specifically, the following situations are involved: (a) For All detection targets at all times , according to its detection confidence, it is divided into high confidence detection targets and low confidence detection targets , set the confidence threshold =0.75, when the confidence level is greater than or equal to When , the confidence level is high, otherwise, the confidence level is low; (b) First-level association, using the Kuhn-Munkres algorithm to associate high-confidence detection targets Tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (c) Secondary association, using the Kuhn-Munkres algorithm to associate low-confidence detection targets Unmatched tracking target Matching is performed and three outputs are obtained: Unmatched tracking target , unmatched detection target and tracking-detection target matching pairs; (d) For a tracking-detection target pair, the state of the detection target is used to update the state of the tracking target to obtain , used for matching at the next moment; (e) For unmatched tracking targets , it is considered as a temporarily disappeared target, and it is retained to continue the target state prediction until it is re-matched at a later time. If it is not re-matched for more than 50 frames, it is considered that the target has completely disappeared; (f) For unmatched detection targets with high confidence , considered as a newly appeared target, the bounding box and RoI features of the detected target are initialized as a new tracking target; (g) For unmatched detection targets with low confidence , it is considered to be the wrong detection target; Step 4: Train a joint target detection and tracking network based on spatiotemporal feature aggregation of multi-sensor data. During the training process, the Adam optimizer is used, where the first decay exponent and the second decay exponential Set to and ; The batch size during training is set to 2, and the training is set to 40 epochs; Step 5: Visualization of vehicle target tracking results. After the network training is completed, the final vehicle target detection and tracking results are obtained, and the tracking results are visualized on the color image data and point cloud image data respectively.

Images