Multi-vehicle cooperative three-dimensional target detection method for heterogeneous intelligent network connection vehicle group

Abstract

The invention discloses a multi-vehicle cooperative three-dimensional target detection method for a heterogeneous intelligent network connection vehicle group, and the method comprises the following steps: carrying out the preprocessing of obtained multi-source heterogeneous vehicle-mounted sensor data, and extracting the features of multi-mode heterogeneous data; constructing a dynamic pose compensation network, learning and predicting a pose compensation amount by using a time sequence, correcting a pose error in multi-vehicle cooperative perception, and ensuring a space reference of multi-vehicle features; performing cross-modal feature fusion on the multi-vehicle multi-modal heterogeneous data features through a global-local cross-modal attention mechanism; designing a multi-scale feature distillation strategy, and reducing the semantic fusion gap of the cross-modal features; an end-to-end combined training framework is constructed to integrate dynamic pose compensation, cross-modal feature fusion and multi-scale feature distillation strategies, and a multi-vehicle collaborative three-dimensional target detection model is optimized through an adaptive weighted loss function. The technical problems of multi-vehicle heterogeneous sensor fusion, pose drift correction and cross-modal semantic alignment are solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent connected vehicle environmental perception technology, and in particular to a multi-vehicle collaborative three-dimensional target detection method for a heterogeneous intelligent connected vehicle group, which is suitable for the fusion of heterogeneous sensor data from multiple vehicles and three-dimensional target detection in a dynamic environment. Background Art

[0002] Traditional autonomous driving systems typically rely on a single type of sensor, such as lidar or cameras, for object detection. However, this single-sensor approach has significant limitations in complex environments. For example, lidar cannot effectively process scenes with complex textures and has low detection accuracy at long distances. While cameras can provide rich texture information, they lack depth information, resulting in poor performance at night or in poor lighting conditions. Furthermore, single-vehicle perception systems have blind spots, making it difficult to fully perceive the surrounding environment, limiting their performance in complex scenarios.

[0003] To address the shortcomings of a single sensor, multi-vehicle collaborative perception technology has emerged. By sharing sensor data from multiple vehicles, it can effectively expand the perception range and enhance the robustness and accuracy of 3D object detection. However, existing multi-vehicle collaborative perception methods still face many challenges, especially in the fusion of heterogeneous sensor data, sensor coordinate system alignment, and compensation for dynamic pose errors. There is still a lack of efficient solutions, which are mainly reflected in the following aspects: First, different vehicles are equipped with different types of sensors, which leads to differences in the feature expression of perception data. Direct fusion often leads to information redundancy or loss.

[0004] Secondly, due to the communication delay and vehicle motion in the multi-vehicle collaborative perception process, it is difficult to accurately align the coordinate systems of multiple vehicles, which affects the fusion effect and the accuracy of target detection.

[0005] Furthermore, images lack depth information, while point cloud data often lack texture details. Traditional fusion methods cannot fully utilize the complementarity of the two.

[0006] In summary, there is an urgent need for a three-dimensional target detection method for heterogeneous intelligent connected vehicles to meet the needs of high-precision real-time perception in complex dynamic traffic environments. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention provides a multi-vehicle collaborative three-dimensional target detection method for a heterogeneous intelligent connected vehicle group, comprising the following steps: Preprocess the acquired multi-source heterogeneous vehicle sensor data and extract multi-modal heterogeneous data features; Build a dynamic pose compensation network, use time series learning to predict pose compensation, correct pose errors in multi-vehicle collaborative perception, and ensure the spatial reference of multi-vehicle features; Cross-modal feature fusion of multi-vehicle multi-modal heterogeneous data features is performed through the global-local cross-modal attention mechanism; Design a multi-scale feature distillation strategy to narrow the semantic fusion gap of the cross-modal features; An end-to-end joint training framework is constructed to integrate dynamic pose compensation, cross-modal feature fusion and multi-scale feature distillation strategy, and an adaptive weighted loss function is used to optimize the multi-vehicle collaborative 3D target detection model.

[0008] Furthermore, the multi-source heterogeneous vehicle-mounted sensor data includes at least two of multi-vehicle lidar point cloud data, multi-vehicle camera image data, and multi-vehicle GPS / IMU pose data and calibration parameters.

[0009] Furthermore, the preprocessing method of multi-source heterogeneous vehicle-mounted sensor data includes: Perform spatiotemporal synchronization on multi-vehicle LiDAR point cloud data, segment the point cloud using voxelization, extract bird's-eye view features using the PointPillars algorithm, and output the point cloud BEV feature map. ; The car camera image data is distorted and the multi-view image is used to generate 3D cone features through the depth prediction network based on the LSS algorithm, and then projected into the BEV space to output the image BEV feature map. ; According to the multi-vehicle GPS / IMU posture data and calibration parameters, through the coordinate transformation matrix The feature of the kth car Align to the global coordinate system of the i-th vehicle and generate alignment features .

[0010] Furthermore, the method for correcting the posture error in multi-vehicle collaborative perception specifically includes: Build an end-to-end learnable dynamic posture compensation network, use a bidirectional LSTM network to process historical posture sequences, and capture the dynamic information of each vehicle's historical posture; Predict the pose compensation of each vehicle at a certain moment through the fully connected layer; The predicted posture compensation amount is applied to the original observation posture to obtain the corrected posture.

[0011] Further: The hidden state of the LSTM is expressed as: , Where, is the hidden state of the i-th vehicle at time t, is the observed position of the i-th vehicle at time t, is the hidden state, is a trainable parameter; The posture compensation includes translation compensation and rotation compensation, wherein the prediction formula of the posture compensation is: , Where, The translation compensation amount of the i-th vehicle at time t is, is the rotation compensation amount of the i-th vehicle at time t; The corrected posture is , which satisfies the expression: .

[0012] Furthermore, the cross-modal feature fusion method specifically includes: The point cloud BEV feature map The image BEV feature map is used as the query. is used as key and value to calculate the cross-modal association weight, which satisfies the expression: , where , is the size of the feature dimension, After passing through a fully connected layer and the point cloud BEV feature map Splicing to obtain global fusion features : ; The attention weights of different channels and spatial positions are extracted through the local spatial attention module, and are weighted fused with the local features of the corresponding positions to obtain the local fusion features. ; The global fusion feature With the local fusion feature Splicing along the channel dimension, The convolution layer performs processing and outputs the final fusion features , which satisfies the expression: .

[0013] Furthermore, the method for narrowing the semantic fusion gap of the cross-modal features specifically includes: Based on the corrected pose data of each vehicle, combined with the lidar point cloud data of multiple vehicles, a unified multi-vehicle point cloud global map is constructed, and pre-trained lidar detectors are used to extract multi-scale geometric prior features. ; Designing a multi-scale distillation loss function , optimize the alignment effect of image features and point cloud features, where, , where is the scale weight; The image features are upsampled through a feature pyramid network to match the scale of the point cloud features.

[0014] Furthermore, the optimization method of the multi-vehicle collaborative three-dimensional target detection model includes: The dynamic pose compensation network, the cross-modal attention module, and the feature distillation module are connected in series to form an end-to-end network; The total loss function of the multi-vehicle collaborative 3D object detection model is the weighted sum of detection loss, distillation loss, and pose compensation loss: , where The detection loss is , distillation loss and pose loss The weight of The weight distribution of detection loss, distillation loss and pose loss is updated in real time according to the complexity and difficulty of the task:

[0015] Where α, β, and γ are parameters that control the sensitivity of weight adjustment.

[0016] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention solves the technical difficulties of multi-vehicle heterogeneous sensor fusion, posture drift correction and cross-modal semantic alignment through dynamic posture compensation, global-local cross-modal attention mechanism and multi-scale feature distillation strategy, and improves the three-dimensional target detection accuracy and robustness of heterogeneous intelligent connected vehicle groups in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a flowchart of the overall process disclosed in the embodiment of the present invention; Figure 2 Schematic diagram of the global-local cross-modal attention mechanism disclosed in an embodiment of the present invention; Figure 3 A schematic diagram of the multi-scale feature distillation strategy disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the end-to-end multi-vehicle collaborative 3D object detection network architecture disclosed in an embodiment of the present invention; Figure 5This is the effect diagram of target detection by an existing single laser radar; Figure 6 This is the effect diagram of target detection by an existing single camera; Figure 7 This is a rendering of the multi-vehicle collaborative three-dimensional target detection effect disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] See also Figure 1 The present invention aims to provide a multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicles, which solves the limitation of the existing single sensor method that is difficult to fully perceive the surrounding environment in complex environments. The method mainly includes the following steps: S1. Preprocess the acquired multi-source heterogeneous vehicle sensor data and extract multi-modal heterogeneous data features.

[0021] In this embodiment, the multi-source heterogeneous vehicle-mounted sensor data includes at least two of multi-vehicle lidar point cloud data, multi-vehicle camera image data, and multi-vehicle GPS / IMU pose data and calibration parameters.

[0022] S11. Synchronize the time and space of multi-vehicle LiDAR point cloud data and use voxelization to segment the point cloud into The voxel grid is constructed and the PointPillars algorithm is used to extract the bird's-eye view (BEV) features and output the point cloud BEV feature map. .

[0023] S12, perform distortion correction on the car camera image data, generate 3D cone features from the multi-view images through the depth prediction network based on the LSS (Lift-Splat-Shot) algorithm, and project them into the BEV space to output the image BEV feature map .

[0024] S13, according to the multi-vehicle GPS / IMU posture data and calibration parameters, through the coordinate transformation matrix The feature of the kth car Align to the global coordinate system of the i-th vehicle and generate alignment features .

[0025] S2. Build a dynamic posture compensation network, use time series learning to predict the posture compensation amount, correct the posture error in multi-vehicle collaborative perception, and ensure the spatial reference of multi-vehicle features.

[0026] S21. Build an end-to-end learnable dynamic pose compensation network. This network uses a bidirectional LSTM network to process the historical pose sequence, thereby capturing the dynamic information of each vehicle's historical pose, that is, the vehicle's dynamic motion pattern and time dependency. Given the observed pose (including position and rotation) of the 𝑖th vehicle at time 𝑡, the hidden state of the LSTM can be expressed as:

[0027] Where, is the hidden state of the i-th vehicle at time t, is the observed position of the i-th vehicle at time t, is the hidden state, is a trainable parameter.

[0028] S22. Through the learning of the bidirectional LSTM network, the dynamic information of the historical posture of each vehicle can be obtained. Then, a fully connected layer is used to predict the posture compensation of each vehicle at a certain moment. The posture compensation includes translation compensation and rotation compensation. The prediction formula of the posture compensation is:

[0029] Where, The translation compensation of the i-th vehicle at time t is: is the rotation compensation of the i-th vehicle at time t.

[0030] S23, finally the predicted posture compensation Acting on the original observation pose , thus obtaining the corrected pose :

[0031] S3. Cross-modal feature fusion of multi-vehicle multi-modal heterogeneous data features is performed through the global-local cross-modal attention mechanism.

[0032] Those skilled in the art further explained that, since lidar point cloud data and camera image data often have cross-modal feature differences, a global cross-modal attention module is designed to calculate the association weights between image features and lidar features so as to assign different attention levels to each modal feature.

[0033] S31, such as Figure 2 As shown, point cloud BEV feature map As a query, image BEV feature map is used as a key and a value to calculate the cross-modal association weight, which satisfies the expression:

[0034] Where, , is the size of the feature dimension, After passing through a fully connected layer and the point cloud BEV feature map Splicing to obtain global fusion features :

[0035] S32, extract the attention weights of different channels and spatial positions through the local spatial attention module, and perform weighted fusion with the local features of the corresponding positions to obtain the local fusion features .

[0036] S33, global fusion features and local fusion features Splicing along the channel dimension and passing The convolution layer performs processing and outputs the final fusion features , which satisfies the expression:

[0037] S4. Design a multi-scale feature distillation strategy to narrow the semantic fusion gap of cross-modal features.

[0038] S41. Based on the corrected pose data of each vehicle and the LiDAR point cloud data of multiple vehicles, a unified multi-vehicle point cloud global map is constructed. The construction of the global map can be expressed by the following formula:

[0039] Where, For the global map of multi-vehicle point clouds, is the point cloud data of the k-th vehicle, is the transformation matrix from the kth vehicle coordinate system to the ith vehicle global coordinate system. Adaptive voxel filtering (voxel size is dynamically adjusted with point density) is used to construct a dense global map with a voxel resolution of Set according to the scene complexity (for example, 0.2m×0.2m×0.4m).

[0040] The pre-trained SECOND-LiDAR detector (based on a sparse 3D convolutional network) is then used as a teacher model to extract multi-scale geometric prior features. .

[0041] S42. In the process of cross-modal feature alignment, there are significant differences in semantic information and spatial structure between point cloud data and image data. To narrow this difference, a multi-scale distillation loss strategy is adopted, combining KL divergence and mean square error (MSE). KL divergence is used to measure the distribution difference between the teacher model (point cloud features) and the student model (image features) at different scales, while MSE measures the spatial matching error between image features and point cloud features. Figure 3 As shown, by designing a multi-scale distillation loss function , optimize the alignment effect of image features and point cloud features:

[0042] Where, is the scale weight.

[0043] S43. To achieve accurate spatial matching, a Feature Pyramid Network (FPN) is used to upsample image features to match the scale of point cloud features. FPN can process features at different scales and achieve efficient multi-scale representation through cross-layer feature fusion.

[0044] First, the image features are upsampled by FPN and mapped to the same scale as the point cloud features. Assume that the image features are , the point cloud features are , the upsampling process can be expressed as:

[0045] Where, is the upsampled image feature. Then, the upsampled image feature and the point cloud feature are aligned at each scale to ensure that they have the same spatial representation. The alignment operation of image features and point cloud features is expressed as:

[0046] Through feature alignment, image features and point cloud features can be compared and fused at the same spatial scale, providing more accurate input for subsequent detection tasks.

[0047] S5. Build an end-to-end joint training framework to integrate dynamic pose compensation, cross-modal feature fusion, and multi-scale feature distillation strategies, and optimize the multi-vehicle collaborative 3D target detection model through an adaptive weighted loss function.

[0048] S51, such as Figure 4 As shown in the figure, a dynamic pose compensation network, a global-local cross-modal attention module, and a multi-scale feature distillation module are connected in series to form an end-to-end network. The corrected pose output by the pose compensation network is used for multi-vehicle feature alignment. The aligned multimodal features are then fused using a cross-modal attention mechanism, and finally the feature distillation module optimizes cross-modal feature consistency.

[0049] S52. The total loss function of the multi-vehicle collaborative 3D object detection model is a weighted sum of the detection loss, distillation loss, and pose compensation loss to ensure that the optimization objectives of each part are balanced:

[0050] Where, The detection loss is , distillation loss and pose loss Weight Detection loss Used to optimize 3D object detection performance, Focal Loss is used to optimize the 3D detection head output.

[0051] Distillation loss It is used to optimize the alignment effect of image and point cloud features and reduce the semantic differences between cross-modal data. It consists of multi-scale KL divergence and MSE.

[0052] Pose loss It is used to measure the effect of posture error compensation and ensure the accuracy of posture correction during multi-vehicle collaborative perception. The L2 norm of the posture compensation amount is used for supervision.

[0053] S53. Update the weight distribution of detection loss, distillation loss, and pose loss in real time according to the complexity and difficulty of the task:

[0054] Where, are the changes in target detection, distillation and pose compensation losses at time t; α, β, and γ are parameters that control the sensitivity of weight adjustment.

[0055] As training progresses, the weights of the loss function are adaptively adjusted based on the optimization progress of each task. For example, when object detection accuracy is low, the weight of the detection loss is dynamically increased to enable the model to achieve better performance on the object detection task. Conversely, if the distillation loss is too large, the weight of the distillation loss can be adjusted to ensure that the distillation process does not affect object detection performance.

[0056] Figure 7 The results of 3D object detection on the OPV2V and DAIR-V2X public datasets are presented. The results show that the method provided in this embodiment achieves excellent detection performance in a heterogeneous intelligent connected vehicle swarm.

[0057] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-vehicle collaborative three-dimensional target detection method for a heterogeneous intelligent connected vehicle group, characterized by: The following steps are involved: Preprocess the acquired multi-source heterogeneous vehicle sensor data and extract multi-modal heterogeneous data features; Build a dynamic pose compensation network, use time series learning to predict pose compensation, correct pose errors in multi-vehicle collaborative perception, and ensure the spatial reference of multi-vehicle features; Cross-modal feature fusion of multi-vehicle multi-modal heterogeneous data features is performed through the global-local cross-modal attention mechanism; Design a multi-scale feature distillation strategy to narrow the semantic fusion gap of the cross-modal features; An end-to-end joint training framework is constructed to integrate dynamic pose compensation, cross-modal feature fusion and multi-scale feature distillation strategy, and an adaptive weighted loss function is used to optimize the multi-vehicle collaborative 3D target detection model.

2. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 1 is characterized in that: The multi-source heterogeneous vehicle-mounted sensor data includes at least two of multi-vehicle lidar point cloud data, multi-vehicle camera image data, and multi-vehicle GPS / IMU pose data and calibration parameters.

3. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 2 is characterized in that: The preprocessing method of multi-source heterogeneous vehicle-mounted sensor data includes: Perform spatiotemporal synchronization on multi-vehicle LiDAR point cloud data, segment the point cloud using voxelization, extract bird's-eye view features using the PointPillars algorithm, and output the point cloud BEV feature map. ; The car camera image data is distorted and the multi-view image is used to generate 3D cone features through the depth prediction network based on the LSS algorithm, and then projected into the BEV space to output the image BEV feature map. ; According to the multi-vehicle GPS / IMU posture data and calibration parameters, through the coordinate transformation matrix The feature of the kth car Align to the global coordinate system of the i-th vehicle and generate alignment features .

4. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 2 is characterized in that: The method for correcting the posture error in multi-vehicle collaborative perception specifically includes: Build an end-to-end learnable dynamic posture compensation network, use a bidirectional LSTM network to process historical posture sequences, and capture the dynamic information of each vehicle's historical posture; Predict the pose compensation of each vehicle at a certain moment through the fully connected layer; The predicted posture compensation amount is applied to the original observation posture to obtain the corrected posture.

5. The multi-vehicle collaborative 3D target detection method for heterogeneous intelligent connected vehicle groups according to claim 4 is characterized by: The hidden state of the LSTM is expressed as: ， Where, is the hidden state of the i-th vehicle at time t, is the observed position of the i-th vehicle at time t, is the hidden state, is a trainable parameter; The posture compensation includes translation compensation and rotation compensation, wherein the prediction formula of the posture compensation is: ， Where, The translation compensation amount of the i-th vehicle at time t is, is the rotation compensation amount of the i-th vehicle at time t; The corrected posture is , which satisfies the expression: 。 6. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 3 is characterized in that: The cross-modal feature fusion method specifically includes: The point cloud BEV feature map The image BEV feature map is used as the query. is used as key and value to calculate the cross-modal association weight, which satisfies the expression: , where , is the size of the feature dimension, After passing through a fully connected layer and the point cloud BEV feature map Splicing to obtain global fusion features : ; The attention weights of different channels and spatial positions are extracted through the local spatial attention module, and are weighted fused with the local features of the corresponding positions to obtain the local fusion features. ; The global fusion feature With the local fusion feature Splicing along the channel dimension, The convolution layer performs processing and outputs the final fusion features , which satisfies the expression: .

7. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 2 is characterized in that: The method for narrowing the semantic fusion gap of the cross-modal features specifically includes: Based on the corrected pose data of each vehicle, combined with the lidar point cloud data of multiple vehicles, a unified multi-vehicle point cloud global map is constructed, and pre-trained lidar detectors are used to extract multi-scale geometric prior features. ; Designing a multi-scale distillation loss function , optimize the alignment effect of image features and point cloud features, where, , where is the scale weight; The image features are upsampled through a feature pyramid network to match the scale of the point cloud features.

8. The multi-vehicle collaborative three-dimensional target detection method for heterogeneous intelligent connected vehicle groups according to claim 2 is characterized in that: The optimization method of the multi-vehicle collaborative three-dimensional target detection model includes: The dynamic pose compensation network, the cross-modal attention module, and the feature distillation module are connected in series to form an end-to-end network; The total loss function of the multi-vehicle collaborative 3D object detection model is the weighted sum of detection loss, distillation loss, and pose compensation loss: , where The detection loss is , distillation loss and pose loss The weight of The weight distribution of detection loss, distillation loss and pose loss is updated in real time according to the complexity and difficulty of the task: Where α, β, and γ are parameters that control the sensitivity of weight adjustment.

Images