A three-dimensional target detection method for an automatic driving scene

Abstract

The application provides a three-dimensional target detection method for an automatic driving scene, which is applied to a perception system of an automatic driving vehicle and comprises the following steps: collecting three-dimensional point cloud data of the environment around the vehicle through a laser radar and synchronously collecting an RGB image at a corresponding moment through a camera; extracting BEV feature maps at different scales from the three-dimensional point cloud data and extracting corresponding target candidate features; performing multi-level feature extraction on the RGB image through a ResNet network, and extracting high-order image features of each layer of feature maps by using a high-order convolution layer; mapping the high-order image features of the last layer to a corresponding BEV space; performing multi-modal feature fusion on the high-order image features and the target candidate features; and predicting target categories, center coordinates, size coefficients and orientation angles by using the fused features. The method can effectively improve the detection accuracy of long-distance targets, small targets and point cloud sparse targets, and has good robustness and stability in a complex traffic scene.

Description

Technical Field

[0001] This application belongs to the field of target detection technology, and in particular relates to a three-dimensional target detection method for autonomous driving scenarios. Background Technology

[0002] With the rapid development of autonomous driving technology, environmental perception systems have become a core component of the safe operation of intelligent vehicles. 3D object detection technology detects and locates traffic participants such as cars, pedestrians, and bicycles in the vehicle's surrounding environment, providing crucial perception information for path planning, decision-making and control, and active safety systems. Currently, autonomous vehicles typically carry multiple sensors, including LiDAR and cameras. LiDAR provides precise 3D spatial geometric information, while cameras provide rich texture, color, and semantic information. Therefore, 3D object detection technology based on LiDAR-Camera fusion has become an important research direction in the field of autonomous driving perception.

[0003] Existing LiDAR-Camera fusion methods are mainly divided into three categories: pre-fusion, post-fusion, and deep fusion. By fusing complementary information from the two sensors, accurate detection and localization of target objects in autonomous driving scenarios can be achieved. Pre-fusion, also known as data-level fusion, projects the point cloud data from the LiDAR sensor onto the image plane, then uses a pre-trained 2D detector to extract image features, and finally processes the image features and corresponding point cloud features. Post-fusion, also known as result-level fusion, uses a mature 2D detector to predict bounding boxes in the image, and then projects the bounding boxes onto the point cloud to fuse the point cloud and image data. The key idea of ​​deep fusion is to perform a deep fusion of feature data from the two modalities at the feature level, thereby fully utilizing the data from both sensors. It mainly uses traditional deep convolutional networks to extract point cloud features and image features, then finds the correlation between LiDAR point clouds and image pixels based on a calibration matrix, and projects the image features onto the LiDAR BEV feature map, thus achieving deep fusion of LiDAR BEV features and image features.

[0004] In autonomous driving scenarios, due to varying target distances, significant size differences, and frequent occlusion, existing LiDAR-Camera fusion detection methods generally suffer from the following technical problems: (1) The feature representation capabilities of distant targets, small targets, and occluded targets in camera images are insufficient. Existing convolutional networks mainly rely on first-order statistical features for feature extraction, which makes it difficult to fully model the high-order correlation between different feature channels, resulting in a decrease in target detection accuracy.

[0005] (2) Object queries are usually generated using single-scale BEV features, which is difficult to adapt to the detection requirements of different scale targets in autonomous driving scenarios, thus affecting the detection performance of targets such as vehicles, pedestrians and traffic facilities. Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a 3D target detection method for autonomous driving scenarios, so as to improve the detection accuracy of distant targets, small targets and sparse point cloud targets.

[0007] This application provides a 3D target detection method for autonomous driving scenarios, applied to the perception system of autonomous vehicles, including: The vehicle's surrounding environment is collected using LiDAR, and RGB images are simultaneously captured by a camera at the corresponding time. BEV feature maps at different scales are extracted from 3D point cloud data, and corresponding target candidate features are extracted; the targets include one or more of vehicles, pedestrians, bicycles and traffic facilities; The ResNet network is used to extract multi-level features from RGB images, and high-order image features of each feature map are extracted using high-order convolutional layers. Based on the calibration relationship between the lidar and the camera, the final layer of high-order image features are mapped to the corresponding BEV space; The Transformer decoder is used to fuse high-order image features and target candidate features into multimodal features; By utilizing the fused features, the target category, center coordinates, size factor, and orientation angle are predicted to generate 3D bounding box detection results, thereby realizing 3D target detection and localization in road scenes.

[0008] Furthermore, the extraction of BEV feature maps at different scales from the 3D point cloud data includes: After voxelizing the 3D point cloud data, the 3D spatial features are extracted using the VoxelNet network. The 3D spatial features are then encoded using a multi-scale 3D sparse convolutional network to obtain BEV feature maps at different scales.

[0009] Furthermore, after extracting BEV feature maps at different scales from the 3D point cloud data, the method further includes: Target query vectors are constructed in BEV feature maps at different scales, and a scale linear regression strategy is used to uniformly map the target query vectors at different scales.

[0010] Furthermore, the extraction of high-order image features from each feature map using higher-order convolutional layers includes: The covariance matrix between each channel of the feature map is calculated to describe the statistical correlation between different feature channels. The covariance matrix is ​​decomposed into eigenvalues ​​and the covariance matrix is ​​calculated to obtain the feature representation containing high-order statistical information. The important feature channels are enhanced by the high-order channel attention mechanism to generate high-order image features.

[0011] Furthermore, high-order image features of the feature map are extracted using the following method: Given the height of the feature map is h、 Width The number of channels is ,Will Feature map of size Straighten Column vector of size Any two column vectors The correlation covariance between them is: ; In the formula, For the first The average value of the feature vectors of each channel. For the first The average value of the feature vectors of each channel; Then feature map The correlation covariance matrix is ​​as follows: ; set up , and then the feature map Each column vector spliced ​​into indivual Sample matrix composed of 3D samples The sample matrix The covariance matrix is: ; In the formula, yes The identity matrix, It is M 3D column vector, , T Indicates matrix transpose; Known The The first row in the representation space The correlation weight values ​​between each channel feature and all channel features, and Given a real symmetric matrix, the sample covariance matrix can be expressed using the following formula. Perform eigenvalue decomposition: ; In the formula, For the reason A diagonal matrix composed of the eigenvalues ​​of . , where eigenvalues In the matrix Arranged in non-increasing order. It is an orthogonal matrix. ,in, for The middle belongs to the eigenvalue eigenvectors; The sample covariance matrix is ​​calculated using the following formula. Transforming powers into eigenvalues power: ; In the formula, It is a real number. It is an eigenvalue The power of, ; when At that time, the sample covariance matrix is ​​obtained. The square root value, i.e., the co-standard deviation matrix, is: ; The covariance matrix is ​​expressed by the following formula. Expand: ; In the formula, Let be the costandard deviation vector, representing the th The set of costandard deviations of a channel feature map with respect to other channel feature maps is used to describe the correlation between the channel feature map and other channel feature maps. The covariance vector is calculated using the following formula. The average value of each component, i.e., the first... Data distribution of feature maps for each channel: ; Calculated using the following formula indivual The feature vector of a dimensional sample, i.e., the feature map High-order image features: .

[0012] Furthermore, the calibration relationship between the lidar and the camera is obtained in the following way: For 3D point cloud data and RGB images, spatial alignment between the point cloud coordinate system and the image coordinate system is achieved using sensor calibration parameters, thereby obtaining the calibration relationship between the LiDAR and the camera. The 3D target detection method for autonomous driving scenarios provided in this application can effectively improve the detection accuracy of distant targets, small targets, and targets in sparse point clouds, while exhibiting good robustness and stability in complex traffic scenarios. Compared with existing mainstream LiDAR-Camera fusion methods, it achieves a significant improvement in detection performance and can better meet the requirements of autonomous driving environmental perception systems for 3D target detection accuracy. Attached Figure Description

[0013] Figure 1 A flowchart of a three-dimensional target detection method for autonomous driving scenarios provided in an embodiment of this application is shown; Figure 2 A schematic diagram of the structure of the three-dimensional target detection algorithm provided in an embodiment of this application is shown; Figure 3 This illustration shows a forward propagation diagram of a high-order convolutional layer provided in an embodiment of this application; Figure 4 This illustration shows a qualitative analysis diagram of the first-order attention mechanism and higher-order attention mechanism provided in the embodiments of this application on a 2D image; Figure 5 The illustration shows the visualization effects of the same scene provided in the embodiments of this application on different algorithms. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this technical solution clearer, the following detailed description, in conjunction with specific embodiments, further illustrates this technical solution. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this technical solution.

[0015] Please see as follows Figure 1 The flowchart shown is for a 3D object detection method for autonomous driving scenarios. Figure 1 As shown, the method includes: S101. Collect three-dimensional point cloud data of the vehicle's surrounding environment using LiDAR, and simultaneously collect RGB images of the corresponding time using a camera.

[0016] Next, we will combine such as Figure 2 The schematic diagram of the 3D target detection algorithm shown illustrates the specific process of realizing 3D target detection and localization in a road scene.

[0017] S102. Extract BEV feature maps at different scales from the three-dimensional point cloud data, and extract corresponding target candidate features; the target includes one or more of vehicles, pedestrians, bicycles and traffic facilities.

[0018] In this step, the obtained BEV feature map not only preserves the spatial location information of the target, but also effectively expresses the geometric structural features of the target, providing accurate spatial prior information for subsequent target detection.

[0019] In practice, BEV feature maps at different scales are extracted from 3D point cloud data using the following methods: After voxelizing the 3D point cloud data, the 3D spatial features are extracted using the VoxelNet network. The 3D spatial features are then encoded using a multi-scale 3D sparse convolutional network to obtain BEV feature maps at different scales.

[0020] Furthermore, after extracting BEV feature maps at different scales from the 3D point cloud data, the method further includes: Target query vectors are constructed in BEV feature maps at different scales, and a scale linear regression strategy is used to uniformly map the target query vectors at different scales.

[0021] Here, to address the issue of significant differences in target scale in autonomous driving scenarios, candidate target features are extracted from BEV feature maps of different scales, and a multi-scale object query vector is constructed. A scale-based linear regression strategy is used to uniformly map the queries across different scales, enabling the network to simultaneously meet the detection needs of both large targets at close range and small targets at long distances.

[0022] S103. Multi-level feature extraction is performed on the RGB image using the ResNet network, during which high-order image features of each feature map are extracted using high-order convolutional layers.

[0023] In this step, traditional convolutional networks mainly focus on local region information and have difficulty modeling high-order statistical relationships between different feature channels. Therefore, high-order convolutional layers are embedded in the ResNet network to enhance the ability to express image features.

[0024] In practice, higher-order image features of each feature map layer are extracted using higher-order convolutional layers in the following manner: The covariance matrix between each channel of the feature map is calculated to describe the statistical correlation between different feature channels. The covariance matrix is ​​decomposed into eigenvalues ​​and the covariance matrix is ​​calculated to obtain the feature representation containing high-order statistical information. The important feature channels are enhanced by the high-order channel attention mechanism to generate high-order image features.

[0025] In this step, a convolutional neural network is first used to extract texture, color, and semantic features from the image. To enhance the expressive power of image features, a higher-order attention mechanism is introduced during the image feature extraction process. By modeling the correlation between different feature channels, higher-order statistical information that is difficult for traditional convolutional networks to express is mined. Unlike existing methods that mainly utilize local convolutional features, this application can fully utilize the correlation information between different feature channels, thereby improving the network's ability to perceive distant targets, small targets, and occluded targets. The image features enhanced by the higher-order attention mechanism not only contain rich semantic information but also more accurately reflect the feature distribution of targets in complex traffic scenes.

[0026] Next, we will discuss the combination of... Figure 3 The schematic diagram of the forward propagation of the high-order convolutional layer illustrates the process of extracting high-order image features from the feature map.

[0027] Given the height of the feature map is h、 Width The number of channels is ,Will Feature map of size Straighten Column vector of size Any two column vectors The correlation covariance between them is: (1) In the formula, For the first The average value of the feature vectors of each channel. For the first The average value of the feature vectors of each channel; Then feature map The correlation covariance matrix is ​​as follows: (2) set up , and then the feature map Each column vector spliced ​​into indivual Sample matrix composed of 3D samples The sample matrix The covariance matrix is: (3) In the formula, yes The identity matrix, It is M 3D column vector, , T Indicates matrix transpose; Known The The first row in the representation space The correlation weight values ​​between each channel feature and all channel features, and Given a real symmetric matrix, the sample covariance matrix can be expressed using the following formula. Perform eigenvalue decomposition: (4) In the formula, For the reason A diagonal matrix composed of the eigenvalues ​​of . , where eigenvalues In the matrix Arranged in non-increasing order. It is an orthogonal matrix. ,in, for The middle belongs to the eigenvalue eigenvectors; The sample covariance matrix is ​​calculated using the following formula. Transforming powers into eigenvalues power: (5) In the formula, It is a real number. It is an eigenvalue The power of, ; when At that time, the sample covariance matrix is ​​obtained. The square root value, i.e., the co-standard deviation matrix, is: (6) Since the purpose of global covariance pooling is to find a representation that can describe the distribution of the entire feature map data, the covariance matrix is ​​calculated using the following formula. Expand: (7) In the formula, Let be the costandard deviation vector, representing the th The set of costandard deviations of a channel feature map with respect to other channel feature maps is used to describe the correlation between the channel feature map and other channel feature maps. The covariance vector is calculated using the following formula. The average value of each component, i.e., the first... Data distribution of feature maps for each channel: (8) Here, for a scale of The feature map, the output of global covariance pooling is of size A three-dimensional tensor, whose value for each component in the channel dimension is .

[0028] Calculated using the following formula indivual The feature vector of a dimensional sample, i.e., the feature map High-order image features: (9) Here, compared with the statistics used in global average pooling, the global covariance pooling in this application considers the relationship between various feature channels, similar to the channel attention mechanism, which this application calls the higher-order channel attention mechanism.

[0029] S104. Based on the calibration relationship between the lidar and the camera, map the last layer of high-order image features to the corresponding BEV space.

[0030] In practical implementation, the calibration relationship between the lidar and the camera is obtained through the following methods: For 3D point cloud data and RGB images, spatial alignment between the point cloud coordinate system and the image coordinate system is achieved using sensor calibration parameters, thereby obtaining the calibration relationship between the lidar and the camera.

[0031] Here, a spatial correspondence between the two sensors is established based on the calibration parameters between the lidar and the camera, thereby realizing the information association between data from different modalities.

[0032] S105. Use the Transformer decoder to perform multimodal feature fusion of high-order image features and target candidate features.

[0033] In this step, a Transformer decoder is used to establish the correlation between point cloud features, image features, and object queries, thereby achieving deep fusion of multimodal features. During the fusion process, the target query vector interacts with both point cloud features and image features, enabling the network to simultaneously acquire spatial geometric information provided by the LiDAR and semantic information provided by the camera. This cross-modal information interaction method fully leverages the complementary advantages of the two sensors, improving target perception capabilities in complex traffic scenarios.

[0034] S106. Using the fused features, predict the target category, center coordinates, size factor and orientation angle to generate a 3D bounding box detection result, thereby realizing 3D target detection and localization in road scenes.

[0035] In this step, the fused features are input into a 3D target detection head to predict the target category, spatial location, size, and orientation, ultimately generating a 3D bounding box detection result for the target. The detection result can be output in real time to the environmental perception module and decision-making module of the autonomous vehicle, providing accurate and reliable environmental information for vehicle path planning, obstacle avoidance control, and assisted driving decisions.

[0036] It should be noted that this application is implemented using the MMDetection3D framework and has been validated on the nuScenes autonomous driving dataset.

[0037] Please see as follows Figure 4 The diagram illustrates the qualitative analysis of first-order and higher-order attention mechanisms on 2D images.

[0038] Experimental results show that the proposed method achieves superior detection performance compared to existing mainstream LiDAR-Camera fusion methods on the nuScenes public dataset for autonomous driving. The effects of first-order and higher-order attention mechanisms on 2D images are visualized on the nuScenes dataset. Figure 4 As can be seen, higher-order attention mechanisms can focus on small, distant targets. Compared to first-order attention mechanisms, higher-order attention mechanisms show significantly stronger focus on the target object.

[0039] Please see as follows Figure 5 The image shows the visualization of the same scene using different algorithms.

[0040] The experiment involved visual analysis of the same scene using different methods. As shown in the figures, different algorithms yielded different detection results for the same target in the same scene. A pedestrian walking down stairs is clearly visible within the yellow box in the image data, but this pedestrian target was not detected by the LiDAR-Only method. Both the LiDAR-Camera fusion method and the method proposed in this application (labeled HAMFNs in the figure) accurately detected this target. Meanwhile, the vehicle target within the dark blue box in the image was only detected by the HAMFNs method. These visualization results clearly demonstrate that the HAMFNs algorithm performs exceptionally well when dealing with targets with limited point cloud data. Even with limited point cloud information, it can still accurately detect these targets. This characteristic is crucial for urban traffic environments, as targets may only appear within the sensor's field of view for a short time, or point cloud data may be limited due to occlusion. In complex traffic scenarios, the method proposed in this application demonstrates good detection performance for targets with limited point cloud data, and it can also accurately detect some distant targets.

[0041] The above content is only a preferred embodiment of the present invention. For those skilled in the art, many changes can be made in the specific implementation and application scope based on the ideas of the present invention. As long as these changes do not depart from the concept of the present invention, they all fall within the protection scope of the present invention.

Claims

1. A three-dimensional target detection method for autonomous driving scenarios, characterized in that, A perception system applied to autonomous vehicles, the method comprising: The vehicle's surrounding environment is collected using LiDAR, and RGB images are simultaneously captured by a camera at the corresponding time. BEV feature maps at different scales are extracted from 3D point cloud data, and corresponding target candidate features are extracted; the targets include one or more of vehicles, pedestrians, bicycles and traffic facilities; The ResNet network is used to extract multi-level features from RGB images, and high-order image features of each feature map are extracted using high-order convolutional layers. Based on the calibration relationship between the lidar and the camera, the final layer of high-order image features are mapped to the corresponding BEV space; The Transformer decoder is used to fuse high-order image features and target candidate features into multimodal features; By utilizing the fused features, the target category, center coordinates, size factor, and orientation angle are predicted to generate 3D bounding box detection results, thereby realizing 3D target detection and localization in road scenes.

2. The method as described in claim 1, characterized in that, The extraction of BEV feature maps at different scales from 3D point cloud data includes: After voxelizing the 3D point cloud data, the 3D spatial features are extracted using the VoxelNet network. The 3D spatial features are then encoded using a multi-scale 3D sparse convolutional network to obtain BEV feature maps at different scales.

3. The method as described in claim 1, characterized in that, After extracting BEV feature maps at different scales from 3D point cloud data, the method further includes: Target query vectors are constructed in BEV feature maps at different scales, and a scale linear regression strategy is used to uniformly map the target query vectors at different scales.

4. The method as described in claim 1, characterized in that, The extraction of high-order image features from each feature map using high-order convolutional layers includes: The covariance matrix between each channel of the feature map is calculated to describe the statistical correlation between different feature channels. The covariance matrix is ​​decomposed into eigenvalues ​​and the covariance matrix is ​​calculated to obtain the feature representation containing high-order statistical information. The important feature channels are enhanced by the high-order channel attention mechanism to generate high-order image features.

5. The method as described in claim 1, characterized in that, High-order image features of the feature map are extracted using the following method: Given the height of the feature map is h、 Width The number of channels is ,Will Feature map of size Straighten Column vector of size Any two column vectors The correlation covariance between them is: ； In the formula, For the first The average value of the feature vectors of each channel. For the first The average value of the feature vectors of each channel; Then feature map The correlation covariance matrix is ​​as follows: ； set up , and then the feature map Each column vector spliced ​​into indivual Sample matrix composed of 3D samples The sample matrix The covariance matrix is: ； In the formula, yes The identity matrix, It is M 3D column vector, , T Indicates matrix transpose; Known The The first row in the representation space The correlation weight values ​​between each channel feature and all channel features, and Given a real symmetric matrix, the sample covariance matrix can be expressed using the following formula. Perform eigenvalue decomposition: ； In the formula, For the reason A diagonal matrix composed of the eigenvalues ​​of . , where eigenvalues In the matrix Arranged in non-increasing order. It is an orthogonal matrix. ,in, for The middle belongs to the eigenvalue eigenvectors; The sample covariance matrix is ​​calculated using the following formula. Transforming powers into eigenvalues power: ； In the formula, It is a real number. It is an eigenvalue The power of, ; when At that time, the sample covariance matrix is ​​obtained. The square root value, i.e., the co-standard deviation matrix, is: ； The covariance matrix is ​​expressed by the following formula. Expand: ； In the formula, Let be the costandard deviation vector, representing the th The set of costandard deviations of a channel feature map with respect to other channel feature maps is used to describe the correlation between the channel feature map and other channel feature maps. The covariance vector is calculated using the following formula. The average value of each component, i.e., the first... Data distribution of feature maps for each channel: ； Calculated using the following formula indivual The feature vector of a dimensional sample, i.e., the feature map High-order image features: 。 6. The method as described in claim 1, characterized in that, The calibration relationship between the lidar and the camera is obtained in the following way: For 3D point cloud data and RGB images, spatial alignment between the point cloud coordinate system and the image coordinate system is achieved using sensor calibration parameters, thereby obtaining the calibration relationship between the lidar and the camera.

Images