3D target detection and calibration parameter enhancement method

Abstract

The invention discloses a 3D target detection and calibration parameter enhancement method. The method comprises the steps of S1, point cloud feature extraction: performing feature extraction on a point cloud top view, a point cloud front view and an image through various sensors; s2, parameter enhancement processing: disturbing the parameters obtained in the step S1 to realize parameter enhancement; s3, calculating a candidate region; s4, regional feature integration; and S5, data network regional fusion. According to the 3D target detection and calibration parameter enhancement method, only the point cloud top view and the point cloud front view are extracted in the S1 point cloud feature extraction stage, so that only the point cloud top view and the point cloud front view need to be calculated in the S3 candidate region calculation stage, the calculation amount can be reduced, excessive information cannot be lost, and the accuracy of the detection result is improved. In addition, the method enables the model to adjust parameters to adapt to data with inaccurate calibration parameters in a training process through enhanced calibration, thereby improving the robustness of the model, and further improving the accuracy of a 3D target detection result.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a 3D target detection and calibration parameter enhancement method thereof. Background Art

[0002] As one of the core foundations of autonomous driving perception systems, 3D object detection can be widely used in path planning, motion prediction, collision avoidance and other functions. Generally, cars with corresponding 3D laser radar sensors have become standard configurations in the field of autonomous driving, which can provide accurate depth information, and the processing of point cloud data is becoming more and more common and important. Although there have been many advances, due to the high sparsity and irregular characteristics of point clouds, traditional convolutional neural networks cannot accurately learn point cloud data. In addition, due to the misalignment between the camera view and the laser radar bird's-eye view, the processing of three-dimensional point clouds is much more difficult than two-dimensional images, so its parameters need to be enhanced. After searching, it is found that the 3D object detection method in the prior art is typical, such as the publication number CN117173257B, a 3D object detection and calibration parameter enhancement method, electronic device, and medium, which aims to solve the problem that the 3D object detection performance of the model is greatly reduced when the calibration parameters obtained during actual detection are inaccurate. To this end, the 3D target detection calibration parameter enhancement method of the present application includes: obtaining calibration parameters to be enhanced, the calibration parameters including first calibration parameters and / or second calibration parameters; obtaining enhancement parameters for enhancing the calibration parameters based on vehicle-mounted sensors; and perturbing the calibration parameters based on the enhancement parameters to achieve calibration parameter enhancement. Its main feature is to enhance the calibration parameters so that the model can adjust the parameters during the training process to adapt to data with inaccurate calibration parameters, thereby improving the robustness of the model and making the results of 3D target detection more accurate.

[0003] In summary, in order to avoid information loss, the existing 3D object detection method has a huge amount of calculation. To address the above problems, it is necessary to improve the existing equipment. Summary of the invention

[0004] The purpose of the present invention is to provide a 3D object detection and calibration parameter enhancement method thereof, so as to solve the problem that the existing 3D object detection method proposed in the above background technology has a huge amount of calculation in order to avoid information loss.

[0005] To achieve the above object, the present invention provides the following technical solution: a 3D target detection and calibration parameter enhancement method, comprising the following steps:

[0006] S1. Point cloud feature extraction: extract features from the point cloud top view, point cloud front view and image through various sensors;

[0007] S2, parameter enhancement processing: perturbing the parameters obtained in the S1 stage, and thereby achieving parameter enhancement;

[0008] S3, calculate candidate regions: use RPN to perform region calculation on the top view obtained in the S1 stage, that is, perform sliding window operation through the shared feature map, output 2k scores and 4k coordinates, and combine with Fast RCNN to form an overall network structure;

[0009] S4, regional feature integration: project the candidate region of the top view into the front view and image, and integrate them into the same dimension through ROI pooling;

[0010] S5. Data network area fusion: Fusion methods include Deep Fusion, Late Fusion and Early Fusion, that is, the final target detection result is obtained through any of the above methods.

[0011] Preferably, the sensors used in the S1 point cloud feature extraction stage include vehicle-mounted sensors, GPS sensors and lidar, and the image feature extraction in the S1 point cloud feature extraction stage is implemented by VGG-16.

[0012] Preferably, in the S1 point cloud feature extraction stage, the point cloud top view consists of height, intensity, and density, which are projected into a two-dimensional grid with a resolution of 0.1, wherein the height feature is obtained by the highest value in the point cloud cell; in order to encode more height features, the point cloud is divided into M blocks, and the corresponding height map is calculated for each block, thereby obtaining M height maps.

[0013] Preferably, the intensity in the point cloud top view in the S1 point cloud feature extraction stage is the mapping value of the point with the maximum height in each cell.

[0014] Preferably, the density in the point cloud top view in the S1 point cloud feature extraction stage represents the number of points in each cell, and the normalized feature is obtained by a calculation formula, which is:

[0015] Where, N is the number of points in the cell.

[0016] Preferably, in the S1 point cloud feature extraction stage, the point cloud front view is projected onto the cylinder, and the 3D coordinates are set to p = (x, y, z), and the forward view coordinates are obtained as pfv = (r, c), where c = [atan2 (y, x) / Δθ],

[0017]

[0018] Preferably, the input of ROIPooling in the S4 regional feature integration stage includes two parts, namely, the input of the feature map and rois. The input of the feature map is in Fast RCNNQ, which is before Rol Pooling. In Faster RCNN, the feature map is shared with RPN.

[0019] Preferably, the rois in the input of ROIPooling in the S4 regional feature integration stage refers to the output of Selective Search in FastRCNN; and refers to the output of RPN in Faster RCNN, with a shape of 1x5x1x1.

[0020] Preferably, the output of ROIPooling in the S4 regional feature integration stage is a batch of vectors, wherein the value of the batch is equal to the number of Rols, and the size of the vector is channel*w*h; meanwhile, the process of Rol Pooling is to map box rectangles of different sizes into rectangles of fixed size (w*h).

[0021] Preferably, a calibration parameter enhancement method is applied to the above-mentioned 3D target detection method, including obtaining the calibration parameters to be enhanced of the detection of vehicle-mounted sensors, GPS sensors and lidars, the calibration parameters to be enhanced include point cloud extrinsic enhancement parameters, displacement enhancement parameters, point cloud data, image internal enhancement parameters, image external enhancement parameters and image data, and the vehicle-mounted sensors include vehicle-mounted cameras and vehicle-mounted images.

[0022] Compared with the prior art, the invention has the following beneficial effects: the 3D target detection and calibration parameter enhancement method thereof,

[0023] The present invention can effectively solve the problem of large amount of calculation in existing 3D target detection methods by using the S1 point cloud feature extraction stage and the S3 candidate area calculation stage in coordination. In order to avoid information loss, firstly, in the S1 point cloud feature extraction stage, only the top view and the front view of the point cloud are extracted, so that the S3 candidate area calculation stage only needs to calculate the top view and the front view of the point cloud, which can reduce the amount of calculation without losing too much information. In addition, the method enhances calibration to allow the model to adjust parameters during the training process to adapt to data with inaccurate calibration parameters, thereby improving the robustness of the model and further improving the accuracy of the 3D target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the overall detection process of the present invention;

[0025] Figure 2This is a schematic diagram of the calculation process of the point cloud top view in the S3 candidate area calculation stage of the present invention;

[0026] Figure 3 This is a schematic diagram of the workflow of the S5 data network regional integration stage of the present invention;

[0027] Figure 4 This is a flow chart of the calibration parameter enhancement method of the present invention;

[0028] Figure 5 It is a schematic diagram of the structure of the FPN of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1-5 The present invention provides a technical solution: a 3D target detection and calibration parameter enhancement method,

[0031] Embodiment 1

[0032] A 3D object detection method includes the following steps:

[0033] S1. Point cloud feature extraction: extract features from the point cloud top view, point cloud front view and image through various sensors;

[0034] S2, parameter enhancement processing: perturb the parameters obtained in the S1 stage to achieve parameter enhancement;

[0035] S3, calculate candidate regions: use RPN to perform region calculation on the top view obtained in the S1 stage, that is, perform sliding window operation through the shared feature map, output 2k scores and 4k coordinates, and combine with Fast RCNN to form an overall network structure;

[0036] S4, regional feature integration: project the candidate region of the top view into the front view and image, and integrate them into the same dimension through ROI pooling;

[0037] S5. Data network area fusion: Fusion methods include Deep Fusion, Late Fusion and Early Fusion, that is, the final target detection result is obtained through any of the above methods.

[0038] Late Fusion: Improves detection performance by combining detection results from different layers (before the final fusion is completed, detection begins on partially fused layers, there will be multiple layers of detection, and multiple detection results will eventually be fused). It can be manifested in two ways:

[0039] Feature is not fused, multi-scale features are predicted separately, and then the prediction results are integrated, such as Single Shot MultiBox Detector (SSD), Multi-scale CNN (MS-CNN)

[0040] The features are pyramid-fused and predicted after fusion, such as Feature Pyramid Network (FPN).

[0041] The FPN algorithm uses both the high resolution of low-level features and the high semantic information of high-level features to achieve prediction by fusing the features of these different layers. And the prediction is performed separately on each fused feature layer, which is different from the conventional feature fusion method.

[0042] FPN upsamples the deep information and adds it to the shallow information element by element, thus constructing a feature pyramid structure of different sizes. It has excellent performance and has become a standard component of target detection algorithms. The FPN structure is shown in the attached figure. Figure 5 As shown, the following results are obtained;

[0043] From bottom to top: The leftmost is a common convolutional network. The default ResNet structure is used to extract semantic information. C1 represents the first few convolution and pooling layers of ResNet, while C2 to C5 are different ResNet convolution groups. These convolution groups contain multiple Bottleneck structures. The feature maps in the group have the same size, and the size between groups decreases.

[0044] From top to bottom: First, perform 1×1 convolution on C5 to reduce the number of channels to obtain P5, and then upsample them in sequence to obtain P4, P3 and P2. The purpose is to obtain features with the same length and width as C4, C3 and C2, so as to facilitate the next step of element-by-element addition. Here, 2x nearest neighbor upsampling is used, that is, the adjacent elements are directly copied instead of linear interpolation;

[0045] Horizontal connection: The purpose is to fuse the upsampled high semantic features with the shallow positioning detail features. After upsampling, the length and width of the high semantic features are the same as the corresponding shallow features, and the number of channels is fixed to 256. Therefore, it is necessary to perform 11 convolutions on the underlying features C2 to C4 to make the number of channels 256, and then add the two element by element to obtain P4, P3 and P2. Since the feature map size of C1 is large and the semantic information is insufficient, C1 is not placed in the horizontal connection;

[0046] Convolution fusion: After obtaining the added features, the generated P2 to P4 are fused again using 3×3 convolution to eliminate the overlapping effect caused by the upsampling process to generate the final feature map.

[0047] Early Fusion refers to the direct fusion of data from different modalities or sources before model training to construct a unified input feature vector. For example, in the fusion of images and texts, the pixel values ​​of the image and the embedding vector of the text can be concatenated as a large input vector; it has the characteristics of first fusing multiple layers of features, then training the predictor on the fused features, and only after complete fusion, unified detection is performed. This type of method is also called skip connection, that is, using concat and add operations:

[0048] Concat: Serial feature fusion, directly concatenate two features. If the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p+q;

[0049] Add: Parallel strategy, combining the two feature vectors into a complex vector, for input features x and y, z = x + iy, where i is the imaginary unit.

[0050] The sensors used in the S1 point cloud feature extraction stage include vehicle-mounted sensors, GPS sensors, and lidar, and the image feature extraction in the S1 point cloud feature extraction stage is implemented through VGG-16.

[0051] In a further embodiment, in the S1 point cloud feature extraction stage, the point cloud top view consists of height, intensity, and density, and is projected into a two-dimensional grid with a resolution of 0.1, where the height feature is obtained by the highest value in the point cloud cell; in order to encode more height features, the point cloud is divided into M blocks, and the corresponding height map is calculated for each block, thereby obtaining M height maps.

[0052] In a further embodiment, the intensity in the point cloud top view in the S1 point cloud feature extraction stage is the mapping value of the point with the maximum height in each cell.

[0053] In a further embodiment, the density in the point cloud top view in the S1 point cloud feature extraction stage represents the number of points in each cell, and the normalized feature is obtained by a calculation formula, and the calculation formula is:

[0054] Where, N is the number of points in the cell.

[0055] In a further embodiment, in the S1 point cloud feature extraction stage, the point cloud front view is projected onto the cylinder, and the 3D coordinates are set to p = (x, y, z), and the front view coordinates are obtained as pfv = (r, c), where c = [atan2 (y, x) / Δθ],

[0056] In a further embodiment, the input of ROIPooling in the S4 regional feature integration stage includes two parts, namely, the input of the feature map and rois. The input of the feature map is in Fast RCNNQ, which is before Rol Pooling. In Faster RCNN, the feature map is shared with RPN.

[0057] In a further embodiment, the rois in the input of ROIPooling in the S4 regional feature integration stage refers to the output of Selective Search in Fast RCNN; and refers to the output of RPN in Faster RCNN, with a shape of 1x5x1x1.

[0058] In a further embodiment, the output of ROIPooling in the S4 regional feature integration stage is a batch of vectors, where the value of the batch is equal to the number of Rols, and the size of the vector is channel*w*h; at the same time, the process of Rol Pooling is to map box rectangles of different sizes into rectangles of fixed size (w*h).

[0059] Embodiment 2

[0060] This embodiment is a further description of the above embodiment. It should be understood that this embodiment includes all the aforementioned technical features and is further described in detail.

[0061] In a further embodiment, a calibration parameter enhancement method is applied to the above-mentioned 3D target detection method, including obtaining the calibration parameters to be enhanced of the detection of vehicle-mounted sensors, GPS sensors and lidars, the calibration parameters to be enhanced include point cloud extrinsic parameter enhancement parameters, displacement enhancement parameters, point cloud data, image internal enhancement parameters, image external enhancement parameters and image data, and the vehicle-mounted sensors include vehicle-mounted cameras and vehicle-mounted images.

[0062] Specifically, the on-board sensors also include environmental perception sensors, body perception sensors and position sensors; obtaining the calibration parameters to be enhanced includes obtaining point cloud data based on lidar; obtaining point cloud external parameters based on point cloud data; and obtaining image data based on the on-board camera.

[0063] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D object detection method, characterized in that , including the following steps: S1. Point cloud feature extraction: extract features from the point cloud top view, point cloud front view and image through various sensors; S2, parameter enhancement processing: perturbing the parameters obtained in the S1 stage, and thereby achieving parameter enhancement; S3, calculate candidate regions: use RPN to perform region calculation on the top view obtained in the S1 stage, that is, perform sliding window operation through the shared feature map, output 2k scores and 4k coordinates, and combine with Fast RCNN to form an overall network structure; S4, regional feature integration: project the candidate region of the top view into the front view and image, and integrate them into the same dimension through ROI pooling; S5. Data network area fusion: Fusion methods include Deep Fusion, Late Fusion and Early Fusion, that is, the final target detection result is obtained through any of the above methods.

2. A 3D object detection method according to claim 1, characterized in that: The sensors used in the S1 point cloud feature extraction stage include vehicle-mounted sensors, GPS sensors and lidars, and the image feature extraction in the S1 point cloud feature extraction stage is implemented through VGG-16.

3. A 3D object detection method according to claim 1, characterized in that: In the S1 point cloud feature extraction stage, the point cloud top view consists of height, intensity, and density, which are projected into a two-dimensional grid with a resolution of 0.1, where the height feature is obtained by the highest value in the point cloud cell; in order to encode more height features, the point cloud is divided into M blocks, and the corresponding height map is calculated for each block, thereby obtaining M height maps.

4. A 3D object detection method according to claim 1, characterized in that: The intensity in the point cloud top view in the S1 point cloud feature extraction stage is the mapping value of the point with the maximum height in each cell.

5. The 3D object detection method according to claim 1, characterized in that: The density in the point cloud top view in the S1 point cloud feature extraction stage represents the number of points in each cell, and the normalized feature is obtained by the calculation formula, which is: Where, N is the number of points in the cell.

6. A 3D object detection method according to claim 1, characterized in that: In the S1 point cloud feature extraction stage, the point cloud front view is projected onto the cylinder, and the 3D coordinates are set to p = (x, y, z), and the front view coordinates are obtained as pfv = (r, c), where c = [atan2 (y, x) / Δθ], 7. A 3D object detection method according to claim 1, characterized in that: The input of ROIPooling in the S4 regional feature integration stage includes two parts, namely the input of the feature map and rois. The input of the feature map is in FastRCNNQ, which is before Rol Pooling. In Faster RCNN, it shares the feature map with RPN.

8. The 3D object detection method according to claim 1, characterized in that: The rois in the input of ROIPooling in the S4 regional feature integration stage refers to the output of Selective Search in Fast RCNN; in Faster RCNN, it refers to the output of RPN, with a shape of 1x5x1x1.

9. A 3D object detection method according to claim 1, characterized in that: The output of ROIPooling in the S4 regional feature integration stage is a batch of vectors, where the value of the batch is equal to the number of Rols, and the size of the vector is channel*w*h; at the same time, the process of Rol Pooling is to map box rectangles of different sizes into rectangular boxes of fixed size (w*h).

10. A calibration parameter enhancement method, applied to the 3D object detection method according to any one of claims 1 to 9, characterized in that: It includes obtaining the calibration parameters to be enhanced detected by vehicle-mounted sensors, GPS sensors and laser radars, wherein the calibration parameters to be enhanced include point cloud extrinsic enhancement parameters, displacement enhancement parameters, point cloud data, image internal enhancement parameters, image external enhancement parameters and image data, and the vehicle-mounted sensors include vehicle-mounted cameras and vehicle-mounted images.

Images