3D Object Detection Method for Dense Segmentation by Fusing LiDAR Point Cloud and Image

Abstract

The present invention discloses a 3D object detection method for dense segmentation integrating LiDAR point cloud and image, which relates to the technical field of 3D object detection. The present invention at least includes the following: building an application framework with a frustum proposal generation, a point dense segmentation module, a multi-scale centroid estimation module and a detection head; performing 2D detection on a 2D image of a given scene through a 2D detector to generate frustum proposals, performing instance segmentation on the points in the proposals, and performing centroid estimation on the segmented points so as to generate 3D bounding boxes by using the detection head to complete the detection. The present invention integrates LiDAR point cloud and RGB image, overcomes the problem of poor detection performance for small objects such as pedestrians and cyclists at present, while maintaining the detection accuracy for vehicles, and introduces a point dense segmentation module and a centroid estimation method of multi-scale features, retains the texture information of the object and improves the detection accuracy. The proposed method performs better than the existing methods on the KITTI dataset, proving its effectiveness.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D object detection, and specifically to a 3D object detection method for dense segmentation that fuses lidar point clouds and images. Background Art

[0002] Autonomous driving is widely regarded as one of the most promising remedies to protect humans from severe collisions. In this regard, three-dimensional object detection, as the foundation of an autonomous driving perception system, especially in guiding path planning, motion prediction, and collision avoidance, plays a crucial role.

[0003] As a leading artificial intelligence technology, deep learning has achieved remarkable success in solving various two-dimensional vision problems. However, compared with two-dimensional images, there are still great challenges in processing three-dimensional data using deep neural networks. Although 3D data has the potential to provide richer scene information and facilitate 3D perception tasks, deep learning for point clouds is still in its infancy.

[0004] Existing three-dimensional object detection methods are all based on the currently mature two-dimensional object detection technology. These methods usually involve converting three-dimensional point clouds into two-dimensional images projected through multiple angles, or into a regular grid of voxels that are further encoded in a convolutional neural network (CNN). Although these methods effectively solve the complex geometric features of point clouds, they will cause the loss of key three-dimensional information during the conversion process of point clouds. With the development of deep learning networks for point clouds, such as PointNet and PoitnNet++, existing methods directly extract features from the original point clouds, reducing the loss of spatial geometric information to a certain extent.

[0005] To improve the performance of three-dimensional object detection, using a multi-modal strategy can simultaneously utilize the rich and dense semantic information in images and the valuable spatial geometric information in point clouds. The F-PointNet network is a pioneering work in this field. It first finds the local points corresponding to pixels in the 2D region, generates a frustum proposal for each defined object in 3D space, then uses PointNet to segment foreground points from these proposals, and finally estimates the non-modal 3D boxes from the foreground points.

[0006] In summary, the prior art has the following problems:

[0007] 1. In three-dimensional space, the points of small objects such as pedestrians and cyclists do not have obvious geometric structures, and there are fewer data points, making it more difficult to distinguish them from other objects in the environment. Therefore, compared with large objects such as cars, most existing three-dimensional object detection methods show lower performance when detecting small objects such as pedestrians and cyclists.

[0008] 2. Currently, most methods use PointNet and PointNet++ to extract point cloud features and regress bounding box parameters, but they are mostly used for 3D object classification and segmentation in indoor scenes and are not suitable for outdoor driving scenes with complex backgrounds. This is because PointNet / PointNet++ learns point cloud features but ignores the information flow between layers. As a result, some local information is still lost, resulting in poor segmentation effect.

[0009] 3. In 3D object detection, the center position of an object is usually represented by its center point. Therefore, in order to determine the center position of each object in a typical coordinate system, it is necessary to estimate the center point of the point cloud. T-Net can be more robust to rotation, translation, and scaling changes of the input point cloud. It is usually used in the center point estimation module of most methods to predict the center position of the target object and the size of the bounding box. However, T-Net is a small neural network module composed of a multi-layer perceptron and an affine transformation, and its performance is affected by the density and distribution of the point cloud. When the point cloud is sparse or unevenly distributed, the model may make inaccurate predictions about the center position and bounding box. In addition, when the size and distance of the target object vary greatly, T-Net may have difficulty handling multi-scale problems.

[0010] Therefore, a new solution to the above problems needs to be proposed. Summary of the Invention

[0011] The purpose of the present invention is to provide a 3D target detection method that integrates lidar point cloud and dense segmentation of images to solve the problems raised in the background technology.

[0012] To achieve the above objectives, the present invention provides the following technical solution: a 3D target detection method integrating lidar point cloud and image dense segmentation, comprising at least the following:

[0013] Build an application framework with frustum proposal generation, point dense segmentation module, multi-scale centroid estimation module and detection head;

[0014] Generate a frustum proposal by performing 2D detection on the 2D image of a given scene, and perform instance segmentation on the points in the proposal. The discrete points after segmentation are used to predict the center of mass of the object and estimate the 3D bounding box of the object;

[0015] A 2D detector is used to generate 2D candidate regions from RGB images and classify the objects. The 2D regions are lifted to 3D space to generate a frustum proposal containing the object of interest. The detected categories are encoded as one-hot vectors and input into the point dense segmentation module to segment the object instances in the proposal.

[0016] The point dense segmentation module uses DenseNet as the backbone network for point dense segmentation. By retaining low-level features through dense connections, accurate instance segmentation in the three-dimensional space containing the target of interest is achieved. Since the positions of each frustum are different, the target points in one frustum may become occluded points or clutter points in another frustum. Therefore, the point dense segmentation module can not only learn occlusion and noise, but also recognize the geometric shapes of specific category objects;

[0017] The multi-scale centroid estimation module is used to estimate the center points of three-dimensional objects at different scales, and then through coordinate transformation, the predicted center points are made the origin to solve the problem that T-Net may estimate the center position inaccurately;

[0018] At the top of the network, the predicted 3D bounding boxes of the objects are obtained through a detection head designed for the given object point cloud. The output of the detection head contains the parameters defining the 3D bounding box, rather than the classification scores.

[0019] Preferably, one-hot vectors are used in the point dense segmentation module, where the k dimensions of the predefined k categories are used to encode semantic categories to replace the classification layer in DenseNet, which is used to reduce the loss of spatial geometric information and connect the one-hot vectors with the intermediate point cloud features.

[0020] Preferably, the point dense segmentation module includes at least four dense blocks to learn features, and a transition layer is added between each dense block. The transition layer includes a Batch Normalization layer, a ReLu layer, a 1×1 convolutional layer, and a max pooling layer;

[0021] The dense block introduces direct connections from any layer to all subsequent layers. Therefore, the layer receives the features of all previous layers as input:

[0022]

[0023] where represents the feature string of the layer;

[0024] represents a single tensor concatenating multiple inputs;

[0025] Due to the use of dense connections, the number of channels increases sharply. A transition layer is introduced to halve the feature dimension to control the model complexity;

[0026] The feature resolutions of the initial convolutional layer and the max pooling layer also remain unchanged to better preserve the texture features.

[0027] Preferably, the multi-scale centroid estimation module consists of 4 main feature extraction blocks: a convolutional block, a multi-scale feature block, a deconvolutional block, and a fully connected block. It should be noted that different resolutions of features are not generated by different convolutional blocks. Instead, in the multi-scale feature block, the point cloud features of the segmented object are fused through downsampling and convolutional operations. The low-level features generated by downsampling are concatenated with the one-hot encoded class vectors and then convolved and fused to generate high-level features. Since the spatial dimension of the high-level features has been reduced, it is difficult to directly use them for output generation. Therefore, a deconvolutional block is used to remap the high-level features to a higher resolution in the original input space, thereby realizing object centroid estimation and producing a finer and more accurate output.

[0028] Preferably, the parameters of the 3D bounding box at least include the center of the 3D bounding box , the bounding box size and the heading angle .

[0029] Preferably, for the bounding box size, the parameters of the 3D bounding box are parameterized as N;

[0030] For the box size, N anchor boxes of size S are predefined. The heading angle is discretized into N heading bins of size L within . These predefined anchor boxes and heading bins are classified into predefined categories, and the number of predicted parameters for each category is determined. Finally, the total number of output parameters formula is:

[0031] .

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] The present invention integrates LiDAR point clouds and RGB images, overcomes the problem of poor detection performance for small objects such as pedestrians and cyclists, while maintaining the detection accuracy for cars. Moreover, a point dense segmentation module and a central point estimation method for multi-scale features are introduced to retain the texture information of objects and improve the detection accuracy. The proposed method performs better than existing methods on the KITTI dataset, demonstrating its effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0035] Figure 1 is a schematic diagram of the prior art;

[0036] Figure 2 It is a structural schematic diagram of the overall architecture of the present invention;

[0037] Figure 3 It is a structural schematic diagram of the architecture of the point dense segmentation module of the present invention;

[0038] Figure 4 It is a structural schematic diagram of the multi-scale centroid estimation module of the present invention;

[0039] Figure 5 It is a schematic diagram of the detection result of the present invention. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0041] Embodiment 1:

[0042] Please refer to Figures 1-5 , a 3D object detection method for fusing lidar point cloud and dense segmentation of images, at least including the following:

[0043] Build an application framework with frustum proposal generation, point dense segmentation module, multi-scale centroid estimation module and detection head;

[0044] Generate frustum proposals through 2D detection of a given scene's 2D image, and perform instance segmentation on the points in the proposals. The segmented discrete points are used to predict the centroid of the object and estimate the 3D bounding box of the object. The overall architecture of the network is as Figure 2 shown;

[0045] Generate 2D candidate regions from the RGB image through a 2D detector and classify the targets, lift the 2D regions to the 3D space to generate frustum proposals containing the targets of interest. The detected classes are encoded as one-hot vectors and input into the point dense segmentation module to segment the object instances in the proposals;

[0046] Perform point dense segmentation through the point dense segmentation module with DenseNet as the backbone network. Retain low-level features through dense connections to achieve accurate instance segmentation in the three-dimensional space containing the targets of interest. Since the positions of each frustum are different, the target points in one frustum may become occluded points or clutter points in another frustum. Therefore, the point dense segmentation module can not only learn occlusion and noise, but also recognize the geometric shapes of specific category objects. The architecture is as Figure 3 shown;

[0047] Estimate the center points of 3D objects at different scales through a multi-scale centroid estimation module, and then make the predicted center points the origin through coordinate transformation to solve the problem that T-Net may inaccurately estimate the center position. The specific module structure is as Figure 4 shown;

[0048] At the top of the network, obtain the predicted 3D bounding boxes of the objects through a detection head designed for the given object point cloud. The output of the detection head contains the parameters defining the 3D bounding boxes, rather than the classification scores.

[0049] One-hot vectors are used in the point dense segmentation module, where the k dimensions of k predefined classes are used to encode semantic classes to replace the classification layer in DenseNet, which is used to reduce the loss of spatial geometric information and connect the one-hot vectors with the intermediate point cloud features;

[0050] The point dense segmentation module includes at least four dense blocks to learn features, and a transition layer is added between each dense block;

[0051] The dense block introduces direct connections from any layer to all subsequent layers. Therefore, the layer receives the features of all previous layers as input:

[0052]

[0053] where denotes the feature string of the layer;

[0054] denotes a single tensor concatenating multiple inputs;

[0055] Due to the use of dense connections, the number of channels increases sharply. A transition layer is introduced to halve the feature dimension to control the model complexity;

[0056] The feature resolutions of the initial convolutional layer and the max pooling layer also remain unchanged to better preserve the texture features;

[0057] The multi-scale centroid estimation module consists of 4 main feature extraction blocks: a convolutional block, a multi-scale feature block, a deconvolutional block, and a fully connected block. It should be noted that different resolutions of features are not generated by different convolutional blocks. Instead, in the multi-scale feature block, the point cloud features of the segmented objects are fused through downsampling and convolutional operations. The low-level features generated by downsampling are concatenated with the one-hot encoded class vectors and then convolved and fused to generate high-level features. Since the spatial dimension of the high-level features has been reduced, it is difficult to directly use them for output generation. Therefore, the deconvolutional block is used to remap the high-level features to a higher resolution in the original input space, thereby realizing object centroid estimation and producing a finer and more accurate output;

[0058] The parameters of the 3D bounding box include at least the center of the 3D bounding box , the bounding box size and the heading angle ;

[0059] The bounding box size parameterizes the N of the 3D bounding box;

[0060] For the box size, N anchor boxes of size S are predefined. The heading angle is discretized into N heading bins of size L within . These predefined anchor boxes and heading bins are classified into predefined categories, and the number of predicted parameters for each category is determined. Finally, the total number of output parameters formula is:

[0061] .

[0062] Example 2:

[0063] All experiments and evaluations were conducted on the RGB images of 7481 training pairs and 7518 test pairs and the point clouds of three object classes, namely cars, pedestrians, and cyclists, on the KITTI benchmark dataset using the present invention.

[0064] For each category, the detection results were evaluated according to 3 difficulty levels, namely easy, medium, and hard.

[0065] Since the ground truth of the test set is not available, the existing work was followed, and the original training set was split into 3712 and 3769 new training sets and validation sets.

[0066] Following the workflow of the prior art, the results on the validation set at 11 recall rate thresholds were reported.

[0067] For the car, pedestrian, and cyclist categories, the present invention uses the official 3DIoU evaluation metrics of 0.7, 0.5, and 0.5.

[0068] The proposed method was compared with the current state-of-the-art LiDAR-only method and multi-modal (LiDAR & RGB) methods.

[0069] Table 1 shows the AP results for 3D object detection on the KITTI validation set.

[0070] Table 1

[0071]

[0072] As shown in Table 1, overall, the LiDAR-only method, which uses a single modality, has better detection performance for cars than the multi-modal method. Although the multi-modal method has more abundant available information, the calibration error between different sensors may affect data fusion. Therefore, there are still some challenges in applying the multi-modal method to the autonomous driving scenario. However, for small targets such as pedestrians and cyclists, since the multi-modal method can obtain more comprehensive and accurate data, the detection effect is relatively better than the LiDAR-only method.

[0073] Through a specially designed dense structure focusing on texture features and an effective center point estimation scheme, the method of the present invention has achieved state-of-the-art performance on the KITTI dataset, especially for challenging categories such as pedestrians and cyclists. For the moderately difficult pedestrian category that has received more attention, the method of the present invention is 0.93% higher than the second-best EPNet. In the cyclist category, the present invention significantly outperforms the existing methods, with an mAP improvement of 2.98% compared to the second-best method, PointRCNN. In particular, the improvement is 2.16% in the moderate mode and 6.94% in the difficult mode. Qualitative result examples of the present invention on the KITTI validation set are shown Figure 5 as follows. The present invention shows the detection results of three different scenarios mainly including cars, pedestrians, and cyclists. The original image is on the top, the detection results on the image are in the middle, and the detection results under LiDAR are at the bottom. Among them, red represents cars, green represents pedestrians, and blue represents cyclists.

[0074] Example 3:

[0075] This example is used to further disclose the overall operation process of the point dense segmentation module on the premise of the above two examples;

[0076] Refer to Figure 3 , taking n points as input, and using 4 dense blocks to extract the dense features of the frustum proposals. A transition layer is added after each dense block, and the transition layer includes a BN (Batch Normalization) layer, a ReLu layer, a 1×1 convolutional layer, and a max pooling layer. The transition layer can control the complexity of the output data.

[0077] After extracting features through four dense blocks, the feature information is concatenated with the one-hot classification vector, and the score of each point is output through a multi-layer perceptron (MLP). Among the output n×m information, the m dimensions represent m object classifications.

[0078] Embodiment 4:

[0079] This embodiment further discloses a method for estimating the center of multi-scale features on the premise of the above two embodiments;

[0080] Refer to Figure 5 , and the n×m point cloud information is masked through coordinate transformation to obtain the segmented object point cloud of m×c.

[0081] Taking the segmented object point cloud of m×c as the input, first extract features through a convolution, and then input it into the multi-scale module.

[0082] In the multi-scale module, first perform downsampling to generate low-level features. After concatenating the low-level features with the one-hot vector, then perform convolutional feature fusion to obtain high-level features.

[0083] The multi-scale module outputs high-level features, but the spatial dimension of the high-level features is reduced and it is difficult to directly use for output. Therefore, it is also necessary to pass through a deconvolution block (Deconv Block) to remap the high-level features to a higher resolution in the original input space. Then, pass through a fully connected layer to output the specific 3D bounding box parameters.

[0084] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, in any aspect, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. A 3D object detection method for dense segmentation that fuses lidar point clouds and images, characterized in that: At least include the following: Build an application framework with a frustum proposal generation, point dense segmentation module, multi-scale centroid estimation module, and detection head; Generate frustum proposals by performing 2D detection on the 2D image of a given scene, and perform instance segmentation on the points in the proposals. The segmented discrete points are used to predict the centroid of the object and estimate the 3D bounding box of the object; Generate 2D candidate regions from the RGB image through a 2D detector and classify the targets, lift the 2D regions to 3D space to generate frustum proposals containing the targets of interest. The detected classes are encoded as one-hot vectors and input into the point dense segmentation module to segment the object instances in the proposals; Perform point dense segmentation through the point dense segmentation module with DenseNet as the backbone network. Retain low-level features through dense connections to achieve accurate instance segmentation in the three-dimensional space containing the targets of interest; Estimate the center points of three-dimensional objects at different scales through the multi-scale centroid estimation module, and then make the predicted center points the origin through coordinate transformation; At the top of the network, obtain the predicted 3D bounding box of the object through a detection head designed for the given object point cloud. The output of the detection head contains the parameters defining the 3D bounding box; One-hot vectors are used in the point dense segmentation module, where the k dimensions of k predefined classes are used to encode semantic classes to replace the classification layer in DenseNet, which is used to reduce the loss of spatial geometric information and connect the one-hot vectors with the intermediate point cloud features; The point dense segmentation module includes at least four dense blocks to learn features, and a transition layer is added between each dense block. The transition layer includes a Batch Normalization layer, a ReLu layer, a 1×1 convolutional layer, and a max pooling layer; The dense block introduces direct connections from any layer to all subsequent layers. Thus, the layer receives the features of all previous layers as input: ; Among them denotes the feature string of the layer Represents a single tensor that concatenates multiple inputs; Due to the sharp increase in the number of channels caused by the use of dense connections, a transition layer is introduced to halve the feature dimension to control the model complexity; The feature resolutions of the initial convolutional layer and the max pooling layer also remain unchanged to maintain texture features; The multi-scale centroid estimation module consists of 4 main feature extraction blocks: a convolutional block, a multi-scale feature block, a deconvolution block, and a fully connected block. In the multi-scale feature block, the point cloud features of the segmented object are fused through downsampling and convolutional operations. The low-level features generated by downsampling are concatenated with the one-hot encoded class vectors and then convolutionally fused to generate high-level features.

2. The 3D object detection method for dense segmentation of fused lidar point cloud and image according to claim 1, wherein: The parameters of the 3D bounding box at least include the center of the 3D bounding box , the bounding box size and the heading angle . [[ID= ​ For the bounding box size, N anchor boxes of size S are predefined, and the heading angle is discretized into N heading bins of size L within . These predefined anchor boxes and heading bins are classified into predefined categories, and the number of parameters to be predicted for each category is determined. Finally, the formula for the total number of output parameters is: 。

Images