3D target detection method based on color point cloud

Abstract

The invention discloses a 3D target detection method based on color point cloud, and the method comprises the steps: firstly, collecting point cloud data through a laser radar, and collecting RGB data through a camera; then, a color point cloud generation module generates color point cloud data only containing a target object from the point cloud data and the RGB data; then, inputting the color point cloud data into a pseudo 2D image generation module to obtain a pseudo 2D image; and finally, inputting the pseudo 2D image into a feature fusion module, combining an overall loss function for constructing three types of losses, performing feature mapping through convolution and deconvolution layers, and predicting a bounding box and a category of an object. The feature fusion module comprises receptive field expansion, the receptive field expansion introduces self-defined expansion factors among all levels of a pyramid, jump connection is applied at the same time, and for low-level and high-level features, an expansion feature fusion network is used for feature fusion. According to the 3D target detection method based on the color point cloud, processing of the point cloud is accelerated, and the bounding box and the category of the target are predicted more accurately.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a 3D target detection method based on color point cloud. Background Art

[0002] Object detection is the most critical and fundamental prerequisite for artificial intelligence applications such as robotics, autonomous driving, and augmented reality. Its purpose is to locate and classify objects with bounding boxes in 3D space for subsequent path planning.

[0003] Smart driving cars are usually equipped with RGB cameras and 3D LiDAR sensors, which can provide information for perceiving the surrounding environment. RGB images can provide rich color and texture information, but cannot provide depth information, making it difficult to accurately detect targets in three-dimensional space. Although LiDAR can capture the three-dimensional structure and accurate depth information of the target's geometric information, it lacks color and texture information, and usually becomes more sparse as the distance from the center of the scanner increases. In order to apply deep learning methods to 3D point clouds, researchers have proposed different representations of point clouds, which can be specifically divided into 2D projection-based, voxel grid-based, and multimodal data fusion methods. The common disadvantage of 2D projection and voxel grid is that they inevitably lead to the loss of important 3D information. Among them, the PointPillars algorithm processes point clouds through the concept of pillars, but due to limited resolution, it will cause a certain degree of information loss.

[0004] In direct modeling of point clouds, although PointNet can extract features from a small number of point clouds, its scalability on larger point clouds is still unclear. For example, small targets such as vehicles and pedestrians do not have unique geometric structures, and the number of point clouds is small, making it difficult to distinguish them from other targets in the environment. Multimodal data fusion methods benefit from the complementary characteristics of point clouds and images, and can effectively and accurately locate in three-dimensional space, solving the problem of difficult detection of small distant targets and occlusions. However, due to the different perspectives of the feature maps, the correspondence between pixels cannot be one-to-one. In addition, for point cloud data at different distances, the preset receptive field size is fixed, and there will be information missing or redundancy in the network.

[0005] Based on multimodal data fusion, we can take advantage of the complementary advantages of different modal data, make up for each other's shortcomings, greatly improve the perception level in complex traffic environments, and enable smart cars to have robust traffic scene perception capabilities. Multimodal fusion strategies are usually divided into pixel-level fusion, feature-level fusion, and decision-level fusion strategies. Pixel-level fusion directly integrates the observation data of different sensors to create a new multi-channel data type; the feature-level fusion strategy extracts information multiple times in different feature layers, which will cause feature redundancy and make it difficult to achieve point correspondence between the two modal features; the decision-level fusion method uses different feature learning network models to extract features of different modalities, and there is less interactivity between different modalities.

[0006] Since the pixel-level fusion strategy can map the two modal data into a unified coordinate system, the fusion of multi-dimensional color point cloud data can be achieved; then the generated color point cloud data is sent to the feature extraction network with expanded receptive field. This not only reduces the amount of data, but also improves the efficiency of the feature extraction network. However, early fusion methods have not been deeply explored. Summary of the invention

[0007] Purpose of the invention: In order to overcome the above shortcomings, the purpose of the present invention is to provide a 3D target detection method based on color point cloud, taking lidar point cloud data and camera RGB images as research objects, and converting the two modal data into a unified scene representation space according to the different structures of the two modal data, thereby accelerating the processing of point cloud and more accurately predicting the bounding box and category of the target.

[0008] Technical solution: In order to achieve the above object, the present invention provides a 3D target detection method based on color point cloud, comprising the following steps:

[0009] S1): Use LiDAR to collect point cloud data, and the camera to collect RGB data;

[0010] S2): generating color point cloud data containing only the target object from the point cloud data and RGB data through a color point cloud generation module;

[0011] S3): inputting the color point cloud data into a pseudo 2D image generation module to generate a pseudo 2D image;

[0012] S4): The pseudo 2D image is input into the feature fusion module and combined with L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir The loss constructs an overall loss function and performs feature mapping through convolution and deconvolution layers to predict the bounding box and category of the object;

[0013] S401): The feature fusion module includes receptive field expansion; the receptive field expansion introduces a custom expansion factor between each level of the pyramid, and a jump connection is applied at the same time, the convolution size is 3×3, and when the jump connection layer is T, the expansion factor is 2T; for low-level and high-level features, an expanded feature fusion network is used for feature fusion.

[0014] Furthermore, the feature fusion module in S401) further includes an encoder and a decoder; the encoder extracts pyramid features based on ResNet-50; the decoder includes ResBlock and upsampling; the output uses a SoftMax function. The encoder extracts pyramid features based on ResNet-50 to obtain high-level features with multiple receptive fields; the semantic features further generate pyramid features to provide rich multi-scale information for each layer.

[0015] Furthermore, the overall loss function includes: L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir loss;

[0016] Position regression loss L loc The formula is as follows:

[0017]

[0018] Where x is the coordinate of the center point of the bounding box;

[0019] Target classification loss L cls The formula is as follows:

[0020] L cls = -α(1-p α ) γ log(p α )

[0021] Among them, p is the classification probability of the prediction box, the parameter α is used to balance the classification categories, and γ is a hyperparameter;

[0022] Object direction (heading angle) loss L dir , use the softmax loss function to determine the object orientation information, the formula is as follows:

[0023] L dir =smooth_l1(sin(θ v -θ t ))

[0024] Among them, θ v Represents the heading angle predicted by the model, which is a constant output by the model and indicates the orientation of the object; θt is the true heading angle (target heading angle), which represents the desired output in the training data;

[0025] The overall loss function formula is as follows:

[0026]

[0027] Among them, N pos is the number of correctly detected bounding boxes, β loc , β cls and β dir are the weights for regression, classification and direction, respectively.

[0028] Furthermore, the target classification loss L cls The focal loss function is used to solve the imbalanced distribution of foreground and background.

[0029] Furthermore, the color point cloud generation module in S2) includes a pixel-level fusion module; the formula of the pixel-level fusion module is:

[0030]

[0031] p cam =T proj ·P cam (2)

[0032]

[0033] Among them, P cam is the matrix in the camera coordinate system, P lidar is the matrix in the laser radar coordinate system, is the rotation matrix, is the transformation matrix, is the rotation matrix from the camera coordinate system to the lidar coordinate system, is the transformation matrix from the camera coordinate system to the lidar coordinate system; T proj is the projection matrix, obtained by joint calibration of the camera and lidar;

[0034] Although the point cloud is sparse, the limited pixels can still provide rough texture information for the point cloud, especially for close-range objects, which can form rich texture features;

[0035] According to the camera's intrinsic and extrinsic parameter matrices and formulas (1), (2) and (3), the image pixels are projected onto the corresponding point cloud data in the three-dimensional space, and the color features of the image pixels are assigned to the corresponding 3D point cloud to generate 7-dimensional color point cloud XYZrRGB data, as shown below:

[0036] p i =(xi ,y i ,z i ,r i ,R i ,G i ,B i )

[0037] Among them, i is the i-th data, x, y, z are the three-dimensional coordinates of the radar point cloud, r is the density data, r, g, b are the RGB3 channel data.

[0038] Furthermore, the detection range is limited to the area 70m in front of the laser radar and 40m on the left and right, that is, {[x,y,z] T |x∈[0,70]m,y∈[–40,40]m,z∈[–3,3]m}, which improves the real-time performance and reduces unnecessary calculations.

[0039] Furthermore, the color point cloud generation module also includes a height filtering module; the height filtering module, for any point cloud data P, according to the corresponding reference point P r The height difference is used to judge the category. The formula is as follows:

[0040]

[0041] Among them, the parameter z i is the height of point P, parameter z j is the reference point P r The height value, parameter t h is the height difference threshold; according to the height difference relationship between point cloud data, the point category of point cloud data can be determined; when the return value of the above formula is 0, the category of point P is determined to be a target object and it is retained; otherwise, it is a road point and is filtered. RGB images provide color and texture information of road environment information, and 3D point clouds present the spatial information of the target scanned by the sensor and its surrounding environment; multi-dimensional color point clouds significantly enhance the color and texture information of 3D point clouds.

[0042] Furthermore, the pseudo 2D image generation module in S3) divides the color point cloud data evenly into a 2D grid shape (H, W) with a resolution of r. The non-empty grid is resampled into N Pillar grids. After adding color texture features, the feature representation of each Pillar grid is:

[0043] (x v ,y v ,z v ,r v ,R v ,G v ,B v )

[0044] Next, (Cin ,N) as the input of the basic PointNet network, where C in =7, the output is:

[0045] (N,C out )

[0046] Among them, C out is a tensor of features or pillar features extracted from each non-empty cell, thus creating a tensor of size (H, W, C out ). By converting 3D point cloud data into 2D pseudo images, relatively mature 2D object detection methods can be used for fast and accurate object detection, reducing the complexity of calculation.

[0047] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0048] 1. The present invention provides a 3D target detection method based on color point cloud, constructs a multi-dimensional color point cloud data for filtering ground point cloud data, and accelerates the processing of point cloud;

[0049] 2. The present invention provides a 3D target detection method based on color point cloud. In the feature fusion module, an enlarged receptive field network model for multi-scale feature fusion is proposed, which accurately retains the effective information in the field of view while maintaining high computational efficiency, and more accurately predicts the bounding box and category of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the overall process of a 3D target detection method based on color point cloud according to the present invention;

[0051] Figure 2 A feature extraction network model diagram for increasing the receptive field in a 3D target detection method based on color point cloud according to the present invention;

[0052] Figure 3 A flowchart of generating 7D color point cloud data in a 3D target detection method based on color point cloud according to the present invention;

[0053] Figure 4 A visualization diagram of a color point cloud in a 3D target detection method based on a color point cloud according to the present invention (A-RGB original image; B-panoramic view of a 3D point cloud; C-projected image; D-3D point cloud within the image field of view; E-color point cloud within the image field of view; F-color point cloud with ground data filtered);

[0054] Figure 5This is the visualization result of the network model and the true value in the 3D object detection method based on color point cloud described in the present invention (purple is the visualization result of the true value, and blue is the visualization result of the network prediction);

[0055] Figure 6 This is the visualization result of the 3D object detection method based on color point cloud described in the present invention on the KITTI 0001 sequence set. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0057] Example

[0058] In this embodiment, Figure 1 The present invention discloses a 3D target detection method based on color point cloud, comprising the following steps:

[0059] S1): Use LiDAR to collect point cloud data, and the camera to collect RGB data;

[0060] S2): generating color point cloud data containing only the target object from the point cloud data and RGB data through a color point cloud generation module;

[0061] S3): inputting the color point cloud data into a pseudo 2D image generation module to generate a pseudo 2D image;

[0062] S4): The pseudo 2D image is input into the feature fusion module and combined with L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir The loss constructs an overall loss function and performs feature mapping through convolution and deconvolution layers to predict the bounding box and category of the object;

[0063] S401): The feature fusion module includes receptive field expansion; the receptive field expansion introduces a custom expansion factor between each level of the pyramid, and a jump connection is applied at the same time, the convolution size is 3×3, and when the jump connection layer is T, the expansion factor is 2T; for low-level and high-level features, an expanded feature fusion network is used for feature fusion.

[0064] In this embodiment, if Figure 2, the feature fusion module in S401) also includes an encoder and a decoder; the encoder is based on pyramid feature extraction of ResNet-50; the decoder includes ResBlock and upsampling; the output uses a SoftMax function.

[0065] In this embodiment, the overall loss function includes: L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir loss;

[0066] Position regression loss L loc The formula is as follows:

[0067]

[0068] Where x is the coordinate of the center point of the bounding box;

[0069] Target classification loss L cls The formula is as follows:

[0070] L cls = -α(1-p α ) γ log(p α )

[0071] Wherein, p is the classification probability of the prediction box; the parameter α is used to balance the classification categories, which is set to 0.25 in this embodiment; γ is a hyperparameter, which is set to 2 in this embodiment;

[0072] Object direction (heading angle) loss L dir , use the softmax loss function to determine the object orientation information, the formula is as follows:

[0073] L dir =smooth_l1(sin(θ v -θ t ))

[0074] Among them, θ v Represents the heading angle predicted by the model, which is a constant output by the model and indicates the orientation of the object; θ t is the true heading angle (target heading angle), which represents the desired output in the training data;

[0075] The overall loss function formula is as follows:

[0076]

[0077] Among them, N pos is the number of correctly detected bounding boxes, β loc , βcls and β dir They are the weights for regression, classification and direction respectively, which are set to 2.0, 1.0 and 0.2 in this embodiment.

[0078] Specifically, the 3D bounding box parameters are defined by (x, y, z, w, l, h, θ), where x, y, z are the coordinates of the center point of the bounding box, w, l, h are the width, length, and height of the 3D bounding box respectively; θ is the object orientation (heading angle); the parameters involved are as follows:

[0079]

[0080] Among them, Δx, Δy, Δz are the offsets between the true value box and the predicted box, the subscripts g and a represent the true value box and the predicted box respectively; d represents the diagonal length of the box, which can be obtained by Normalize.

[0081] In this embodiment, if Figure 3 The color point cloud generation module in S2) includes a pixel-level fusion module; the formula of the pixel-level fusion module is:

[0082]

[0083] p cam =T proj ·P cam (2)

[0084]

[0085] Among them, P cam is the matrix in the camera coordinate system, P lidar is the matrix in the laser radar coordinate system, is the rotation matrix, is the transformation matrix, is the rotation matrix from the camera coordinate system to the lidar coordinate system, is the transformation matrix from the camera coordinate system to the lidar coordinate system; T proj is the projection matrix, obtained by joint calibration of the camera and lidar;

[0086] According to the camera's intrinsic and extrinsic parameter matrices and formulas (1), (2) and (3), the image pixels are projected onto the corresponding point cloud data in the three-dimensional space, and the color features of the image pixels are assigned to the corresponding 3D point cloud to generate 7-dimensional color point cloud XYZrRGB data, as shown below:

[0087] p i =(x i ,y i ,z i ,r i ,Ri ,G i ,B i )

[0088] Among them, i is the i-th data, x, y, z are the three-dimensional coordinates of the radar point cloud, r is the density data, r, g, b are the RGB3 channel data.

[0089] Specifically, each 7D color lidar point cloud contains not only three-dimensional coordinates (3D) and reflection intensity (1D), but also the color information (3D) of the corresponding pixel points on the image plane; through joint calibration, the point cloud data and image data are aligned in the same coordinate system, the point cloud coordinates are converted and projected onto the image to ensure that the spatial positions of the two are consistent.

[0090] In this embodiment, the detection range is limited to the area 70m forward and 40m to the left and right of the laser radar, that is, {[x, y, z] T |x∈[0,70]m,y∈[–40,40]m,z∈[–3,3]m}.

[0091] Specifically, the detection range limitation can be adjusted according to the complexity of different scenes to filter out irrelevant points outside the camera's field of view.

[0092] In this embodiment, if Figure 3 The color point cloud generation module also includes a height filtering module; the height filtering module is for any point cloud data P, according to the corresponding reference point P r The height difference is used to judge the category. The formula is as follows:

[0093]

[0094] Among them, the parameter z i is the height of point P, parameter z j is the reference point P r The height value, parameter t h is the height difference threshold, in this embodiment, t h Set to 1; According to the height difference relationship between the point cloud data, the point category of the point cloud data can be determined; When the return value of the above formula is 0, the category of point P is determined to be a target object and it is retained; otherwise, it is a road point and it is filtered out.

[0095] Specifically, Figure 4 , RGB images provide color and texture information of road environment information, and 3D point clouds present the spatial information of the target scanned by the sensor and its surrounding environment; multi-dimensional color point clouds significantly enhance the color and texture information of 3D point clouds; the constructed 7-dimensional color point cloud data not only retains the spatial characteristics of the point cloud, but also enriches the color and texture information of the surface points. After limiting the detection area, the computational complexity of data processing is greatly reduced.

[0096] In this embodiment, if Figure 1 The pseudo 2D image generation module in S3) divides the color point cloud data into a 2D grid shape (H, W) with a resolution of r. The non-empty grid is resampled into N Pillar grids. After adding color texture features, the feature representation of each Pillar grid is:

[0097] (x v ,y v ,z v ,r v ,R v ,G v ,B v )

[0098] Next, (C in ,N) as the input of the basic PointNet network, where C in =7, the output is:

[0099] (N,C out )

[0100] Among them, C out is a tensor of features or pillar features extracted from each non-empty cell, thus creating a tensor of size (H, W, C out )’s 2D pseudo image.

[0101] To evaluate the performance of our approach, we used a single Nvidia RTX3090 GPU and pytorch 1.3 on a local Ubtuntu 18.04 system for model training and deployment.

[0102] The detection performance of three categories, vehicles, pedestrians and cyclists, was verified and evaluated on the KITTI public dataset, and compared with other advanced methods. 3712 groups of samples were used for training and 3769 groups of samples were used for verification. In the training part, the deep network used the Adam optimizer to update the model parameters, with a momentum range of 0.95 to 0.85, an initial learning rate of 0.001, a fixed weight decay coefficient of 0.0001, a batch size of 12, and a training epoch of 300, which took about 30 hours. In the model evaluation process, only the method in this paper involved 7D color point cloud as network input, while other methods used 3D point cloud data.

[0103] 1) Comparative experiment of multimodal data fusion and single modal data input

[0104] In order to verify the effectiveness of the model under different data combinations, 3D point cloud and 7D color point cloud are used as network input on the KITTI validation set, and the detection results are shown in Table 1. When 3D point cloud is used as network input, the network input channel is changed to 3.

[0105] Table 1. Comparison of detection accuracy of different input data on the KITTI validation set (%)

[0106]

[0107] According to the results in Table 1, it can be observed that in the three categories of vehicles, pedestrians and cyclists, the detection results using 7D color point clouds as input are higher than those using 3D point clouds. In particular, the accuracy improvement of using 7D color point clouds is more obvious in detecting pedestrians and cyclists, which is 1.37% and 1.75% higher at the "medium" difficulty level, respectively. The reason is that color information makes the input semantic information richer, and the use of a feature network with an increased receptive field has certain advantages in detecting distant targets.

[0108] In addition, this paper compares the prediction results with the three-dimensional bounding box renderings of the true value labels, and the visualization results are as follows Figure 5 As shown in Figure 1. The magenta color represents the ground-truth 3D bounding box, and the blue color represents the predicted 3D bounding box. Figure 5 It can be seen that the accuracy of the center point and length, width and height of the three-dimensional bounding box is infinitely close to the true value label, indicating that the detection network based on multimodal data fusion has good accuracy and can alleviate the common missed detection and false detection problems in traffic scenes.

[0109] 2) Comparison with other methods

[0110] Since KITTI does not provide the true value of the test set, for unlabeled testing, the prediction results of the test set will be submitted to the official website for evaluation to obtain accuracy indicators. In addition, this model (using 7D data) is compared with the current advanced 3D target detection methods, including lidar-based methods (BirdNet+, Pointpillars, VoxelNet, SECOND, and SegVoxelNet) and algorithms based on multimodal data fusion (MV3D, AVOD, MMF, MVAF-Net, Contfuse, and F-PointNet). Table 2 lists the evaluation results (average accuracy, mAP) of several 3D object detectors from the official KITTI leaderboard on the KITTI test set. The difficulty levels of "easy", "medium", and "difficult" are defined by the KITTI official website. The best results in each category are shown in bold.

[0111] Table 2. Object detection accuracy of different algorithms on the KITTI data test set (%)

[0112]

[0113] In order to intuitively display the detection effect of 7D color point cloud, Figure 6 The visualization results of the proposed method on the KITTI 0001 sequence set are shown, where the red bounding box represents the cyclist and the blue bounding box represents the vehicle. It can be seen that the proposed method has a good effect on target detection at close distances, and can still accurately identify and locate targets even when there is partial truncation and severe occlusion between targets.

[0114] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A 3D object detection method based on color point cloud, characterized in that: The steps include: S1): Use LiDAR to collect point cloud data, and the camera to collect RGB data; S2): generating color point cloud data containing only the target object from the point cloud data and RGB data through a color point cloud generation module; S3): inputting the color point cloud data into a pseudo 2D image generation module to generate a pseudo 2D image; S4): The pseudo 2D image is input into the feature fusion module and combined with L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir The loss constructs an overall loss function and performs feature mapping through convolution and deconvolution layers to predict the bounding box and category of the object; S401): The feature fusion module includes receptive field expansion; the receptive field expansion introduces a custom expansion factor between each level of the pyramid, and a jump connection is applied at the same time, the convolution size is 3×3, and when the jump connection layer is T, the expansion factor is 2T; for low-level and high-level features, an expanded feature fusion network is used for feature fusion.

2. The 3D target detection method based on color point cloud according to claim 1, characterized in that: The feature fusion module in S401) further includes an encoder and a decoder; The encoder is based on pyramid feature extraction of ResNet-50; the decoder includes ResBlock and upsampling; The output uses the SoftMax function.

3. The 3D target detection method based on color point cloud according to claim 1, characterized in that: The overall loss function includes: L for position regression loc loss, L for object classification cls loss and L for object orientation (heading angle) regression dir loss; Position regression loss L loc The formula is as follows: Where x is the coordinate of the center point of the bounding box; Target classification loss L cls The formula is as follows: L cls =-α(1-p α ) γ log(p α ) Among them, p is the classification probability of the prediction box, the parameter α is used to balance the classification categories, and γ is a hyperparameter; Object direction (heading angle) loss L dir , use the softmax loss function to determine the object orientation information, the formula is as follows: L dir =smooth_l1(sin(θ v -θ t )) Among them, θ v Represents the heading angle predicted by the model, which is a constant output by the model and indicates the orientation of the object; θ t is the true heading angle (target heading angle), which represents the desired output in the training data; The overall loss function formula is as follows: Among them, N pos is the number of correctly detected bounding boxes, β loc , β cls and β dir are the weights for regression, classification and direction, respectively.

4. The 3D target detection method based on color point cloud according to claim 1, characterized in that: The color point cloud generation module in S2) includes a pixel-level fusion module; The formula of the pixel-level fusion module is: p cam =T proj ·P cam (2) Among them, P cam is the matrix in the camera coordinate system, P lidar is the matrix in the laser radar coordinate system, is the rotation matrix, is the transformation matrix, is the rotation matrix from the camera coordinate system to the lidar coordinate system, is the transformation matrix from the camera coordinate system to the lidar coordinate system; T proj is the projection matrix, obtained by joint calibration of the camera and lidar; According to the camera's intrinsic and extrinsic parameter matrices and formulas (1), (2) and (3), the image pixels are projected onto the corresponding point cloud data in the three-dimensional space, and the color features of the image pixels are assigned to the corresponding 3D point cloud to generate 7-dimensional color point cloud XYZrRGB data, as shown below: p i =(x i ,y i ,z i ,r i ,R i ,G i ,B i ) Among them, i is the i-th data, x, y, z are the three-dimensional coordinates of the radar point cloud, r is the density data, r, g, b are the RGB3 channel data.

5. The 3D target detection method based on color point cloud according to claim 4, characterized in that: The detection range is limited to the area 70m in front of the laser radar and 40m on the left and right, that is, {[x,y,z] T |x∈[0,70]m,y∈[–40,40]m,z∈[–3,3]m}.

6. The 3D target detection method based on color point cloud according to claim 4, characterized in that: The color point cloud generation module also includes a height filtering module; The height filtering module is for any point cloud data P, according to the corresponding reference point P r The height difference is used to judge the category. The formula is as follows: Among them, the parameter z i is the height of point P, parameter z j is the reference point P r The height value, parameter t h is the height difference threshold; according to the height difference relationship between the point cloud data, the point category of the point cloud data can be determined; when the return value of the above formula is 0, the category of point P is determined to be a target object and it is retained; otherwise, it is a road point and is filtered out.

7. The 3D target detection method based on color point cloud according to claim 1, characterized in that: The pseudo 2D image generation module in S3) divides the color point cloud data into a 2D grid shape (H, W) with a resolution of r. The non-empty grid is resampled into N Pillar grids. After adding color texture features, the feature representation of each Pillar grid is: (x v ,y v ,z v ,r v ,R v ,G v ,B v ) Next, (C in , N) as the input of the basic PointNet network, where C in =7, the output is: (N,C out ) Among them, C out is a tensor of features or pillar features extracted from each non-empty cell, thus creating a tensor of size (H, W, C out )’s 2D pseudo image.

Images