An anchor-free 3D object detection method based on multi-sensor fusion

Abstract

The present invention relates to a method for anchor-free 3D target detection based on multi-sensor fusion, comprising: acquiring a color RGB image and a laser point cloud; performing semantic segmentation on the color RGB image to obtain category information of each pixel; enhancing the features of the laser point cloud using the category information to obtain an enhanced laser point cloud; encoding the point cloud geometric features and the point cloud visibility features of the enhanced laser point cloud to obtain geometric features and visibility features; stacking the geometric features and the visibility features to obtain stacked features; inputting the stacked features into a multi-layer feature extraction network to extract feature information of different levels, and then stacking the feature information of each level to obtain fused features; outputting the fused features to an anchor-free target detector to obtain a 3D target detection result. Compared with the prior art, the present invention enhances the 3D target detection performance by fusing multimodal data and utilizing the complementary advantages between sensors, thereby achieving the purpose of accurate and rapid detection.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to an anchor-frame-free 3D target detection method based on multi-sensor fusion. Background Art

[0002] 3D object detection is widely used in scenarios such as unmanned driving, robotics, and augmented reality. Compared with ordinary 2D detection, 3D detection provides additional information about the length, width, height, and deflection angle of the target object, and is an important perception basis for three-dimensional scene understanding and autonomous decision-making and planning.

[0003] LiDAR and cameras are the most commonly used on-board sensors in the field of 3D target detection. The images collected by cameras have rich semantic and contextual information, while LiDAR can obtain accurate spatial information. Both are mainstream sensors for autonomous driving. In addition, many current autonomous driving algorithms use sensor fusion, which can combine the advantages of different sensors to deal with problems such as poor lighting and occlusion.

[0004] In addition, target detection algorithms mainly include one-stage and two-stage. Two-stage means that the detection algorithm needs to be completed in two steps: first, the candidate area needs to be obtained, and then the classification is performed, such as the R-CNN series; in contrast, one-stage detection can be understood as one-step, without the need to find candidate areas separately, and typical examples include SSD. In two-stage detection algorithms, such as the Faster R-CNN algorithm, candidate boxes are first generated, and then each candidate box is classified (the position is also corrected). The detection speed of this type of algorithm is slow because it needs to run the detection and classification process multiple times; while the one-stage detection method only needs to be sent to the network once to predict all bounding boxes, so it is fast and very suitable for mobile terminals. The above algorithms all need to set carefully formulated anchor boxes for target detection in different areas and sizes. Researchers generally believe that pre-defined parameters are the key to the success of target detection models. In related experiments in the past, researchers have also proved that the hyperparameters of the anchor points have a considerable impact on the prediction ability of the model.

[0005] However, the above-mentioned anchor-frame-based target detection methods have the following shortcomings:

[0006] (1) Hyperparameters such as the size, ratio, and number of anchor boxes are difficult to adjust, and the accuracy obtained with different hyperparameters can fluctuate by about 4%. The hyperparameters of the preset anchor boxes reduce the robustness of the network model. If applied to a new dataset, the hyperparameters of the anchor boxes need to be redesigned, which increases the complexity of model parameter debugging.

[0007] (2) Even after careful design, a serious problem will arise because the proportion and size of the anchor box must be fixed when the model is established. That is, the detection difficulty of the model is significantly increased when the detection model is used for target sets with large shape variations, especially small targets.

[0008] (3) In order to further obtain a better recall rate, the model usually lays dense anchor boxes on each feature layer. However, most of the anchor boxes will be labeled as negative samples during the training process. Too many negative samples aggravate the imbalance in the number of positive and negative samples of the model, making the model insensitive to the background. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an anchor-free 3D target detection method based on multi-sensor fusion, so as to accurately and quickly perform 3D target detection.

[0010] The purpose of the present invention can be achieved by the following technical solution: A method for detecting 3D objects without anchor frames based on multi-sensor fusion, comprising the following steps:

[0011] S1, obtain color RGB image and laser point cloud;

[0012] S2, perform semantic segmentation on the color RGB image to obtain the category information of each pixel;

[0013] S3, using the category information of each pixel to enhance the features of the laser point cloud to obtain an enhanced laser point cloud;

[0014] S4. According to the enhanced laser point cloud, point cloud geometric feature encoding and point cloud visibility feature encoding are performed to obtain geometric features and visibility features respectively;

[0015] S5. Stacking the geometric features and the visibility features to obtain stacked features;

[0016] S6, inputting the stacked features into a multi-layer feature extraction network, extracting feature information at different levels, and then stacking the feature information at each level to obtain a fused feature;

[0017] S7. Output the fused features to the anchor-free object detector to obtain the 3D object detection result.

[0018] Furthermore, the step S2 specifically uses a semantic segmentation network to perform semantic segmentation on the color RGB image, and the semantic segmentation network includes U-Net, SegNet, DeepLab or BiSeNet;

[0019] The category information of each pixel in step S2 includes four categories, namely, vehicle, pedestrian, non-motor vehicle and background.

[0020] Furthermore, the specific process of step S3 is: by projecting transformation from the laser radar coordinate system to the camera coordinate system and then from the camera coordinate system to the image coordinate system, the spatial point cloud is associated with the pixel position of the corresponding color RGB image, and the semantic information represented by the color RGB image pixels is supplemented to each point cloud projected from the laser radar coordinate system to the image coordinate system, which is equivalent to adding a new data dimension to the original input laser point cloud and giving category attributes to the laser point cloud, thereby enhancing the semantics of the point cloud.

[0021] Furthermore, the specific process of performing point cloud geometric feature encoding in step S4 is as follows:

[0022] First, a three-dimensional grid is set on the horizontal plane within the sensing range of the vehicle. The number of the three-dimensional grid along the X and Y axes is W and H respectively, and the number of the Z axis is 1, so as to form a body column, and the laser point cloud is mapped to the corresponding body column;

[0023] The features of each laser point in the body column include (x, y, z, i, t, c, ox, oy, dx, dy, dz), where (x, y, z, i, t) are the original features of the laser point, corresponding to the coordinate value, reflectivity, and timestamp of the spatial rectangular coordinate system, respectively. c is the category feature assigned by the semantic segmentation of the image. (ox, oy) is the deviation between the laser point and the central axis x, y of the body column. (dx, dy, dz) is the position mean of the laser point and all laser points. Deviation;

[0024] All enhanced laser points in the body column are feature encoded by a multi-layer perceptron to generate point cloud geometric features with a fixed feature quantity D, and a maximum pooling operation is performed in the quantity direction to encode and generate geometric features with a size of (W, H, D).

[0025] Furthermore, the specific process of encoding the point cloud visibility feature in step S4 is as follows:

[0026] First, a three-dimensional grid is set on a horizontal plane within the sensing range of the vehicle, wherein the number of the three-dimensional grid along the X and Y axes is W and H respectively, and the number of the three-dimensional grid along the Z axis is O, thereby forming voxels;

[0027] Each laser point in the laser point cloud forms a line segment with the center of the laser radar. The voxels that the line segment passes through are marked as free, the voxels where the laser point is located are marked as occupied, and the remaining voxels are marked as unknown;

[0028] All voxels form feature vectors along the Z-axis direction and are converted into visibility features of length K through a convolutional layer, thereby encoding a visibility feature size of (W, H, K).

[0029] Furthermore, the step S5 specifically stacks the geometric features and visibility features along the depth direction: the geometric features and visibility features are respectively mapped to the grid map generated by the bird's-eye view of the three-dimensional grid, and stacked along the depth direction to obtain a stacked feature with a size of (B, (D+K), H, W), where B, (D+K), H, and W correspond to batch, depth, height, and width, respectively.

[0030] Furthermore, in the step S6, the multi-layer feature extraction network specifically extracts feature maps of three different scales through convolutional layers of different step sizes, and the feature maps of each scale are then respectively subjected to deconvolution layers of corresponding step sizes to expand the size to the same size, and finally stacked along the depth direction to obtain fused features.

[0031] Furthermore, the anchor-free object detector in step S7 includes five detection heads, specifically a key point heat map detection head, a local offset detection head, a z-axis positioning detection head, a 3D object size detection head and a direction detection head.

[0032] Furthermore, the five detection heads are used to regress the fused features through the convolutional layers respectively to obtain the probability that each pixel point of the fused feature is the category center point, the offset in the X and Y directions from the true value center point, the offset in the Z direction from the true value center point, and the geometric size and orientation of the three-dimensional detection box.

[0033] Furthermore, the step S7 specifically uses five detection heads to predict the center of the object on the bird's-eye view plane and regress different attributes of the 3D bounding box. Finally, the output results of the five detection heads are merged together to generate a 3D target detection result.

[0034] Compared with the prior art, the present invention proposes a one-stage end-to-end 3D point cloud target detection method without anchor boxes and simple post-processing. The method mainly connects the fused feature map extracted by the backbone network to five different detection heads to predict the center of the object on the bird's-eye view plane and regress different attributes of the 3D bounding box. In the backbone network, the color RGB image is subjected to a semantic segmentation network to obtain the category of each pixel point, and the category information can expand the features of the laser point cloud. In addition, according to the laser beam propagation principle, the space between the sensor and the laser point is not blocked by obstacles and can be considered as a free area, so that the 3D target detection task can use image semantic features and spatial visibility features to improve the detection accuracy of 3D targets. Finally, a multi-layer feature extraction network is used to obtain fused features, thereby realizing the enhancement of the original point cloud data by image semantic information in the data pre-processing stage. A ray casting algorithm is used to successfully reconstruct the spatial visibility state of the point cloud. A convolutional neural network is used to aggregate the point set geometry and semantic features, and the point set visibility features, respectively, thereby ensuring the accuracy of 3D target detection.

[0035] The present invention combines aggregated point set geometric features, point set visibility features, and point set semantic features into an end-to-end three-dimensional object detection framework, which not only realizes the data fusion of point cloud and image, but also innovatively proposes the concept of spatial "visibility" of point cloud, effectively reconstructs the spatial occupancy state of point cloud, and helps to accurately perceive the real situation of three-dimensional space. In addition, in the inference stage, each hot peak will be picked out through the maximum pooling operation, and then no multiple regressed anchor points will be tiled to one position, so there is no need to use traditional NMS, which allows the entire detector to run on a typical CNN accelerator or GPU, saving CPU resources for other key tasks of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0037] Figure 2 3D target detection data processing flow chart in the embodiment;

[0038] Figure 3 Extract schematic diagram for semantic features and visibility features;

[0039] Figure 4 This is the overall network structure diagram for target detection;

[0040] Figure 5 Schematic diagram of voxel state labeling;

[0041] Figure 6 This is the structural diagram of the five detection head modules. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Example

[0044] like Figure 1 As shown, a method for anchor-free 3D target detection based on multi-sensor fusion includes the following steps:

[0045] S1, obtain color RGB image and laser point cloud;

[0046] S2. Perform semantic segmentation on the color RGB image to obtain the category information of each pixel. In practical applications, the semantic segmentation network of the mainstream framework can be used to perform semantic segmentation on the color RGB image, mainly including network architectures such as U-Net, SegNet, DeepLab or BiSeNet;

[0047] The color RGB image is input into the mainstream semantic segmentation network to obtain the category of each pixel in the color image. The main categories of traffic scenes include vehicles, pedestrians, non-motor vehicles, background, etc.

[0048] S3. Using the category information of each pixel, the features of the laser point cloud are enhanced to obtain an enhanced laser point cloud. The category information of each pixel is assigned to the laser point cloud through the external reference between sensors, so as to enhance the features of the laser point cloud. That is, the laser point cloud is aligned with each pixel of the color image through the external reference of the sensor, so as to assign category attributes to the laser point cloud.

[0049] The specific process is as follows: through the projection transformation from the laser radar coordinate system to the camera coordinate system and then from the camera coordinate system to the image coordinate system, the spatial point cloud is associated with the pixel position of the corresponding color RGB image, and the semantic information represented by the color RGB image pixel is added to each point cloud projected from the laser radar coordinate system to the image coordinate system, which is equivalent to adding a new data dimension to the original input laser point cloud and giving category attributes to the laser point cloud, thereby enhancing the semantics of the point cloud;

[0050] S4. According to the enhanced laser point cloud, point cloud geometric feature encoding and point cloud visibility feature encoding are respectively performed to obtain geometric features and visibility features respectively. The specific process of performing point cloud geometric feature encoding is as follows:

[0051] First, a three-dimensional grid is set on the horizontal plane within the sensing range of the vehicle. The number of three-dimensional grids along the X and Y axes is W and H respectively, and the number of Z axis is 1, so as to form a body column, and the laser point cloud is mapped to the corresponding body column;

[0052] The features of each laser point in the body column include (x, y, z, i, t, c, ox, oy, dx, dy, dz), where (x, y, z, i, t) are the original features of the laser point, corresponding to the coordinate value, reflectivity, and timestamp of the spatial rectangular coordinate system, respectively. c is the category feature assigned by the semantic segmentation of the image. (ox, oy) is the deviation between the laser point and the central axis x, y of the body column. (dx, dy, dz) is the position mean of the laser point and all laser points. Deviation;

[0053] All enhanced laser points in the body column are feature encoded by a multi-layer perceptron to generate point cloud geometric features with a fixed feature quantity D, and a maximum pooling operation is performed in the quantity direction, so that the encoded geometric feature size is (W, H, D);

[0054] The specific process of encoding point cloud visibility features is as follows:

[0055] First, a three-dimensional grid is set on the horizontal plane within the vehicle's sensing range. The number of three-dimensional grids along the X and Y axes is W and H respectively, and the number of the Z axis is O, thereby forming voxels.

[0056] Each laser point in the laser point cloud forms a line segment with the center of the laser radar. The voxels that the line segment passes through are marked as free, the voxels where the laser point is located are marked as occupied, and the remaining voxels are marked as unknown;

[0057] All voxels form feature vectors along the Z-axis direction and are converted into visibility features of length K through the convolution layer, thereby encoding the visibility feature size of (W, H, K);

[0058] S5. Stacking the geometric features and the visibility features to obtain stacked features, specifically, stacking the geometric features and the visibility features along the depth direction: corresponding the geometric features and the visibility features to the grid map generated by the bird's-eye view of the three-dimensional grid, and stacking them along the depth direction, so as to obtain a stacked feature with a size of (B, (D+K), H, W), where B, (D+K), H, and W correspond to batch, depth, height, and width, respectively;

[0059] S6. Input the stacked features into a multi-layer feature extraction network to extract feature information at different levels, and then stack the feature information at each level to obtain a fused feature. In this embodiment, the multi-layer feature extraction network specifically extracts feature maps of three different scales through convolutional layers of different step lengths. The feature maps of each scale are then respectively subjected to deconvolution layers of corresponding step lengths to expand the size to the same size, and finally stacked along the depth direction to obtain a fused feature.

[0060] S7, output the fusion feature to the anchor-free object detector to obtain the 3D object detection result. In this embodiment, the anchor-free object detector includes five detection heads, specifically a key point heat map detection head, a local offset detection head, a z-axis positioning detection head, a 3D object size detection head, and a direction detection head;

[0061] The five detection heads are used to regress the fused features through the convolutional layer to obtain the probability that each pixel of the fused feature is the category center point, the offset in the X and Y directions from the true value center point, the offset in the Z direction from the true value center point, and the geometric size and orientation of the 3D detection box;

[0062] Five detection heads are used to predict the object center on the bird’s-eye view plane and regress different attributes of the 3D bounding box. Finally, the output results of the five detection heads are merged together to generate the 3D object detection result.

[0063] In summary, the present invention performs 3D target detection based on neural network, and its input data includes color image and laser point cloud. The main data processing flow is as follows: Figure 2 As shown:

[0064] 1. The color image obtains the category of each pixel through the semantic segmentation network. The category information is assigned to the laser point cloud through the external parameters between sensors, which enhances the features of the laser point cloud and obtains the enhanced laser point cloud;

[0065] 2. Enhance the laser point cloud to encode geometric features and visibility features (such as Figure 3 As shown), the bird's-eye view features obtained by each module are stacked to obtain stacked features;

[0066] 3. The stacked features are input into a multi-layer feature extraction network to extract feature information at different levels, and then the feature information of each layer is stacked to obtain fused features;

[0067] 4. The fusion features are passed through different detection heads to obtain the category, offset, Z value, shape, and orientation of the 3D target.

[0068] In the process of neural network training, the present invention uses a data augmentation method that combines images and point clouds to alleviate the problem of data category imbalance, thereby improving the accuracy of 3D target detection. The data augmentation method mainly includes random flipping along the X-axis and random insertion of true value targets. Among them, randomly inserting true value targets means saving the image 2D frame, image semantic segmentation pixel points, true value target point cloud and 3D frame of all true value targets in the data set, and randomly selecting some true value targets to insert into the currently retrieved data set; the image 2D frame of the randomly selected true value target has no overlap with the 2D frame of the currently retrieved data set, and according to the contour area formed by the randomly selected true value target point cloud, all laser points of the currently retrieved data set in the contour area are deleted.

[0069] This embodiment applies the above technical solution, and the main input sensor data includes color images and laser point clouds, wherein the color image is subjected to a semantic segmentation network to obtain the category of each pixel point, and the category information can expand the features of the laser point cloud. In addition, according to the principle of laser beam propagation, the space between the sensor and the laser point is not blocked by obstacles and can be considered as a free area. Therefore, the 3D target detection task can utilize image semantic features and spatial visibility features to improve the detection accuracy of 3D targets. Finally, a multi-layer feature extraction network is used to obtain fusion features, and the 3D target detection task is completed through the detection head. The overall network structure of the target detection is as follows: Figure 4 shown.

[0070] Specific:

[0071] Step 1: Use mainstream semantic segmentation methods (such as DeeplabV3+) to classify the categories of each pixel in the color image, and project the laser point cloud onto the image to obtain the image semantic information of each laser point and enhance the features of the laser point cloud;

[0072] Step 2: Divide the body column (W, H, 1) in the perception area space, map the feature-enhanced laser point cloud to each body column, and further enhance the features of the laser points in the body column using the geometric information of all points in the body column and the body column position information. Then, extract the features in the body column through a multi-layer perceptron to obtain the geometric features (W, H, D).

[0073] Step 3: Divide the voxels (W, H, O) in the perception area space and use the ray casting algorithm, such as Figure 5 As shown, the spatial visibility state of each voxel is obtained, and the high-dimensional features are further extracted through the convolution layer to obtain the visibility features (W, H, K);

[0074] Step 4: Stack the geometric features and visibility features in the depth direction and input them into the multi-layer feature extraction network to extract feature information at different levels. Then, stack the feature information of each layer to obtain the fusion feature. The structure of the multi-layer feature extraction network is as follows: Figure 4 As shown;

[0075] Step 5: Connect the feature maps output by the multi-layer feature extraction network to five different detection heads (such as Figure 6 ), to predict the center of objects on the bird’s-eye view plane and regress different attributes of the 3D bounding box.

[0076] For the above steps, further detailed explanation is given:

[0077] 1. Image semantic features enhance point cloud features

[0078] Pre-trained semantic segmentation model (such as using Cityscapes dataset), the semantic segmentation results contain 4 categories (class_id) of car, pedestrian, cyclist and background.

[0079] By projecting from the LiDAR coordinate system to the camera coordinate system, and then from the camera coordinate system to the image coordinate system, the spatial point cloud can be associated with the pixel position of the corresponding image. This method adds semantic information represented by image pixels to each point cloud projected from the LiDAR coordinate system to the image coordinate system, which is equivalent to adding a new data dimension to the original input point cloud, thereby enhancing the semantics of the point cloud. Ultimately, the original point set is expanded from 4 dimensions (x, y, z, r) to 5 dimensions (x, y, z, r, class_id).

[0080] 2. Dynamic Voxelization

[0081] For hard voxelization, the number of non-empty body columns and the number of points in the body columns are restricted. For example, for each input sample, the maximum number of non-empty body columns needs to be set. Generally speaking, the number of non-empty body columns in H×W individual columns will not exceed the set maximum value, otherwise the body columns will be randomly sampled. At the same time, if the number of points in each body column is set to N, each body column that exceeds the set number of points will be randomly sampled; if the number of points in the body column is less than N, zero padding will be performed. Each modified point has D=9-dimensional features, so a point cloud set with a dimension of (N, D) is formed in a body column. If the number of body columns selected is P, it will eventually be converted into a point cloud set with a dimension of (P, N, D).

[0082] However, for dynamic voxelization, all points in each body column are retained for learning body column level features. For each input sample, we get a point set of dimension (N', D). Where N' is the total number of points in each input sample. Dynamic voxelization does not fix the number of point clouds in each body column, but stores the coordinates of each point in the body column to calculate the body column level features.

[0083] 3. Point set visibility feature encoding

[0084] The number of input point sets is N, and each point contains x, y, and z spatial coordinates. First, the present invention initializes all voxels to an unknown state, and then executes a ray casting algorithm for each ray. If the voxel is already occupied, continue to execute. If the final voxel traversed by the ray is reached, the voxel is marked as an occupied state. Otherwise, all traversed voxels are marked as free states. Finally, the entire three-dimensional point cloud space is represented by three states: unknown, occupied, and free.

[0085] This step mainly includes reconstructing the spatial visibility state, calculating the voxel-level visibility features, and finally integrating them into the 3D object detection framework. Here we first introduce the two-dimensional plane and then expand it to the three-dimensional space for easier understanding.

[0086] After obtaining the voxel coordinates of the ray traversal, the visibility state of each voxel in the entire three-dimensional space can be determined. Here, the visibility of the three-dimensional space is defined as three different states, namely unknown space, occupied space, and free space, and is represented by specific values. We stack these voxel-level visibility features along the z-axis and then apply a 1×1 convolutional layer to aggregate them to obtain visibility features with dimensions of (H, W, K).

[0087] 4. Anchor-free detection head

[0088] The anchor-free object detector of the present invention consists of five detection heads, namely, key point heat map detection head, local offset detection head, z-axis positioning detection head, 3D object size detection head and direction detection head. Figure 5 Showing some details of the anchor-free object detector.

[0089] For the heatmap detection head and the offset detection head, the keypoint heatmap and the local offset regression map are predicted. The keypoint heatmap is used to find the location of the target object center in the bird's-eye view. The offset regression map can not only help the heatmap find a more accurate target object center in the bird's-eye view, but also compensate for the discretization error caused by the columnarization process.

[0090] For the offset regression detection head, there are two main functions: First, it is used to remove the errors caused by the columnarization process, in which the floating-point target object center is assigned to the integer body column location in the bird's-eye view. Second, it plays an important role in refining the prediction of the heatmap target object center, especially when the heatmap predicts the wrong center. Specifically, once the heatmap predicts a wrong center that is several pixels away from the true value center, the offset detection head has the ability to mitigate or even eliminate the pixel error relative to the true value target object center. A square region with radius r around the target object center pixel is selected in the offset regression heatmap. The farther the distance from the object center, the larger the offset value, and the L1 loss is used to train the offset.

[0091] After locating the target object in the bird's-eye view, there is only the xy location of the target object. Therefore, a z-axis positioning detection head is needed to regress the z-axis value. The L1 loss function is used to regress the z value.

[0092] The present invention firstly realizes the enhancement of original point cloud data by image semantic information in the data pre-processing stage; secondly, a ray casting algorithm is used to successfully reconstruct the spatial visibility state of the point cloud; then, a convolutional neural network is used to respectively aggregate the point set geometric and semantic features and the point set visibility features.

[0093] The upsampling network output feature map is then applied and connected to five different detection heads to predict the center of objects on the bird's-eye view plane and regress different attributes of the 3D bounding box. Finally, the output results of the five detection heads are merged together to generate the detection result. Among them, the key point heat map prediction detection head is used to predict the center of objects in the bird's-eye view plane. Each object will be encoded into a small area centered on the heat peak.

[0094] The present invention combines aggregated point set geometric features, point set visibility features, and point set semantic features into an end-to-end three-dimensional object detection framework, which not only realizes the data fusion of point cloud and image, but also innovatively proposes the concept of spatial "visibility" of point cloud, effectively reconstructs the spatial occupancy state of point cloud, and helps to accurately perceive the real situation of three-dimensional space. In addition, in the reasoning stage, each hot peak will be picked out through the maximum pooling operation, after which no multiple regressed anchor points will be tiled to one position, so there is no need to use traditional NMS. This allows the entire detector to run on a typical CNN accelerator or GPU, saving CPU resources for other key tasks of autonomous driving.

Claims

1. A method for anchor-free 3D object detection based on multi-sensor fusion, characterized in that: The following steps are involved: S1, obtain color RGB image and laser point cloud; S2, perform semantic segmentation on the color RGB image to obtain the category information of each pixel; S3, using the category information of each pixel to enhance the features of the laser point cloud to obtain an enhanced laser point cloud; S4. According to the enhanced laser point cloud, point cloud geometric feature encoding and point cloud visibility feature encoding are performed to obtain geometric features and visibility features respectively; S5. Stacking the geometric features and the visibility features to obtain stacked features; S6, inputting the stacked features into a multi-layer feature extraction network, extracting feature information at different levels, and then stacking the feature information at each level to obtain a fused feature; S7, output the fused features to the anchor-free object detector to obtain the 3D object detection result; The specific process of step S3 is: by projecting transformation from the laser radar coordinate system to the camera coordinate system and then from the camera coordinate system to the image coordinate system, the spatial point cloud is associated with the pixel position of the corresponding color RGB image, and the semantic information represented by the color RGB image pixel is added to each point cloud projected from the laser radar coordinate system to the image coordinate system, which is equivalent to adding a new data dimension to the original input laser point cloud and giving the laser point cloud a category attribute, thereby enhancing the semantics of the point cloud; The specific process of encoding the point cloud geometric features in step S4 is as follows: First, a three-dimensional grid is set on the horizontal plane within the sensing range of the vehicle. The number of the three-dimensional grid along the X and Y axes is W and H respectively, and the number of the Z axis is 1, so as to form a body column, and the laser point cloud is mapped to the corresponding body column; The features of each laser point in the body column include (x, y, z, i, t, c, ox, oy, dx, dy, dz), where (x, y, z, i, t) are the original features of the laser point, corresponding to the coordinate value, reflectivity, and timestamp of the spatial rectangular coordinate system, respectively. c is the category feature assigned by the semantic segmentation of the image. (ox, oy) is the deviation between the laser point and the central axis x, y of the body column. (dx, dy, dz) is the position mean of the laser point and all laser points. Deviation; All enhanced laser points in the body column are feature encoded by a multi-layer perceptron to generate point cloud geometric features with a fixed feature quantity D, and a maximum pooling operation is performed in the quantity direction, so that the encoded geometric feature size is (W, H, D); The specific process of encoding the point cloud visibility features in step S4 is as follows: First, a three-dimensional grid is set on a horizontal plane within the sensing range of the vehicle, wherein the number of the three-dimensional grid along the X and Y axes is W and H respectively, and the number of the three-dimensional grid along the Z axis is O, thereby forming voxels; Each laser point in the laser point cloud forms a line segment with the center of the laser radar. The voxels that the line segment passes through are marked as free, the voxels where the laser point is located are marked as occupied, and the remaining voxels are marked as unknown; All voxels form feature vectors along the Z-axis direction and are converted into visibility features of length K through the convolution layer, thereby encoding the visibility feature size of (W, H, K); Step S5 specifically stacks the geometric features and the visibility features along the depth direction: the geometric features and the visibility features are respectively mapped to the grid map generated by the bird's-eye view of the three-dimensional grid, and stacked along the depth direction, so as to obtain a stacked feature with a size of (B, (D+K), H, W), where B, (D+K), H, and W correspond to batch, depth, height, and width, respectively; In step S6, the multi-layer feature extraction network specifically extracts feature maps of three different scales through convolutional layers of different step lengths, and the feature maps of each scale are then respectively expanded to the same size through deconvolution layers of corresponding step lengths, and finally stacked in the depth direction to obtain fused features; The anchor-free object detector in step S7 includes five detection heads, specifically a key point heat map detection head, a local offset detection head, a z-axis positioning detection head, a 3D object size detection head, and a direction detection head.

2. The method for anchor-free 3D object detection based on multi-sensor fusion according to claim 1, characterized in that: The step S2 specifically uses a semantic segmentation network to perform semantic segmentation on the color RGB image, and the semantic segmentation network includes U-Net, SegNet, DeepLab or BiSeNet; The category information of each pixel in step S2 includes four categories, namely, vehicle, pedestrian, non-motor vehicle and background.

3. The anchor-free 3D object detection method based on multi-sensor fusion according to claim 1 is characterized in that: The five detection heads are used to regress the fused features through the convolutional layers respectively to obtain the probability that each pixel point of the fused feature is the category center point, the offset in the X and Y directions from the true value center point, the offset in the Z direction from the true value center point, and the geometric size and orientation of the three-dimensional detection box.

4. The anchor-free 3D object detection method based on multi-sensor fusion according to claim 3 is characterized in that: The step S7 specifically uses five detection heads to predict the center of the object on the bird's-eye view plane and regress different attributes of the 3D bounding box. Finally, the output results of the five detection heads are merged together to generate a 3D target detection result.

Images