Laser radar point cloud 2D-3D bimodal interaction enhanced target detection method

By perspective projecting the lidar point cloud and recording the 2D-3D correspondence relationship, combining 2D image features and other modal data for fusion, the real-time and accuracy problems of the lidar point cloud target detection model in the existing technology are solved, and the real-time and accuracy of the lidar point cloud target detection model is realized. More efficient autonomous driving target detection.

CN119068441AInactive Publication Date: 2024-12-03BEIJING SHENGDATONG INTELLIGENT TECHNOLOGY CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411271297.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the target detection model based on lidar point cloud is difficult to achieve real-time and high-precision detection under the conditions of limited computing hardware resources. In addition, there is error or huge calculation amount during multimodal fusion. Detection accuracy and speed.

Method used

By rendering the lidar point cloud into a 2D image through perspective projection and recording a 2D-3D correspondence lookup table, combining 2D image features and other modal data for fusion, reducing the computational amount and improving the accuracy of feature space alignment.

Benefits of technology

The accuracy and speed of target detection have been significantly improved, the mAP of the model has been increased by 7.8%, and the detection frequency has been increased without reducing the inference speed. It is suitable for engineering applications of autonomous driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068441A_ABST
    Figure CN119068441A_ABST
Patent Text Reader

Abstract

The invention provides a laser radar point cloud 2D-3D bimodal interaction enhanced target detection method. The method comprises the following steps: carrying out perspective projection on each frame of laser radar 3D point cloud by taking a laser radar as a viewpoint, and rendering the perspective projection into a 2D perspective image; extracting features of the 2D perspective image; fusing the features extracted from the 2D perspective image and the features extracted from other modal data; through the above scheme, the model learns object contour feature details presented by the 2D perspective image to make up for feature information loss caused by point cloud voxelization, and the target detection capability of the model can be significantly improved; by generating the 2D-3D corresponding relation lookup table, when the extracted features are subjected to feature space alignment, the 2D-3D corresponding relation lookup table is directly looked up to obtain the 3D points corresponding to the pixels, the calculation speed of space conversion alignment during feature fusion is increased, and the problem of calculation errors or inaccurate prediction existing in a conventional 2D-to-3D conversion method is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a target detection method for laser radar point cloud 2D-3D dual-modal interactive enhancement. Background Art

[0002] At present, the target detection deep learning model based on one or more sensors such as cameras, lidar, millimeter-wave radar, etc. in the autonomous driving system (including target segmentation, which is actually a fine-grained target detection. Therefore, the target detection mentioned here is broad and includes the so-called target detection and target segmentation) usually uses one or more modal data from camera images, lidar point clouds, and millimeter-wave radar point clouds to input deep learning models for target detection. The detection effect of the multimodal fusion model that extracts features and fuses multimodal data is generally better than the detection effect of the model using single modal data and has better reliability. Among the multimodal fusion models, the multimodal fusion models based on lidar point cloud plus camera image or lidar point cloud plus millimeter-wave radar point cloud or lidar point cloud plus camera image plus millimeter-wave radar point cloud are more commonly used. These all use lidar point cloud, because lidar point cloud can not only detect obstacles but also measure distance more accurately, and its point cloud is much denser than millimeter-wave radar point cloud, and can show the shape, size and even direction of the target obstacle at a certain granularity. Although it is not as clear as the imaging of camera images, its point cloud imaging capability is beyond the reach of millimeter-wave radar. The point cloud of millimeter-wave radar is just the reflection of millimeter waves hitting the object. The very sparse points returned can only be used to perceive the existence of objects and calculate the speed of moving objects, but cannot display the more accurate outline and size of objects. The image of the object taken by the camera is clear and the texture features of nearby objects are clear, but the 2D image does not have the 3D spatial position information of the object, and the camera shooting is easily affected by the ambient light. Generally, the imaging of objects is not good at night when there is insufficient light. Therefore, it is relatively reliable to use lidar to detect obstacles. Therefore, the general autonomous driving perception algorithm uses a multimodal fusion deep learning model based on multi-sensor data, and the lidar point cloud is used as one of the input data of the multimodal fusion model.

[0003] There are usually two solutions for how deep learning models can detect objects by acquiring object features through lidar point cloud data:

[0004] One is to extract features point by point, such as PointNet and its improved sequence model. This implementation scheme has a huge amount of calculation and is usually not used in engineering applications due to the limitations of computing hardware resources and the system's real-time requirements for model reasoning.

[0005] The other is to voxelize the point cloud first and then extract features from the voxels, such as PointPillars and CenterPoint models. This solution greatly reduces the amount of calculation and can usually meet the real-time requirements of model reasoning in engineering applications under the condition of limited computing hardware resources. Therefore, this solution is usually used in engineering applications.

[0006] Multimodal fusion models usually need to extract features from multimodal data and perform feature fusion before detecting targets. Taking the multimodal fusion model whose input data includes at least camera images and lidar point clouds as an example, when the model fuses 2D image features and 3D point cloud features, it is generally necessary to obtain the 3D coordinates of the actual space points represented by the points on the feature map extracted from the 2D image based on the pixel coordinates of the points, and then attribute the 2D features to the corresponding 3D positions to achieve the conversion from 2D features to 3D features. There are usually three solutions:

[0007] First, we assume that the ground is completely flat and use the camera's intrinsic and extrinsic parameters to calculate the 3D coordinate points corresponding to the 2D coordinate points.

[0008] The second is to use a depth prediction model (such as the commonly used LSS model) to predict the depth of each point on the image and obtain its coordinates in 3D space;

[0009] The third is to use the Transformer's attention mechanism to associate the pixel coordinates in the 2D image with the corresponding 3D space point coordinates based on the query method.

[0010] The above three common solutions have the following problems:

[0011] Solution 1 assumes that the ground is completely flat, which is often not in line with the actual situation. In addition, the camera's internal and external parameters may be inaccurate, which will cause inaccurate conversion of the image point's 2D coordinates to 3D space coordinates, thus affecting the accuracy of model detection.

[0012] Solution 2 uses a depth prediction model to predict the depth of each point in the image, but the calculated 3D coordinates are inaccurate. In addition, the depth prediction model is trained using data from a specific scene. The depth prediction effect will decrease when the scene is changed, resulting in even less accurate depth prediction.

[0013] Solution 3 uses the Transformer's attention mechanism, which results in a huge amount of computation and places high demands on the hardware resources for model deployment. Without specialized acceleration hardware and advanced model optimization technology, it will be difficult to implement engineering applications.

[0014] In addition, the defect of the solution commonly used in engineering applications to voxelize point clouds and then extract data features such as 3D coordinates and reflection intensity of the point clouds is that voxelizing 3D point clouds and then extracting features is like mosaicking a 2D image and then extracting features. Obviously, there is a large loss of feature information contained in the data, which causes the accuracy and recall rate of the model to detect objects to decrease.

[0015] For models whose input data includes lidar point clouds, the common methods to solve the problem of reduced target detection accuracy and recall rate caused by point cloud voxelization are:

[0016] (1) Reduce the voxel size to make the voxel division of the point cloud space denser;

[0017] (2) Superimposing multiple consecutive point cloud frames to increase the density of the point cloud and enhance the features of the point cloud voxels;

[0018] These measures can generally improve the model's detection indicators by several percentage points, but the problem is that the amount of calculation is greatly increased, which slows down the model's reasoning speed. When the point cloud space to be detected is relatively large, the model's reasoning speed cannot meet the requirements of autonomous driving engineering applications. Even after adding sparse convolution and other designs to the model, the model's reasoning speed is still not enough to meet the target detection frequency requirements within a larger range of point cloud space (the field of autonomous driving generally requires that the target detection frequency be no less than 10HZ).

[0019] In summary, the existing deep learning models whose input data includes lidar point clouds mainly learn the 3D coordinates and reflection intensity data features of the point clouds, and in order to achieve a relatively fast model inference speed, the point clouds are generally voxelized before feature extraction. This is equivalent to only performing very coarse-grained feature extraction on the point cloud data, and the object contour image details that the point cloud can present are lost. It is like mosaicing the 2D image of the camera and then extracting features. All you see are the features of the blurred content of the 2D image after mosaicking, and the details of the original image are gone. However, in order to preserve subtle features as much as possible, each point in the point cloud cannot be involved in feature extraction or the voxelized grid size cannot be set very small. In this case, the model has a very large amount of calculation. In addition, the current multimodal fusion model whose input data includes at least images and point clouds uses a 2D feature to 3D feature conversion method when fusing 2D image features and 3D point cloud features. Either it is easy to have large errors, or the amount of calculation is huge, which affects its application in engineering.

[0020] Therefore, there is an urgent need for a 2D-3D dual-modal interactive enhanced target detection method based on LiDAR point cloud, which uses 2D images to enhance 3D features and uses 3D coordinate information to enhance the accuracy of the spatial position corresponding to the calculated image pixels. By perspective projecting the LiDAR point cloud data and rendering it into a 2D image, it is also input into the multimodal fusion model as a modal data. The model can learn the object contour feature details presented by the 2D perspective image of the point cloud while learning the coarse-grained point cloud features after voxelization to compensate for the loss of feature information caused by the voxelization of the point cloud. This can significantly improve the target detection capability of the model without significantly reducing the reasoning speed. While perspective projecting the lidar 3D point cloud into a 2D image, a 2D-3D correspondence lookup table is recorded and generated between the pixel coordinates of each pixel in the 2D image and the 3D coordinates of the corresponding point cloud point. This table is directly looked up to obtain the 3D spatial point corresponding to the pixel when spatially aligning the features of the 2D point cloud perspective view and the features of other modal data before fusion. There is no need to perform conversion calculations from 2D features of the camera image to 3D features like in general models. This not only improves the calculation speed of feature space alignment of multimodal data, but also avoids the problem of inaccurate calculated 3D coordinates due to conversion calculation errors caused by inaccurate parameters or inaccurate model depth prediction. Summary of the invention

[0021] The purpose of the present invention is to provide a target detection method for 2D-3D dual-modal interactive enhancement of laser radar point cloud to solve the problems existing in the above-mentioned prior art.

[0022] To achieve the above object, the present invention provides the following solutions:

[0023] The present invention provides a target detection method for laser radar point cloud 2D-3D dual-modal interactive enhancement, comprising the following steps:

[0024] S1. Perspectively project each frame of the laser radar 3D point cloud using the laser radar as the viewpoint and render it into a 2D perspective image;

[0025] S2. Extract features from 2D perspective images;

[0026] S3. Fusion of features extracted from 2D perspective images and features extracted from other modality data.

[0027] Preferably, in step S1, when each frame of the laser radar 3D point cloud is perspectively projected onto a 2D plane with the laser radar as the viewpoint, the one-to-one correspondence between the coordinates of the 3D point (x, y, z) and the index value (r, c) of the projection point on the 2D plane is directly recorded in a 2D-3D correspondence lookup table. When multiple 3D points are projected onto the same point on the plane, only the coordinates of the 3D point with the smallest z value are recorded.

[0028] Preferably, in step S2, a backbone network for detecting ordinary 2D images is used to extract feature data from the input 2D perspective image.

[0029] Preferably, step S3 specifically includes:

[0030] S31. Assume that F is the feature data extracted from a 2D perspective image, it has C channels, and the data of each channel is two-dimensional data of H rows and W columns, represented by a three-dimensional array FA[C][H][W]. For an element p of a two-dimensional array of a channel of FA, its row and column index values ​​are (r, c), where r represents the row and c represents the column, and the value of element p is expressed as p(r, c);

[0031] S32. Assume that the downsampling scaling factor of the model in the process of extracting features from the original 2D image is S, divide the index value (r, c) of p by the scaling factor S and round it to the integer to obtain the index value (R, C) of p in the original 2D image;

[0032] S33. Query the 2D-3D correspondence lookup table according to the index value (R, C) to obtain the original 3D point coordinates (x, y, z) corresponding to p;

[0033] S34. Obtain the grid id or grid coordinates of the corresponding grid according to the original 3D coordinate point (x, y, z), and record the corresponding relationship between the eigenvalue p(r, c) and the grid id or grid coordinates;

[0034] S35. Use GPU kernel function to implement and assign all element values ​​of F to corresponding grid positions to realize the conversion of 2D features to 3D features, and then fuse the 3D feature data with the feature data extracted from other modal data unified under the grid, and finally send it to the neck and head of the model for feature enhancement and target detection.

[0035] Preferably, in step S34, the grid is a 3D space grid or a BEV top-view grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A flowchart of a target detection method for 2D-3D dual-modal interactive enhancement of laser radar point cloud provided by the present invention;

[0038] Figure 2A perspective imaging principle diagram of a target detection method for 2D-3D dual-modal interactive enhancement of a laser radar point cloud provided by the present invention;

[0039] Figure 3 An example diagram of a 2D perspective image in a target detection method for 2D-3D dual-modal interactive enhancement of a laser radar point cloud provided by the present invention;

[0040] Figure 4 This is the structural diagram of the multimodal fusion model of conventional camera plus lidar;

[0041] Figure 5 A structure diagram of a multi-modal fusion model of a camera plus a laser radar with a point cloud 2D-3D dual-modal interactive enhanced target detection method provided by the present invention;

[0042] Figure 6 This is the detection effect diagram of the multimodal fusion model without 2D perspective image input;

[0043] Figure 7 This is the detection effect diagram of the multimodal fusion model with the addition of 2D perspective image input. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The purpose of the present invention is to provide a target detection method for 2D-3D dual-modal interactive enhancement of laser radar point cloud to solve the problems existing in the prior art.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Embodiment 1:

[0048] The present embodiment provides a target detection method for 2D-3D dual-modal interactive enhancement of lidar point clouds, which performs perspective projection on each frame of lidar point cloud with the lidar as the viewpoint and renders it into a 2D image. The 2D image is then input into a multimodal fusion model as a kind of modal data. A backbone network for detecting 2D images is used inside the model to extract features. The feature data is then converted to the 3D spatial dimension according to a 2D-3D correspondence lookup table, and is fused as an enhanced feature with point cloud image details and features extracted from other modal data.

[0049] The specific steps of the target detection method provided by this invention for 2D-3D dual-modal interactive enhancement of laser radar point cloud are as follows: Figure 1 As shown:

[0050] S1. Perspectively project each frame of the laser radar 3D point cloud using the laser radar as the viewpoint and render it into a 2D perspective image;

[0051] Similar to the perspective imaging of camera photography Figure 2 As shown in the figure, the point cloud data reflected by objects at different distances in the viewing cone are converted to the observation space camera coordinate system with the lidar as the viewpoint, and then projected onto a small plane of a specified size in front of the viewpoint, and rendered with the corresponding color according to the reflection intensity of the point, and drawn into a 2D color image of a certain width and height size (for example, 1920×1080), that is, a 2D perspective image of the point cloud. For the effect of the 2D perspective image of the point cloud, see Figure 3 ; The technology of perspective projecting a 3D point cloud and rendering it into a 2D image belongs to the prior art known in the art, so it will not be described in detail here;

[0052] Since each point of the point cloud is a 3D space point in the laser radar coordinate system, it is still a 3D point after being converted to the observation space camera coordinate system, but the directions of the axes of the camera coordinate system are different from those of the laser radar coordinate system. Assuming that the positive direction of the Z axis in the camera coordinate system is perpendicular to the projection plane and away from the viewpoint, the coordinates of the 3D point are (x, y, z), then the coordinate value z will be lost when the 3D point (x, y, z) is projected onto the 2D plane. If the z value is not recorded, the original 3D point corresponding to the pixel point on the 2D image generated by the perspective projection cannot be calculated in reverse; if the graphics library tool is called for rendering, Maybe the graphics tool library can specify through parameters that these z values ​​are saved when doing perspective projection (such as OpenGL). We generate perspective images based on 3D point clouds implemented by ourselves based on open source code. In order to save calculations and speed up calculations, when we project the points of the 3D point cloud onto the 2D plane, we directly record the one-to-one correspondence between the coordinates (x, y, z) of the 3D point and the index value (r, c) of the projection point on the 2D plane into a 2D-3D correspondence lookup table. When multiple 3D points are projected onto the same point on the plane, only the coordinates of the 3D point with the smallest z value (that is, the z value of the point closest to the projection plane) are recorded.

[0053] S2. Extract features from 2D perspective images;

[0054] The point cloud 2D perspective image is input into the multimodal fusion model as a modality data, and a branch dedicated to extracting the point cloud perspective image features is added to the model (see Figure 4 and Figure 5), using the backbone network for detecting ordinary 2D images to extract feature data from the input 2D point cloud perspective image;

[0055] S3. Fusing the features extracted from the 2D perspective image with the features extracted from other modality data;

[0056] In the multimodal fusion model we use, the feature data extracted from different modal data need to be uniformly attributed to the 3D space before fusion. Therefore, the feature data extracted from the 2D perspective image of the point cloud also need to be attributed to the 3D space accordingly. That is, each value in the feature data F extracted from the 2D perspective image of the point cloud needs to be attributed to a certain position in the 3D space. The specific processing method is:

[0057] 1) Assume that F is the feature data extracted from the 2D perspective view of the point cloud. It has C channels, and the data of each channel is two-dimensional data (H rows and W columns). Then it can be represented by a three-dimensional array FA[C][H][W]. For an element p of the two-dimensional array of a channel of FA, its row and column index values ​​are (r, c), where r represents the row and c represents the column. The value of the element p can be expressed as p(r, c). We need to attribute p(r, c) to the 3D space according to the following steps;

[0058] 2) Since F is obtained by downsampling the original 2D image during feature extraction, assuming that the downsampling scaling factor is S, the index value (r, c) of p is divided by the scaling factor S and rounded to the integer to obtain the index value (R, C) of p in the original 2D image;

[0059] 3) According to the index value (R, C), query the original 3D point coordinates (x, y, z) corresponding to p from the previously generated 2D-3D correspondence lookup table;

[0060] 4) According to the 3D coordinate point (x, y, z), obtain the grid id or grid coordinates of the corresponding 3D space grid (dividing the space into multiple three-dimensional cells according to a certain size) or BEV (bird's eye view) grid (dividing the plane at a bird's eye view angle into multiple two-dimensional cells according to a certain size), and record the corresponding relationship between the eigenvalue p(r, c) and the grid id or grid coordinates.

[0061] 5) Use GPU kernel functions to concurrently assign all element values ​​of F to corresponding grid positions to realize the conversion of 2D features to 3D features, and then fuse (splice or add) the feature data extracted from other modal data (such as camera images, voxelized point clouds) unified under the 3D space grid, and then send them to the neck and head of the model for feature enhancement and target detection. The above model design technology belongs to the prior art known in the art, so it will not be described in detail.

[0062] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0063] The multimodal fusion model of the target detection method based on the 2D-3D dual-modal interactive enhancement of the laser radar point cloud provided by the present invention is tested on different data sets. The test indicators of the model are compared when the input point cloud perspective image is added and when it is not added. After adding the input point cloud perspective image, the mAP of the model can be increased by up to 7.8%. It can be seen from the visualization effect of the test that after adding the input of the point cloud perspective image, the target detection effect is significantly better, mainly manifested in the stronger target detection ability, see Figure 6 and Figure 7 Comparison of effects.

[0064] Moreover, the perspective projection and rendering of the above point cloud into 2D images and the generation of 2D-3D correspondence lookup tables are all encapsulated in the GPU kernel function and executed concurrently by GPU multi-threading. The time spent on commonly used edge devices with GPUs generally does not exceed 10-15 milliseconds, so the execution speed will not have a significant impact on the speed performance of the reasoning process of the perception part of the autonomous driving system.

[0065] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A laser radar point cloud 2D-3D dual-modal interactive enhanced target detection method, characterized by: The following steps are involved: S1. Perspectively project each frame of the laser radar 3D point cloud using the laser radar as the viewpoint and render it into a 2D perspective image; S2. Extract features from 2D perspective images; S3. Fusion of features extracted from 2D perspective images and features extracted from other modality data.

2. The target detection method of 2D-3D dual-modal interactive enhancement of laser radar point cloud according to claim 1, characterized in that: In step S1, when each frame of the laser radar 3D point cloud is perspectively projected onto a 2D plane with the laser radar as the viewpoint, the one-to-one correspondence between the coordinates (x, y, z) of the 3D point and the index value (r, c) of the projection point on the 2D plane is directly recorded in a 2D-3D correspondence lookup table. When multiple 3D points are projected onto the same point on the plane, only the coordinates of the 3D point with the smallest z value are recorded.

3. The target detection method of 2D-3D dual-modal interactive enhancement of laser radar point cloud according to claim 2, characterized in that: In step S2, feature data is extracted from the input 2D perspective image using a backbone network for detecting ordinary 2D images.

4. The target detection method of 2D-3D dual-modal interactive enhancement of laser radar point cloud according to claim 3 is characterized by: Step S3 specifically includes: S31. Assume that F is the feature data extracted from a 2D perspective image, it has C channels, and the data of each channel is two-dimensional data of H rows and W columns, represented by a three-dimensional array FA[C][H][W]. For an element p of a two-dimensional array of a channel of FA, its row and column index values ​​are (r, c), where r represents the row and c represents the column, and the value of element p is expressed as p(r, c); S32. Assume that the downsampling scaling factor of the model in the process of extracting features from the original 2D image is S, divide the index value (r, c) of p by the scaling factor S and round it to the integer to obtain the index value (R, C) of p in the original 2D image; S33. Query the 2D-3D correspondence lookup table according to the index value (R, C) to obtain the original 3D point coordinates (x, y, z) corresponding to p; S34. Obtain the grid id or grid coordinates of the corresponding grid according to the original 3D coordinate point (x, y, z), and record the corresponding relationship between the eigenvalue p(r, c) and the grid id or grid coordinates; S35. Use GPU kernel function to implement and assign all element values ​​of F to corresponding grid positions to realize the conversion of 2D features to 3D features, and then fuse the 3D feature data with the feature data extracted from other modal data unified under the grid, and finally send it to the neck and head of the model for feature enhancement and target detection.

5. The target detection method of 2D-3D dual-modal interactive enhancement of laser radar point cloud according to claim 4, characterized in that: In step S34, the grid is a 3D space grid or a BEV top-view grid.

Citation Information

Patent Citations

  • Point cloud target detection method and device, electronic equipment and storage medium

    CN116168384A

  • Pavement crack identification method based on point cloud-RGB (Red, Green, Blue) heterogenous image multistage registration mapping

    CN117036300A

  • Method and device based on laser point cloud and low-light image fusion

    CN117058051A

  • Roadside 3D target detection deep learning algorithm based on combination of camera and laser radar

    CN117058407A

  • Multi-mode BEV look-around perception method, device and equipment and storage medium

    CN117392625A