Obstacle 3D bounding box detection method based on 4D millimeter wave radar point cloud
By adopting the three-dimensional obstacle border detection method of 4D mmWave radar point cloud in the autonomous driving system, using clustering and convex hull algorithms, the problem of obstacle detection failure in harsh environments in the prior art is solved, and the system is improved Security and robustness provide accurate 3D border data for obstacles.
Patent Information
- Application Number
- CN202210391431.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-14
AI Technical Summary
In the existing autonomous driving systems, the three-dimensional obstacle frame detection method based on deep learning may fail in harsh environments, and relying on high-cost lidar, it is difficult to meet the high requirements of high-level ADAS for perception systems.
Adopt three-dimensional border detection method based on 4D mmWave radar point cloud, identify obstacles from point cloud data through clustering algorithm, calculate their height, and calculate three-dimensional borders through convex hull algorithm to filter miscellaneous points and provide accurate obstacles Three-dimensional border data of objects.
It improves the safety and robustness of the autonomous driving system, reduces dependence on deep learning models, reduces the amount of calculation, and effectively filters 80% of the miscellaneous points in the environment, providing accurate three-dimensional border data of obstacles.
Smart Images

Figure CN114821526B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to obstacle detection technology during vehicle driving, and specifically to a three-dimensional obstacle border detection method based on 4D millimeter-wave radar point cloud. Background Art
[0002] Intelligent three electrics (battery, motor, electronic control), intelligent cockpit, and autonomous driving represent the development trend of future cars. The software system of unmanned driving is generally defined as six modules: perception, prediction, high-precision map, positioning, decision-making planning, and control. The perception module function usually refers to obstacle and traffic light recognition. The sensors used are mainly laser radar, millimeter wave radar, and camera. These three sensors have different advantages in different environments.
[0003] Among them, cameras are low-cost and have advantages in target detection and classification. They are widely used in ADAS autonomous driving systems. However, they are sensitive to changes in light and rely on deep learning network models and training data sets, so they may fail. As an active sensor, lidar has the advantages of high accuracy and strong three-dimensional perception, but it is expensive and easily affected by weather. Millimeter-wave radar is not only relatively low-cost, but also has three-dimensional point cloud and speed perception capabilities. It can work all-weather and has high reliability. Therefore, millimeter-wave radar is widely used in ADAS autonomous driving systems of all levels.
[0004] Compared with traditional 3D millimeter-wave radar, 4D millimeter-wave radar has the ability to detect pitch height. At the same time, the number and quality of output point clouds have also been greatly improved. The perception ability is getting closer and closer to that of laser radar, so it is also called imaging radar. High-level ADAS autonomous driving systems have higher requirements for perception systems. For example, in relatively crowded driving conditions or parking conditions in cities, autonomous driving systems require accurate three-dimensional border outlines of targets, which not only facilitates the decision-making and planning modules to plan safer and more reasonable paths, but also enables better visualization. Therefore, obstacle three-dimensional border detection has gradually become an indispensable part of environmental perception.
[0005] In the prior art, such as the 4D perception method, system and medium for autonomous driving adapted to all-weather environments disclosed in CN113222111A, and application number 202110355978.X; including: calibrating the external parameters of the digital camera and the 4D millimeter wave radar, projecting the point cloud information of the 4D millimeter wave radar onto the camera image and attaching distance information to some pixels of the image, and then performing convolution network calculation on the new image to obtain a dense depth map, and then converting the depth map into a coded point cloud map through the camera internal parameter calibration and sending it into the convolution network for target perception detection, and then mapping the target perception frame to a 3D target frame under the camera coordinates, and then attaching speed information to the 3D target frame through the 4D millimeter wave point cloud, so as to obtain a new 4D target perception result. The present invention can greatly improve the perception ability of the target image, and can still obtain good perception results in rain, snow, fog and other bad weather and wind, sand and dust. The target frame detection method mentioned in the application is to detect the target including the target frame from the image point cloud map using the method of deep learning of neural network. This method uses deep learning, which relies on deep learning network models and training data sets and may fail. It also uses images and lidar point clouds to detect the 3D bounding boxes of obstacle targets. Therefore, it is necessary to invent a three-dimensional obstacle border detection method based on millimeter-wave radar point clouds. Summary of the invention
[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for detecting three-dimensional bounding boxes of obstacles based on 4D millimeter-wave radar point clouds. Through a clustering algorithm, objects can be detected and accurate three-dimensional bounding boxes of obstacles can be obtained to improve the safety of autonomous driving.
[0007] The technical solution of the present invention is achieved in this way:
[0008] 1. A method for detecting three-dimensional boundary of obstacles based on 4D millimeter-wave radar point cloud, characterized by comprising the following steps:
[0009] Step 1, obtain 4D millimeter wave radar point cloud data through Ethernet;
[0010] Step 2, point cloud data preprocessing;
[0011] Step 3, clustering the point cloud data in three-dimensional space to identify obstacles from the point cloud data;
[0012] Step 4, post-processing of clustered point cloud data;
[0013] Step 5: A three-dimensional bounding box algorithm is used based on the clustered point cloud to obtain the vertex coordinates of the three-dimensional bounding box of the obstacle.
[0014] In this way, the detection method of the present invention can detect objects and calculate their heights through a clustering algorithm, distinguish the point clouds reflected by obstacles from the noise, thereby deleting the noise and false points, and can effectively filter out 80% of the noise in the environment, thereby accurately identifying the point clouds reflected by the surface contour of the object, and providing accurate data for constructing the three-dimensional border of the obstacle.
[0015] Further: the related processing of the image described in step 1 first converts a frame of data obtained by the 4D millimeter wave radar into physical values, converts the distance, azimuth and pitch angle values into X, Y, and Z coordinates according to the vehicle coordinate system definition and trigonometric function relationship, and then converts the coordinate values to the vehicle coordinate system and stores them in global variables.
[0016] Further: the clustering of the point cloud data in three-dimensional space described in step 3 specifically includes the following steps:
[0017] 1) Initialize a point object container and store the point cloud data in the container. The point object contains the following member variables: coordinates X / Y / Z, ID, whether it has been visited, cluster ClusterID, whether it is a core point flag IsKey, and neighborhood point list;
[0018] 2) Set clustering parameters, including neighborhood radius R, minimum number of neighborhood data MinPts, and data dimension Dimension;
[0019] 3) Select a point from the point cloud data at random, mark the Visited attribute of the point as visited, calculate the number of points PtsNum within the neighborhood radius R of the point, if the number of points PtsNum is greater than the set threshold MinPts, mark the IsKey attribute of the point as a core point, and assign a cluster ClusterID; at the same time, traverse all points within the neighborhood radius R of the point and perform the following operations, mark the Visited attribute of the point as visited, assign the same cluster ClusterID, and repeat steps 2) and 3) for these points until all points in the point cloud data are traversed and visited;
[0020] 4) For the successfully clustered point cloud clusters, calculate the average Z coordinate of each point cloud cluster and store it in the variable ClassZInfo.
[0021] Further: the post-processing of the clustered point cloud data in step 4 includes deleting from the point cloud data the clutter points that have not been successfully clustered and the cluster points whose height values do not meet the threshold.
[0022] Further: the algorithm for obtaining the three-dimensional bounding box based on the clustered point cloud described in step 5 includes the following steps:
[0023] 1) Reduce the dimension of the clustered 3D point cloud and project it onto the XY coordinate plane;
[0024] 2) Use the convex hull algorithm to calculate the boundary key points of the clustered point cloud. First, find the point with the smallest ordinate y among all points, recorded as p0; then calculate the cosine value of the angle between the line connecting the remaining points and this point and the x-axis, and sort these points from large to small according to their cosine values with respect to the lowest point. The sorted points are recorded as p1, p2, p3, etc.; push the lowest point p0 and the first point p1 among the sorted points into the stack, and then start calculating from p2 to calculate whether the three-point vector of the two top points of the stack and this point rotates counterclockwise. If so, push the point into the stack, otherwise push the top element of the stack out; finally, the remaining points in the stack are all the convex hull boundary key points;
[0025] 3) Sort the boundary key points after the convex hull clockwise around its geometric center, calculate the angle between them and the coordinate origin, and store the sorted boundary key points in the container;
[0026] 4) According to the angle between each boundary key point and the coordinate origin calculated in the previous step, the boundary key points in the obstacle point cloud that are directly illuminated by the radar are calculated and marked as valid boundary key points;
[0027] 5) From the valid boundary key points, take the direction of any two adjacent points as the direction of the rectangle, calculate the minimum area rectangle that includes all boundary key points, and finally output the rectangle with the smallest area as the two-dimensional border of the obstacle, and record the coordinates of the four vertices of the two-dimensional rectangle;
[0028] 6) According to the maximum and minimum Z coordinates in the clustered point cloud, the four vertex coordinates calculated in step 5) are assigned the minimum Z coordinate and the maximum Z coordinate values, respectively, that is, the vertex coordinates of the three-dimensional bounding box are obtained, and the three-dimensional bounding box of the obstacle is obtained.
[0029] Further: the convex hull algorithm is used to calculate the boundary key points of the clustered point cloud, and the ConvexHull algorithm is used to calculate the convex hull boundary key points of each clustered point cloud.
[0030] Further: Set the data dimension parameter Dimension to 2.
[0031] Further: Set the neighborhood radius R to 2 meters, the minimum number of neighborhood data MinPts to 12, and the data dimension Dimension to 3.
[0032] Further: the point cloud data is preprocessed as described in step 2, and noise is filtered out according to the signal-to-noise ratio and the height value.
[0033] In summary, the present invention provides a method for detecting three-dimensional obstacle borders based on 4D millimeter-wave radar point cloud, which has the following beneficial effects:
[0034] 1. The method of the present invention can detect objects and calculate their heights through a clustering algorithm, distinguish the point clouds reflected by obstacles from the clutter, thereby deleting the clutter and false points, and can effectively filter out 80% of the clutter in the environment, thereby accurately identifying the point clouds reflected by the surface contour of the object, and providing accurate data for constructing the three-dimensional border of the obstacle.
[0035] 2. The method of the present invention fully utilizes the advantages of large number, high precision and high coverage of obstacle boundary point clouds of imaging radar, and gives play to the ranging advantages of millimeter wave radar.
[0036] 3. Compared with deep learning methods, the method of the present invention has the advantages of good robustness, high reliability and small computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The present invention is a flowchart of a method for detecting three-dimensional boundary of an obstacle based on 4D millimeter-wave radar point cloud in an embodiment. DETAILED DESCRIPTION
[0038] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0039] like Figure 1 As shown, a method for detecting three-dimensional boundary of an obstacle based on 4D millimeter wave radar point cloud of the present invention comprises the following steps:
[0040] Step 1, obtain 4D millimeter wave radar point cloud data through Ethernet;
[0041] When processing the radar point cloud, according to the requirements of the built-in processing program, the physical value conversion is first performed on a frame of data obtained by the 4D millimeter-wave radar. According to the vehicle coordinate system definition and the trigonometric function relationship, the distance, azimuth and pitch angle values are converted into X, Y, and Z coordinates, and then the coordinate values are converted to the vehicle coordinate system and stored in the global variables.
[0042] Step 2, point cloud data preprocessing;
[0043] The point cloud data preprocessing includes filtering out noise points according to the signal-to-noise ratio and the height value to improve the quality of the point cloud.
[0044] Step 3: cluster the point cloud data in three-dimensional space and identify obstacles from the point cloud data;
[0045] The point cloud data is clustered in three-dimensional space, and the specific process includes the following steps:
[0046] 1) Initialize a point object container and store the point cloud data in the container. The point object contains the following member variables: coordinates X / Y / Z, ID, visited flag Visited, cluster ClusterID, core point flag IsKey, and neighborhood point list.
[0047] 2) Set clustering parameters. Here, three parameters are set, namely, neighborhood radius R, minimum number of neighborhood data MinPts, and data dimension Dimension. Preferably, here, based on the analysis of real vehicle test data, the neighborhood radius R is set to 2 meters, the minimum number of neighborhood data MinPts is set to 12, and the data dimension Dimension is set to 3.
[0048] 3) Select any point from the point cloud data, mark the Visited attribute of the point as visited, calculate the number of points PtsNum within the neighborhood radius R of the point, if the number of points PtsNum is greater than the set threshold MinPts, then mark the IsKey attribute of the point as a core point and assign a cluster ClusterID; at the same time, traverse all points within the neighborhood radius R of the point and perform the following operations, mark the Visited attribute of the point as visited, assign the same cluster ClusterID, and repeat steps 2-3 for these points until all points in the point cloud data are traversed and visited.
[0049] 4) For the successfully clustered point cloud clusters, calculate the average Z coordinate of each point cloud cluster and store it in the variable ClassZInfo.
[0050] Step 4, post-processing of clustered point cloud data;
[0051] The clustering point cloud data post-processing specifically includes: deleting the unsuccessfully clustered miscellaneous points and the clustering points whose height values do not meet the threshold from the point cloud data.
[0052] Step 5: Calculate the 3D bounding box algorithm based on the clustered point cloud;
[0053] The algorithm for finding a three-dimensional bounding box based on clustered point cloud includes the following steps:
[0054] 1) Reduce the dimension of the clustered 3D point cloud and project it onto the XY coordinate plane;
[0055] 2) Use the convex hull algorithm to calculate the boundary key points of the clustered point cloud. Here, the ConvexHull algorithm is used to calculate the convex hull boundary key points of each clustered point cloud. Preferably, the data dimension parameter Dimension is set to 2. First, find the point with the smallest ordinate y among all points, recorded as p0; then calculate the cosine value of the angle between the line connecting the remaining points and this point and the x-axis, and sort these points from large to small according to their cosine values with respect to the lowest point. The sorted points are recorded as p1, p2, p3, etc.; push the lowest point p0 and the first point p1 in the sorted points into the stack, and then start calculating from p2 to calculate whether the three-point vector of the two top points of the stack and this point rotates counterclockwise. If so, push the point into the stack, otherwise push the top element of the stack out; finally, the remaining points in the stack are all the convex hull boundary key points;
[0056] 3) Sort the boundary key points after the convex hull clockwise around its geometric center, calculate the angle between them and the coordinate origin, and store the sorted boundary key points in the container;
[0057] 4) According to the angle between each boundary key point and the coordinate origin calculated in the previous step, the boundary key points in the obstacle point cloud that are within the direct illumination range of the radar are calculated and marked as valid boundary key points;
[0058] 5) From the valid boundary key points, take the direction of any two adjacent points as the direction of the rectangle, calculate the minimum area rectangle that includes all boundary key points, and finally output the rectangle with the smallest area as the two-dimensional border of the obstacle, and record the coordinates of the four vertices of the two-dimensional rectangle;
[0059] 6) According to the maximum and minimum Z coordinates in the clustered point cloud, the four vertex coordinates calculated in step 5) are assigned the minimum Z coordinate and the maximum Z coordinate values, respectively, that is, the vertex coordinates of the three-dimensional bounding box are obtained, and the three-dimensional bounding box of the obstacle is obtained.
[0060] The method of the present invention can detect objects and calculate their heights through a clustering algorithm, distinguish point clouds reflected by obstacles from clutter, thereby deleting clutter and false points, and can effectively filter out 80% of clutter in the environment, thereby accurately identifying point clouds reflected by the surface contour of objects, and providing accurate data for constructing a three-dimensional border of obstacles.
[0061] Finally, it should be noted that the above examples of the present invention are merely examples for illustrating the present invention, and are not intended to limit the embodiments of the present invention. Although the applicant has described the present invention in detail with reference to the preferred embodiments, for those of ordinary skill in the art, other different forms of changes and modifications can be made based on the above description. It is impossible to list all the embodiments here. Any obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud is characterized by The steps include: Step 1, obtain 4D millimeter wave radar point cloud data through Ethernet; Step 2, point cloud data preprocessing; Step 3, clustering the point cloud data in three-dimensional space to identify obstacles from the point cloud data; Step 4, post-processing of clustered point cloud data; Step 5: A three-dimensional bounding box algorithm is used based on the clustered point cloud to obtain the vertex coordinates of the three-dimensional bounding box of the obstacle; The algorithm for obtaining a three-dimensional bounding box based on clustered point clouds includes the following steps: 1) Reduce the dimension of the clustered 3D point cloud and project it onto the XY coordinate plane; 2) Use the convex hull algorithm to calculate the boundary key points of the clustered point cloud. First, find the point with the smallest ordinate y among all points, recorded as p0; then calculate the cosine value of the angle between the line connecting the remaining points and this point and the x-axis, and sort these points from large to small according to their cosine values with respect to the lowest point. The sorted points are recorded as p1, p2, p3, etc.; push the lowest point p0 and the first point p1 among the sorted points into the stack, and then start calculating from p2 to calculate whether the three-point vector of the two top points of the stack and this point rotates counterclockwise. If so, push the point into the stack, otherwise push the top element of the stack out; finally, the remaining points in the stack are all the convex hull boundary key points; 3) Sort the boundary key points after the convex hull clockwise around its geometric center, calculate the angle between them and the coordinate origin, and store the sorted boundary key points in the container; 4) According to the angle between each boundary key point and the coordinate origin calculated in the previous step, the boundary key points in the obstacle point cloud that are directly illuminated by the radar are calculated and marked as valid boundary key points; 5) From the valid boundary key points, take the direction of any two adjacent points as the direction of the rectangle, calculate the minimum area rectangle that includes all boundary key points, and finally output the rectangle with the smallest area as the two-dimensional border of the obstacle, and record the coordinates of the four vertices of the two-dimensional rectangle; 6) According to the maximum and minimum Z coordinates in the clustered point cloud, the four vertex coordinates calculated in step 5) are assigned the minimum Z coordinate and the maximum Z coordinate values, respectively, that is, the vertex coordinates of the three-dimensional bounding box are obtained, and the three-dimensional bounding box of the obstacle is obtained.
2. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to claim 1, characterized in that: In step 1, when the radar point cloud data is processed, a frame of data obtained by the 4D millimeter wave radar is first converted into physical values. According to the vehicle coordinate system definition and the trigonometric function relationship, the distance, azimuth and pitch angle values are converted into X, Y, and Z coordinates, and then the coordinate values are converted to the vehicle coordinate system and stored in the global variable.
3. The obstacle three-dimensional border detection method based on 4D millimeter wave radar point cloud according to claim 2 is characterized in that: The clustering of point cloud data in three-dimensional space described in step 3 specifically includes the following steps: 1) Initialize a point object container and store the point cloud data in the container. The point object contains the following member variables: coordinates X / Y / Z, ID, whether it has been visited, cluster ClusterID, whether it is a core point flag IsKey, and neighborhood point list; 2) Set clustering parameters, including neighborhood radius R, minimum number of neighborhood data MinPts, and data dimension Dimension; 3) Select a point from the point cloud data at random, mark the Visited attribute of the point as visited, calculate the number of points PtsNum within the neighborhood radius R of the point, if the number of points PtsNum is greater than the set threshold MinPts, mark the IsKey attribute of the point as a core point, and assign a cluster ClusterID; at the same time, traverse all points within the neighborhood radius R of the point and perform the following operations, mark the Visited attribute of the point as visited, assign the same cluster ClusterID, and repeat steps 2) and 3) for these points until all points in the point cloud data are traversed and visited; 4) For the successfully clustered point cloud clusters, calculate the average Z coordinate of each point cloud cluster and store it in the variable ClassZInfo.
4. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to any one of claims 1 to 3, characterized in that: The post-processing of the clustered point cloud data in step 4 includes deleting the unsuccessfully clustered points and the cluster points whose height values do not meet the threshold from the point cloud data.
5. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to any one of claims 1 to 3, characterized in that: The convex hull algorithm is used to calculate the boundary key points of the clustered point cloud, and the ConvexHull algorithm is used to calculate the convex hull boundary key points of each clustered point cloud.
6. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to claim 5, characterized in that: Set the data dimension parameter Dimension to 2.
7. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to claim 3, characterized in that: Set the neighborhood radius R to 2 meters, the minimum number of neighborhood data MinPts to 12, and the data dimension Dimension to 3.
8. The obstacle 3D border detection method based on 4D millimeter wave radar point cloud according to claim 4, characterized in that: The point cloud data is preprocessed as described in step 2 to filter out noise points based on the signal-to-noise ratio and height value.
