A point cloud target detection method and device, electronic equipment, and storage medium

Abstract

This application discloses a point cloud target detection method, apparatus, electronic device, and storage medium. The method includes: acquiring a 3D point cloud collected by a lidar, and acquiring 2D views of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view; performing point cloud target detection on the 2D views from different perspectives to obtain 2D target detection boxes from different perspectives; converting the 2D target detection boxes from different perspectives into 3D target detection boxes from the corresponding perspectives in the lidar coordinate system, and updating the pose of the 3D target detection boxes from different perspectives; performing matching processing on the pose-updated 3D target detection boxes from different perspectives to obtain mutually matched 3D target detection boxes; and fusing the mutually matched 3D target detection boxes to obtain the final target detection boxes of the 3D point cloud. The technical solution of this application can improve the detection accuracy of point cloud targets.

Description

Technical Field

[0001] This application relates to the field of lidar technology, and in particular to a point cloud target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, with the increase in car ownership, traffic congestion and traffic safety issues have become increasingly prominent. In order to improve traffic efficiency, increase safety, and free drivers from fatigue, autonomous driving has come into the public eye.

[0003] Commonly used sensors in autonomous driving environmental perception include RGB cameras, LiDAR, and millimeter-wave radar. LiDAR is characterized by its low susceptibility to lighting conditions, its ability to work day and night, and its high accuracy and resolution. However, it is susceptible to adverse environmental conditions such as rain, fog, and snow.

[0004] Currently, LiDAR-based perception technologies detect traffic participants by using 3D point clouds acquired by LiDAR. Existing point cloud target detection technologies utilize bird's-eye views or front views of the 3D point cloud for target detection. Research has found that point cloud views based on a single viewpoint cannot achieve satisfactory detection results. For example, when using a 2D bird's-eye view of a 3D point cloud, the bird's-eye view loses the height and appearance features of objects, causing the detection model to incorrectly learn ground features and resulting in false detections. Similarly, when using a 2D front view of a 3D point cloud, the front view cannot effectively handle occlusion; therefore, in scenarios with significant occlusion, the detection model may lead to incorrect target localization. Summary of the Invention

[0005] In view of the above-mentioned problems existing in the prior art, this application provides a point cloud target detection method, device, electronic device, and storage medium to improve the detection accuracy of point cloud targets.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a point cloud target detection method, the method comprising:

[0008] Acquire 3D point cloud data collected by lidar, and acquire 2D views of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0009] Point cloud object detection is performed on 2D views from different perspectives to obtain 2D object detection boxes from different perspectives;

[0010] The 2D target detection boxes from different viewpoints are converted into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and the pose of the 3D target detection boxes from different viewpoints is updated.

[0011] Matching is performed on the 3D object detection boxes under different viewpoints after pose update to obtain mutually matched 3D object detection boxes;

[0012] The matched 3D object detection boxes are fused to obtain the final object detection boxes of the 3D point cloud.

[0013] Optionally, updating the pose of the 3D object detection bounding boxes from different viewpoints includes:

[0014] The speed of 3D object detection boxes from different perspectives is obtained through a multi-object tracking algorithm;

[0015] Based on the velocity of the 3D target detection box under different viewpoints, obtain the heading angle of the 3D target detection box under the corresponding viewpoint;

[0016] Obtain the rotation matrix corresponding to the 3D target detection box under the corresponding viewpoint based on the heading angle of the 3D target detection box under different viewpoints;

[0017] The 3D object detection box at each viewpoint is rotated using the rotation matrix corresponding to the 3D object detection box at different viewpoints to obtain the 3D object detection box with updated pose.

[0018] Optionally, after obtaining the pose-updated 3D object detection bounding box, the method further includes:

[0019] Cluster the point cloud within the 3D target detection box after pose update, and correct the 3D target detection box according to the clustering results to obtain the minimum 3D target detection box under different viewpoints.

[0020] The process of matching 3D object detection boxes from different viewpoints after pose update to obtain mutually matching 3D object detection boxes includes:

[0021] The minimum 3D object detection boxes under different viewpoints are matched to obtain mutually matching minimum 3D object detection boxes.

[0022] Optionally, the step of matching the minimum 3D object detection boxes from different viewpoints to obtain mutually matching minimum 3D object detection boxes includes:

[0023] A cost matrix is ​​constructed based on the intersection-union ratio of the smallest 3D object detection boxes from different perspectives.

[0024] The cost matrix is ​​optimally matched using the Hungarian matching algorithm to obtain the minimum 3D object detection bounding boxes that match each other from different viewpoints.

[0025] Optionally, the fusion of the matching 3D target detection boxes to obtain the final target detection boxes of the 3D point cloud includes:

[0026] Obtain the weight values ​​of the 3D object detection boxes for each viewpoint;

[0027] The 3D object detection boxes that match each other are weighted and fused according to the weight value corresponding to the 3D object detection box in each viewpoint to obtain the weighted fused 3D object detection box.

[0028] Optionally, the 2D target detection box in the forward-looking view includes the pixel coordinates of the upper left and lower right corners and the target category. The step of converting the 2D target detection boxes from different viewpoints into corresponding 3D target detection boxes in the lidar coordinate system includes:

[0029] Based on the pixel coordinates of the upper left and lower right corners of the 2D target detection box under the forward-looking view, the point cloud coordinates of the upper left and lower right projection points in the lidar coordinate system are obtained.

[0030] The point cloud coordinates of the center point of the 3D target detection box in the forward view are obtained based on the point cloud coordinates of the upper left and lower right projection points. The center point is the point located at the center of the upper left and lower right projection points.

[0031] The length, width, and height information of the 3D target detection box in the forward-looking view are obtained based on the relative distance between the upper left and lower right projection points.

[0032] The target category of the 3D target detection box in the forward-looking view is obtained based on the target category of the 2D target detection box in the forward-looking view.

[0033] Optionally, the 2D target detection box from the bird's-eye view includes the center point pixel coordinates, width and height information, and target category. The step of converting the 2D target detection boxes from different viewpoints into corresponding 3D target detection boxes in the lidar coordinate system includes:

[0034] The system acquires the ground clearance information of the lidar, the object height information corresponding to the target category, and the size information of each voxel unit from a bird's-eye view.

[0035] Based on the pixel coordinates of the center point of the 2D target detection box under the bird's-eye view, the ground clearance information of the lidar, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view, the point cloud coordinates of the center point of the 3D target detection box under the bird's-eye view are obtained.

[0036] Based on the width and height information of the 2D target detection box under the bird's-eye view, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view, the length, width, and height information of the 3D target detection box under the bird's-eye view are obtained.

[0037] The target category of the 3D target detection box in the bird's-eye view is obtained based on the target category of the 2D target detection box in the bird's-eye view.

[0038] Secondly, embodiments of this application also provide a point cloud target detection device, the device comprising:

[0039] A point cloud processing unit is used to acquire 3D point clouds collected by lidar and acquire 2D views of the 3D point clouds from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0040] The detection box detection unit is used to perform point cloud target detection on 2D views from different perspectives and obtain 2D target detection boxes from different perspectives.

[0041] The detection box processing unit is used to convert 2D target detection boxes from different viewpoints into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and update the pose of the 3D target detection boxes from different viewpoints.

[0042] The detection box matching unit is used to match 3D target detection boxes from different viewpoints after pose update to obtain mutually matched 3D target detection boxes.

[0043] The detection box fusion unit is used to fuse mutually matched 3D target detection boxes to obtain the final target detection boxes of the 3D point cloud.

[0044] Thirdly, embodiments of this application also provide an electronic device, including:

[0045] Processor; and

[0046] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned point cloud target detection methods.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the aforementioned point cloud target detection methods.

[0048] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The embodiments of this application first acquire 3D point clouds collected by lidar, and 2D views of the 3D point clouds from different perspectives. By performing point cloud target detection on the 2D views from different perspectives, 2D target detection boxes from different perspectives are obtained. Then, the 2D target detection boxes from different perspectives are converted into 3D target detection boxes from the corresponding perspectives in the lidar coordinate system, and the poses of the 3D target detection boxes from different perspectives are updated. Next, the 3D target detection boxes from different perspectives after the pose update are matched. By fusing the mutually matched 3D target detection boxes, the final target detection boxes of the 3D point cloud are obtained.

[0049] In this application embodiment, when obtaining 3D target detection boxes from different viewpoints, the pose update of the 3D target detection boxes provides a data basis for accurate matching of the 3D target detection boxes, and the detection accuracy of the target detection boxes is improved by fusing the 3D target detection boxes from different viewpoints that match each other. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a flowchart illustrating a point cloud target detection method in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the intersection-union ratio (IUGR) graphic effect in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of a point cloud target detection device in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The execution entity of the point cloud target detection method provided in this application embodiment can be a roadside device (e.g., roadside LiDAR or roadside computing device), a server, or a cloud control platform; the execution entity of the point cloud target detection method provided in this application embodiment can also be a perception system, a fused perception system, or a planning / control system integrating the above systems, such as an assisted driving system or an autonomous driving system. The execution entity of the point cloud target detection method in this application embodiment can be software or hardware.

[0057] Please refer to Figure 1 , Figure 1 Taking roadside equipment as an example, this application provides a point cloud target detection method. Figure 1 As shown, a point cloud target detection method provided in this application embodiment may include the following steps S110 to S150:

[0058] Step S110: Obtain the 3D point cloud collected by the lidar, and obtain the 2D view of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0059] Step S120: Perform point cloud target detection on the 2D view from different perspectives to obtain 2D target detection boxes from different perspectives.

[0060] This application embodiment uses a convolutional neural network to perform point cloud object detection on a 2D view of a 3D point cloud. The convolutional neural network is obtained, for example, by training the network structure of the fifth version of the YOLO (You Only Look Once) framework. Of course, those skilled in the art can choose other types of neural network structures, and no specific limitations are made here.

[0061] When using convolutional neural networks to perform object detection on 3D point clouds, in order to balance time complexity and detection performance, existing technologies convert disordered 3D point clouds into ordered 2D views. For example, projecting 3D point clouds into 2D bird's-eye view or projecting 3D point clouds into 2D front view from a front-view perspective, and then using convolutional neural networks to perform object detection on the regular 2D view, so as to avoid the complex computation of 3D convolution and quickly obtain detection results.

[0062] Research has found that when projecting a 3D point cloud into a 2D view from a specific perspective, some spatial features are inevitably lost, resulting in poor detection results even when using a point cloud view from a single perspective. Therefore, this application performs multi-view projection on the 3D point cloud, improving detection accuracy by fusing target detection results from different projection perspectives.

[0063] Step S130: Convert the 2D target detection boxes under different viewpoints into 3D target detection boxes under the corresponding viewpoints in the lidar coordinate system, and update the pose of the 3D target detection boxes under different viewpoints.

[0064] Because the predicted 3D object detection boxes may have different poses from different viewpoints—for example, in the front-view perspective, the target vehicle A may be moving forward, while in the bird's-eye view, it may be turning; or in the front-view perspective, the target vehicle B may be stationary, while in the bird's-eye view, it may be moving; or in the front-view perspective, the target vehicle C may be flipped over, while in the bird's-eye view, it may be driving normally, and so on—different poses may prevent the 3D object detection boxes of the same target object from being accurately matched.

[0065] Therefore, after obtaining the 3D object detection boxes from different viewpoints, this application also updates the pose of the 3D object detection boxes from different viewpoints so that the 3D object detection boxes from different viewpoints can be compared and analyzed using a set of pose evaluation criteria, so as to provide a data basis for subsequent object detection box matching.

[0066] Step S140: Perform matching processing on the 3D object detection boxes under different viewpoints after pose update to obtain mutually matched 3D object detection boxes.

[0067] This application performs 3D object detection box matching to filter whether the 3D object detection boxes of the same object under different viewpoints meet the fusion conditions. For example, if the overlap of two 3D object detection boxes under different viewpoints reaches a preset percentage, such as 90%, then these two 3D object detection boxes can be determined to be mutually matched 3D object detection boxes; otherwise, they can be determined to be mismatched 3D object detection boxes. In other words, 3D object detection boxes under different viewpoints are only fused if they meet the fusion conditions.

[0068] Step S150: The matching 3D target detection boxes are fused to obtain the final target detection boxes of the 3D point cloud.

[0069] like Figure 1As can be seen from the point cloud target detection method shown, in this embodiment, when acquiring the 3D point cloud collected by the lidar, it also acquires the 2D view of the 3D point cloud from different perspectives. By performing point cloud target detection on the 2D view from different perspectives, 2D target detection boxes are obtained from different perspectives. Then, the 2D target detection boxes from different perspectives are converted into 3D target detection boxes from the corresponding perspectives in the lidar coordinate system, and the pose of the 3D target detection boxes from different perspectives is updated. Next, the 3D target detection boxes from different perspectives after the pose update are matched. By fusing the mutually matched 3D target detection boxes, the final target detection box of the 3D point cloud is obtained.

[0070] In this embodiment, when obtaining 3D target detection boxes from different viewpoints, the pose update of the 3D target detection boxes provides a data basis for accurate matching of the 3D target detection boxes. Furthermore, by fusing the 3D target detection boxes from different viewpoints that match each other, the detection accuracy of the target detection boxes is improved.

[0071] To facilitate understanding of the various steps of the point cloud target detection method of this application, this application first introduces the methods of projecting 3D point clouds into 2D bird's-eye view from a bird's-eye view and projecting 3D point clouds into 2D front view from a front-view perspective.

[0072] First, the 3D point cloud is projected into a 2D front view from a forward-looking perspective.

[0073] When working, lidar typically rotates and scans around an axis. For a single-beam lidar, the rotational scan data can be projected into a circular shape, while for a multi-beam lidar, the rotational scan data can be projected into a cylindrical shape.

[0074] To project a 3D point cloud into a 2D front view from the front-view perspective, the point cloud in 3D space must be projected onto a deployable cylindrical surface to form a 2D plane. Assume the lidar coordinate system is constructed as an XYZ coordinate system with the right as the positive X-axis, the front as the positive Y-axis, and the top as the positive Z-axis. The horizontal resolution of the lidar's laser beam is Δθ, and the vertical resolution is Δφ. For any point P = (x, y, z, i) in the 3D point cloud, (x, y, z) are the point cloud coordinates, i is the reflection intensity, and the corresponding projection point is Pfv = (r, c), where (r, c) are the coordinates of the 2D bird's-eye view. Then, according to the following formula (1), the coordinates of the corresponding projection point in the 2D view from the front-view perspective for each point in the 3D point cloud can be calculated:

[0075]

[0076] In formula (1), atan2() is a function operator, atan2(y,x) means calculating the arctangent of y / x in radians. The function operator atan2() is more stable than atan().

[0077] According to the above formula (1), the 3D point cloud can be projected into a 2D front view under the front view. Optionally, the RGB channel pixel values ​​in the 2D front view under the front view are composed of z, i and y encoding.

[0078] Second, the 3D point cloud is projected into a 2D bird's-eye view from a bird's-eye perspective.

[0079] Before projecting the 3D point cloud into a 2D bird's-eye view from a bird's-eye perspective, first obtain the point cloud range {Xmin, Ymin, Zmin, Xmax, Ymax, Zmax} of the 3D point cloud, and assume the view size {grid_x, grid_y, grid_z} of the 2D bird's-eye view from a bird's-eye perspective, and assume the size of each voxel {pillar_x, pillar_y, pillar_z}. Then, according to the following formula (2), the coordinates of the corresponding projection point of each point in the 3D point cloud in the 2D bird's-eye view from a bird's-eye perspective can be calculated:

[0080]

[0081] In formula (2), pillar_z = Zmax - Zmin.

[0082] According to the above formula (2), the 3D point cloud can be projected into a 2D bird's-eye view. Optionally, the RGB channel pixel values ​​in the 2D bird's-eye view are composed of height, intensity, and density encoding. For each voxel unit, the maximum height value of the points in the voxel unit is used as the height feature, the reflection intensity of the point with the maximum height in the voxel unit is used as the intensity feature, and the density feature is represented by the number of points in the voxel unit. In this embodiment, in order to standardize the density feature, the formula is used. Calculate the density feature of each voxel unit, where N is the number of point clouds in each voxel unit.

[0083] Therefore, by following the above steps, 2D views of the 3D point cloud can be obtained from both bird's-eye view and front-view view. After obtaining the 2D views from different perspectives, a detection model is used to perform point cloud target detection on the 2D views.

[0084] The detection model is obtained by training a convolutional neural network, with one detection model built for each 2D viewpoint. The training process for the front-view detection model corresponding to the front-view 2D view includes:

[0085] Construct the first training dataset, which includes a large number of 2D front views and label information. The 2D front view is formed by projecting 3D point clouds according to the front view angle. The label information includes the 2D target box label {r1,c1,r2,c2,label} in the 2D front view, where (r1,c1) is the pixel coordinate of the upper left corner of the target box, (r2,c2) is the pixel coordinate of the lower right corner of the target box, and label is the target category.

[0086] The convolutional neural network is trained using the first training dataset. When the optimization objective of the convolutional neural network meets the optimization stopping condition, such as when the target box detection error is less than a preset value, the training is stopped, and the current convolutional neural network is used as the forward-looking view detection model.

[0087] Similarly, the training process for the bird's-eye view detection model corresponding to a 2D bird's-eye view includes:

[0088] Construct a second training dataset, which includes a large number of 2D bird's-eye views and label information. The 2D bird's-eye views are formed by projecting 3D point clouds according to the bird's-eye view. The label information includes the 2D target box label {x,y,w,h,name} in the 2D bird's-eye view, where (x,y) are the pixel coordinates of the center point of the target box, (w,h) are the width and height of the target box, and name is the target category.

[0089] The convolutional neural network is trained using the second training dataset. When the optimization objective of the convolutional neural network meets the optimization stopping condition, such as when the target box detection error is less than a preset value, the training is stopped, and the current convolutional neural network is used as the bird's-eye view detection model.

[0090] Thus, after obtaining the forward-looking view detection model and the bird's-eye view detection model, the 2D front view obtained in step S110 is input into the forward-looking view detection model to obtain the 2D target detection box {r_p1,c_p1,r_p2,c_p2,label_p} in the forward-looking view; and the 2D bird's-eye view obtained in step S110 is input into the bird's-eye view detection model to obtain the 2D target detection box {x_p,y_p,w_p,h_p,name_p} in the bird's-eye view.

[0091] Then, the 2D target detection boxes from different perspectives are converted into 3D target detection boxes from the corresponding perspectives in the lidar coordinate system.

[0092] For the 2D object detection bounding box {r_p1,c_p1,r_p2,c_p2,label_p} in the forward-looking view, the transformation process is as follows:

[0093] First, based on the pixel coordinates (r_p1, c_p1) and (r_p2, c_p2) of the upper left corner point p1 and lower right corner point p2 of the 2D target detection box under the forward view, the point cloud coordinates (X1, Y1, Z1) and (X2, Y2, Z2) of the upper left corner projection point and lower right corner projection point under the lidar coordinate system are obtained.

[0094] in Y1 = f3(r_p1, c_p1), Y2 = f3(r_p2, c_p2), As mentioned earlier, the RGB channel pixel values ​​in the 2D view from the front-view perspective are composed of z, i and y encodings, i.e. z = f1(r,c), i = f2(r,c), y = f3(r,c).

[0095] Then, based on the point cloud coordinates of the upper left and lower right projection points, the point cloud coordinates (Xc1, Yc1, Zc1) of the center point of the 3D target detection box in the forward view are obtained, where the center point is the point located at the center of the upper left and lower right projection points.

[0096] Where Xc1 = 0.5 * (X1 + X2), Yc1 = (Y1 + Y2) * 0.5, Zc1 = (Z1 + Z2) * 0.5.

[0097] Next, the length, width, and height information (W1, L1, H1) of the 3D target detection box in the forward-looking view is obtained based on the relative distance between the upper left and lower right projection points.

[0098] Where W1 = X2 - X1, L1 = Y2 - Y1, H1 = Z2 - Z1.

[0099] Finally, based on the target category of the 2D target detection box in the forward-looking view, the target category label1 of the 3D target detection box in the forward-looking view is obtained, i.e., label1 = label_p.

[0100] That is, the 3D target detection box in the lidar coordinate system corresponding to the 2D target detection box in the forward-looking view is {Xc1,Yc1,Zc1,W1,L1,H1,label1}.

[0101] For a 2D object detection bounding box {x_p,y_p,w_p,h_p,name_p} viewed from a bird's-eye view, the transformation process is as follows:

[0102] First, obtain the ground clearance information H_lidar of the lidar, the height information H_obj of the target object corresponding to the target category, and the size information {pillar_x,pillar_y,pillar_z} of each voxel unit from the bird's-eye view.

[0103] The ground clearance information (H_lidar) of the lidar and the height information of the target object (H_obj) can be obtained from the configuration file. After calibration, the lidar records calibration parameters and ground clearance information into its configuration parameters. Furthermore, the height information of different target objects, such as vehicles and pedestrians, can be pre-recorded in this configuration file. When needed, the relevant parameter values ​​are retrieved from the configuration file.

[0104] Then, based on the pixel coordinates (x_p, y_p) of the center point of the 2D target detection box under the bird's-eye view, the ground height information H_lidar of the lidar, the height information H_obj of the target object corresponding to the target category, and the size information {pillar_x, pillar_y, pillar_z} of each voxel unit under the bird's-eye view, the point cloud coordinates (Xc2, Yc2, Zc2) of the center point of the 3D target detection box under the bird's-eye view are obtained.

[0105] Where Xc2 = x_p * pillar_x, Yc2 = y_p * pillar_y, and Zc2 = -H_lidar + H_obj / 2.

[0106] Next, based on the width and height information (w_p, h_p) of the 2D target detection box under the bird's-eye view, the height information H_obj of the target object corresponding to the target category, and the size information {pillar_x, pillar_y, pillar_z} of each voxel unit under the bird's-eye view, the length, width, and height information (W2, L2, H2) of the 3D target detection box under the bird's-eye view are obtained.

[0107] Where W2 = h_p * pillar_y, L2 = w_p * pillar_x, and H2 = H_obj.

[0108] Finally, based on the target category of the 2D target detection box in the bird's-eye view, the target category name2 of the 3D target detection box in the bird's-eye view is obtained, that is, name2 = name_p.

[0109] That is, the 3D target detection box in the lidar coordinate system corresponding to the 2D target detection box in the bird's-eye view is {Xc2,Yc2,Zc2,W2,L2,H2,name2}.

[0110] Therefore, this application converts 2D target detection boxes from different viewpoints into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system through the above embodiments. Since the poses of the 3D target detection boxes differ depending on the viewpoint, and these poses may lead to inaccurate matching of the 3D target detection boxes.

[0111] Considering that the heading angle information of the target detection box indicates the attitude of the target detection box, this application first obtains the velocity of the 3D target detection box through a multi-target tracking algorithm, calculates the heading angle based on the velocity, calculates the rotation matrix based on the heading angle, and updates the attitude of the 3D target detection box through the rotation matrix.

[0112] Specifically, in one embodiment of this application, updating the pose of the 3D object detection bounding box from different viewpoints includes:

[0113] The velocity of the 3D target detection box under different viewpoints is obtained through a multi-target tracking algorithm; the heading angle of the 3D target detection box under the corresponding viewpoint is obtained based on the velocity of the 3D target detection box under different viewpoints; the rotation matrix of the 3D target detection box under the corresponding viewpoint is obtained based on the heading angle of the 3D target detection box under the corresponding viewpoint; the 3D target detection box under the corresponding viewpoint is rotated using the rotation matrix of the 3D target detection box under different viewpoints to obtain the 3D target detection box with updated attitude.

[0114] For example, by inputting the 3D target detection boxes {Xc1,Yc1,Zc1,W1,L1,H1,label1} from the forward-looking view and {Xc2,Yc2,Zc2,W2,L2,H2,name2} from the bird's-eye view into a multi-target tracking algorithm, the velocities of the target object on the XOY plane can be calculated as {Vx1,Vy1} and {Vx2,Vy2}. Assuming the heading angles are heading1 and heading2, then heading1 = arctan(Vy1,Vx1) and heading2 = arctan(Vy2,Vx2). Accordingly, the rotation matrix R can be calculated using formula (3).

[0115]

[0116] After the 3D target detection boxes in the forward-looking view and the bird's-eye view are transformed by the corresponding rotation matrix, 3D target detection boxes with heading angle information in the lidar coordinate system are obtained, namely {Xc1,Yc1,Zc1,W1,L1,H1,heading1,label1} and {Xc2,Yc2,Zc2,W2,L2,H2,heading2,name2}, respectively.

[0117] Accordingly, step S140 specifically involves constructing a cost matrix based on the intersection-union ratio of the 3D object detection boxes under different viewpoints after pose update; and performing optimal matching of the cost matrix using the Hungarian matching algorithm to obtain mutually matched 3D object detection boxes under different viewpoints.

[0118] It should be noted that when the pose of the 3D object detection box changes, the 3D object detection box may not be the optimal object detection box for the point cloud cluster within it. Therefore, in some embodiments of this application, the point cloud within the pose-updated 3D object detection box is clustered, and the 3D object detection box is corrected based on the clustering results to obtain the minimum 3D object detection box under different viewpoints.

[0119] The embodiments of this application, by performing pose updates and corrections on the 3D object detection boxes, can obtain the optimal 3D object detection boxes, providing a data foundation for accurate matching of 3D object detection boxes from different viewpoints.

[0120] As mentioned earlier, the predicted 3D object detection boxes are not the same from different viewpoints. Therefore, it is necessary to fuse the 3D object detection boxes from different viewpoints to obtain the final object detection box. The matching process for the 3D object detection boxes includes:

[0121] A cost matrix is ​​constructed based on the intersection-union ratio (IUU) of the smallest 3D object detection boxes under different viewpoints. The cost matrix is ​​then optimally matched using the Hungarian matching algorithm to obtain the smallest 3D object detection boxes that match each other under different viewpoints.

[0122] Assume that the 2D view from the forward-looking perspective corresponding to the 3D point cloud of the current frame identifies m 3D object detection boxes, and the 2D view from the bird's-eye view identifies n 3D object detection boxes. The cost matrix is ​​set as cost = (c ij ) m×n The matrix elements are 1-IoU, where IoU is the intersection-union ratio of two 3D object detection boxes. For ease of viewing, the two 3D object detection boxes are simplified to... Figure 2 The two rectangular boxes shown have an Intersection over Union (IoU) of IoU = (A∩B) / (A∪B). Figure 2 As shown, IoU is a positive number greater than or equal to 0 and less than or equal to 1. The larger the IoU value, the higher the overlap between the two 3D object detection boxes, the better they match, and the higher the matching score. Therefore, in this embodiment, 1-IoU is used as an element to construct the cost matrix, that is, the expression of the cost matrix is ​​as shown in formula (4):

[0123]

[0124] Since the matching problem in this application is a bipartite graph matching problem, optionally, this application uses the Hungarian matching algorithm to perform optimal matching on the cost matrix, records the 3D object detection boxes under different viewpoints that match each other in the index list, and discards the unmatched 3D object detection boxes.

[0125] Then, obtain the weight value corresponding to the 3D object detection box under each viewpoint. For example, the weight value of the 3D object detection box under the forward viewpoint is λ, and the weight value of the 3D object detection box under the bird's-eye viewpoint is 1-λ. According to the weight value corresponding to the 3D object detection box under each viewpoint, perform weighted fusion on the matching 3D object detection boxes to obtain the weighted fused 3D object detection box {Xc,Yc,Zc,W,L,H,heading,name}.

[0126] Among them, Xc = λXc1 + (1-λ) +(1-λ)L2, H=λH1+(1-λ)H2, heading=λheading1+(1-λ)heading2, name=label1=name2.

[0127] Here, the λ value can be set according to the matching effect. For example, all possible values ​​of λ can be set in advance, and the weighted fusion target detection box corresponding to each possible value can be calculated. The final detection box is selected from these weighted fusion target detection boxes as the final target detection box.

[0128] This application also provides a roadside point cloud target detection device 300, such as... Figure 3 The diagram shows a roadside point cloud target detection device according to an embodiment of this application. The device 300 includes: a point cloud processing unit 310, a detection box recognition unit 320, a detection box processing unit 330, a detection box matching unit 340, and a detection box fusion unit 350, wherein:

[0129] The point cloud processing unit 310 is used to acquire 3D point clouds collected by lidar and acquire 2D views of the 3D point clouds from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0130] The detection box detection unit 320 is used to perform point cloud target detection on 2D views from different perspectives and obtain 2D target detection boxes from different perspectives.

[0131] The detection box processing unit 330 is used to convert 2D target detection boxes from different viewpoints into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and update the pose of the 3D target detection boxes from different viewpoints.

[0132] The detection box matching unit 340 is used to perform matching processing on 3D target detection boxes under different viewpoints after pose update, so as to obtain mutually matched 3D target detection boxes.

[0133] The detection box fusion unit 350 is used to fuse mutually matched 3D target detection boxes to obtain the final target detection boxes of the 3D point cloud.

[0134] In one embodiment of this application, the detection frame processing unit 330 includes an update module and a correction module;

[0135] The update module is used to obtain the velocity corresponding to the 3D target detection box under different viewpoints through a multi-target tracking algorithm; obtain the heading angle corresponding to the 3D target detection box under the corresponding viewpoint based on the velocity corresponding to the 3D target detection box under different viewpoints; obtain the rotation matrix corresponding to the 3D target detection box under the corresponding viewpoint based on the heading angle corresponding to the 3D target detection box under the corresponding viewpoint; and rotate the 3D target detection box under the corresponding viewpoint using the rotation matrix corresponding to the 3D target detection box under different viewpoints to obtain the 3D target detection box after attitude update.

[0136] The correction module is used to cluster the point cloud within the 3D target detection box after the pose update, and correct the 3D target detection box according to the clustering result to obtain the minimum 3D target detection box under different viewpoints.

[0137] Correspondingly, the detection box matching unit 340 is used to perform matching processing on the smallest 3D target detection boxes under different viewpoints to obtain mutually matched smallest 3D target detection boxes.

[0138] In one embodiment of this application, the detection box matching unit 340 is further configured to construct a cost matrix based on the cross-union ratio of the detection boxes between the smallest 3D target detection boxes under different viewpoints; and to perform optimal matching of the cost matrix using the Hungarian matching algorithm to obtain the smallest 3D target detection boxes that match each other under different viewpoints.

[0139] In one embodiment of this application, the detection box fusion unit 350 is further configured to obtain the weight value corresponding to the 3D target detection box under each viewpoint; and to perform weighted fusion on the mutually matched 3D target detection boxes according to the weight value corresponding to the 3D target detection box under each viewpoint to obtain the weighted fused 3D target detection box.

[0140] In one embodiment of this application, the 2D target detection box under the forward-looking view includes the pixel coordinates of the upper left and lower right corners and the target category. The detection box processing unit 330 is further configured to: obtain the point cloud coordinates of the upper left and lower right projection points in the lidar coordinate system based on the pixel coordinates of the upper left and lower right corners of the 2D target detection box under the forward-looking view; obtain the point cloud coordinates of the center point of the 3D target detection box under the forward-looking view based on the point cloud coordinates of the upper left and lower right projection points, wherein the center point is the point located at the center of the upper left and lower right projection points; obtain the length, width, and height information of the 3D target detection box under the forward-looking view based on the relative distance between the upper left and lower right projection points; and obtain the target category of the 3D target detection box under the forward-looking view based on the target category of the 2D target detection box under the forward-looking view.

[0141] In one embodiment of this application, the 2D target detection box under the bird's-eye view includes center point pixel coordinates, width and height information, and target category. The detection box processing unit 330 is further configured to acquire the ground clearance information of the lidar, the height information of the object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view; acquire the point cloud coordinates of the center point of the 3D target detection box under the bird's-eye view based on the center point pixel coordinates of the 2D target detection box under the bird's-eye view, the ground clearance information of the lidar, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view; acquire the length, width, and height information of the 3D target detection box under the bird's-eye view based on the width and height information of the 2D target detection box under the bird's-eye view, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view; and acquire the target category of the 3D target detection box under the bird's-eye view based on the target category of the 2D target detection box under the bird's-eye view.

[0142] It is understood that the above-described point cloud target detection device can implement all the steps of the point cloud target detection method provided in the foregoing embodiments. The relevant explanations of the point cloud target detection method are applicable to the point cloud target detection device, and will not be repeated here.

[0143] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0144] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0145] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0146] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a point cloud target detection device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0147] Acquire 3D point cloud data collected by lidar, and acquire 2D views of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0148] Point cloud object detection is performed on 2D views from different perspectives to obtain 2D object detection boxes from different perspectives;

[0149] The 2D target detection boxes from different viewpoints are converted into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and the pose of the 3D target detection boxes from different viewpoints is updated.

[0150] Matching is performed on the 3D object detection boxes under different viewpoints after pose update to obtain mutually matched 3D object detection boxes;

[0151] The matched 3D object detection boxes are fused to obtain the final object detection boxes of the 3D point cloud.

[0152] The above is as stated in this application. Figure 1 The method executed by the point cloud target detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory, and the processor reads the information from the memory and, in conjunction with its hardware, completes the steps of the point cloud target detection method described above.

[0153] The electronic device can also perform Figure 1 The method for executing a point cloud target detection device, and the implementation of the point cloud target detection device in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0154] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the point cloud target detection device in the illustrated embodiment is specifically used to perform the following:

[0155] Acquire 3D point cloud data collected by lidar, and acquire 2D views of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view.

[0156] Point cloud object detection is performed on 2D views from different perspectives to obtain 2D object detection boxes from different perspectives;

[0157] The 2D target detection boxes from different viewpoints are converted into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and the pose of the 3D target detection boxes from different viewpoints is updated.

[0158] Matching is performed on the 3D object detection boxes under different viewpoints after pose update to obtain mutually matched 3D object detection boxes;

[0159] The matched 3D object detection boxes are fused to obtain the final object detection boxes of the 3D point cloud.

[0160] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0166] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0167] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A point cloud target detection method, characterized in that, The method includes: Acquire 3D point cloud data collected by lidar, and acquire 2D views of the 3D point cloud from different perspectives, including at least a bird's-eye view and a forward-looking view. Point cloud object detection is performed on 2D views from different perspectives to obtain 2D object detection boxes from different perspectives; The 2D target detection boxes from different viewpoints are converted into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and the pose of the 3D target detection boxes from different viewpoints is updated. Matching is performed on the 3D object detection boxes under different viewpoints after pose update to obtain mutually matched 3D object detection boxes; The matching 3D target detection boxes are fused to obtain the final target detection boxes of the 3D point cloud. The 2D target detection bounding box in the forward-looking view includes the pixel coordinates of the upper left and lower right corners and the target category. Converting the 2D target detection bounding boxes from different viewpoints into corresponding 3D target detection bounding boxes in the lidar coordinate system includes: Based on the pixel coordinates of the upper left and lower right corners of the 2D target detection box under the forward-looking view, the point cloud coordinates of the upper left and lower right projection points in the lidar coordinate system are obtained. The point cloud coordinates of the center point of the 3D target detection box in the forward view are obtained based on the point cloud coordinates of the upper left and lower right projection points. The center point is the point located at the center of the upper left and lower right projection points. The length, width, and height information of the 3D target detection box in the forward-looking view are obtained based on the relative distance between the upper left and lower right projection points. The target category of the 3D target detection box in the forward-looking view is obtained based on the target category of the 2D target detection box in the forward-looking view.

2. The point cloud target detection method as described in claim 1, characterized in that, The updating of the pose of the 3D object detection bounding boxes from different viewpoints includes: The speed of 3D object detection boxes from different perspectives is obtained through a multi-object tracking algorithm; Based on the velocity of the 3D target detection box under different viewpoints, obtain the heading angle corresponding to the 3D target detection box under the corresponding viewpoint; Obtain the rotation matrix corresponding to the 3D target detection box under the corresponding viewpoint based on the heading angle of the 3D target detection box under different viewpoints; The 3D object detection box at each viewpoint is rotated using the rotation matrix corresponding to the 3D object detection box at different viewpoints to obtain the 3D object detection box with updated pose.

3. The point cloud target detection method as described in claim 1, characterized in that, After obtaining the pose-updated 3D object detection bounding boxes, the following is also included: Cluster the point cloud within the 3D target detection box after pose update, and correct the 3D target detection box according to the clustering results to obtain the minimum 3D target detection box under different viewpoints. The process of matching 3D object detection boxes from different viewpoints after pose update to obtain mutually matching 3D object detection boxes includes: The minimum 3D object detection boxes under different viewpoints are matched to obtain mutually matching minimum 3D object detection boxes.

4. The point cloud target detection method as described in claim 3, characterized in that, The process of matching the minimum 3D object detection boxes from different viewpoints to obtain mutually matching minimum 3D object detection boxes includes: A cost matrix is ​​constructed based on the intersection-union ratio of the smallest 3D object detection boxes from different perspectives. The cost matrix is ​​optimally matched using the Hungarian matching algorithm to obtain the minimum 3D object detection bounding boxes that match each other from different viewpoints.

5. The point cloud target detection method as described in claim 1, characterized in that, The process of fusing the matched 3D object detection boxes to obtain the final object detection boxes of the 3D point cloud includes: Obtain the weight values ​​of the 3D object detection boxes for each viewpoint; The 3D object detection boxes that match each other are weighted and fused according to the weight value corresponding to the 3D object detection box in each viewpoint to obtain the weighted fused 3D object detection box.

6. The method as described in claim 1, characterized in that, The 2D target detection bounding box from the bird's-eye view includes the center point pixel coordinates, width and height information, and target category. Converting the 2D target detection bounding boxes from different viewpoints into corresponding 3D target detection bounding boxes in the LiDAR coordinate system includes: The system acquires the ground clearance information of the lidar, the object height information corresponding to the target category, and the size information of each voxel unit from a bird's-eye view. Based on the pixel coordinates of the center point of the 2D target detection box under the bird's-eye view, the ground clearance information of the lidar, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view, the point cloud coordinates of the center point of the 3D target detection box under the bird's-eye view are obtained. Based on the width and height information of the 2D target detection box under the bird's-eye view, the height information of the target object corresponding to the target category, and the size information of each voxel unit under the bird's-eye view, the length, width, and height information of the 3D target detection box under the bird's-eye view are obtained. The target category of the 3D target detection box in the bird's-eye view is obtained based on the target category of the 2D target detection box in the bird's-eye view.

7. A point cloud target detection device, characterized in that, The device includes: A point cloud processing unit is used to acquire 3D point clouds collected by lidar and acquire 2D views of the 3D point clouds from different perspectives, including at least a bird's-eye view and a forward-looking view. The detection box detection unit is used to perform point cloud target detection on 2D views from different perspectives and obtain 2D target detection boxes from different perspectives. The detection box processing unit is used to convert 2D target detection boxes from different viewpoints into 3D target detection boxes from the corresponding viewpoints in the lidar coordinate system, and update the pose of the 3D target detection boxes from different viewpoints. The detection box matching unit is used to match 3D target detection boxes from different viewpoints after pose update to obtain mutually matched 3D target detection boxes. The detection box fusion unit is used to fuse mutually matched 3D target detection boxes to obtain the final target detection boxes of the 3D point cloud. The 2D target detection bounding box in the forward-looking view includes the pixel coordinates of the upper left and lower right corners and the target category. The detection bounding box processing unit is specifically used for: Based on the pixel coordinates of the upper left and lower right corners of the 2D target detection box under the forward-looking view, the point cloud coordinates of the upper left and lower right projection points in the lidar coordinate system are obtained. The point cloud coordinates of the center point of the 3D target detection box in the forward view are obtained based on the point cloud coordinates of the upper left and lower right projection points. The center point is the point located at the center of the upper left and lower right projection points. The length, width, and height information of the 3D target detection box in the forward-looking view are obtained based on the relative distance between the upper left and lower right projection points. The target category of the 3D target detection box in the forward-looking view is obtained based on the target category of the 2D target detection box in the forward-looking view.

8. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the point cloud target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the point cloud target detection method according to any one of claims 1 to 6.

Images