A registration method and device for heterogeneous point cloud fusion under road-vehicle cooperation

Abstract

This application provides a registration method and apparatus for heterogeneous point cloud fusion under vehicle-road cooperative systems, relating to the field of autonomous driving technology. The method includes: segmenting vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data using a Polarseg semantic segmenter; segmenting road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data; fusing the morphologically noise-removed vehicle-side ground point cloud data and the outlier-removed vehicle-side non-ground point cloud data to obtain fused vehicle-side point cloud data; fusing the morphologically noise-removed road-side ground point cloud data and the outlier-removed road-side non-ground point cloud data to obtain fused road-side point cloud data; and using the fused road-side point cloud data as the source and the fused vehicle-side point cloud data as the target, matching the source and target using a global point cloud matching method to obtain a pose transformation matrix from the source to the target. This application improves the registration accuracy of heterogeneous point clouds.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a registration method and apparatus for heterogeneous point cloud fusion under road-vehicle cooperation. Background Technology

[0002] LiDAR is installed on autonomous vehicles. However, because the vehicle is constantly moving, the LiDAR sensors on the vehicle are easily obstructed, and long-distance data may be missing. This makes it difficult for the vehicle to make accurate pose estimation at urban intersections.

[0003] Existing vehicle-to-infrastructure (V2I) solutions address the limitation of vehicle-side radar perception by designing complex V2I matching methods to homogeneous point clouds acquired by similar sensors on either the vehicle or the roadside. For example, multi-view image and map-based interactive perception fusion methods map sensor observation data into a semantic description of the vehicle environment, matching homogeneous point cloud data acquired by the original radar sensors of multiple vehicles, and combining homogeneous point cloud data acquired by multiple radar sensors at different spatial locations. Homogeneous point cloud data refers to point cloud data acquired by radar sensors of the same type. Most existing solutions simply align homogeneous point cloud data from different viewpoints.

[0004] There is currently no solution for the registration of heterogeneous point clouds. Summary of the Invention

[0005] In view of this, this application provides a registration method and apparatus for heterogeneous point cloud fusion under road-vehicle cooperation to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a registration method for heterogeneous point cloud fusion under road-vehicle cooperation, including:

[0007] Acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data for the same target scene;

[0008] The Polarseg semantic segmenter is used to segment vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data; the Polarseg semantic segmenter is used to segment road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data.

[0009] The radius density clustering method was used to process the vehicle-side ground point cloud data and the road-side ground point cloud data respectively, resulting in vehicle-side ground point cloud data and road-side ground point cloud data with morphological noise removed.

[0010] The vehicle-side non-ground point cloud data and the road-side non-ground point cloud data are processed to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed.

[0011] The vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed are fused to obtain the fused vehicle-side point cloud data; the road-side ground point cloud data with morphological noise removed and the road-side non-ground point cloud data with outliers removed are fused to obtain the fused road-side point cloud data.

[0012] Using the fused roadside point cloud data as the source and the fused vehicle point cloud data as the target, a global point cloud matching method is used to match the source and target to obtain the pose transformation matrix from the source to the target.

[0013] Furthermore, the vehicle-side non-ground point cloud data and the road-side non-ground point cloud data are processed to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed; including:

[0014] The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data of the vehicle to obtain the target prediction results.

[0015] Based on the vehicle-side target prediction results, a statistical outlier removal method is used to remove outliers from the vehicle-side non-ground point cloud data.

[0016] The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data at the roadside to obtain the target prediction results at the roadside.

[0017] Based on the roadside target prediction results, the statistical outlier removal method is used to remove outliers in the non-ground point cloud data of the roadside.

[0018] For each point in the vehicle-side non-ground point cloud data after removing outliers, add several points in its neighborhood to make the density of each target point cloud close to the density of the corresponding target point cloud in the road-side non-ground point cloud data.

[0019] Furthermore, when the target is a car, the method further includes:

[0020] Connect the outer contour points of the vehicle in the non-ground point cloud data at the roadside to form a line, which corresponds to the outer contour line of the vehicle in the non-ground point cloud data at the roadside.

[0021] Furthermore, using the fused roadside point cloud data as the source and the fused vehicle point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a pose transformation matrix from the source to the target; including:

[0022] The fused vehicle endpoint cloud data is denoted as point set P = {p}, where p represents a point in the fused vehicle endpoint cloud data; the fused road endpoint cloud data is denoted as point set Q = {q}, where q represents a point in the fused road endpoint cloud data.

[0023] Calculate the normal vectors of point set P and point set Q;

[0024] Calculate the feature F(P) of point set P based on the normal vector of point set P: F(P) = {F(p): p∈P}; Calculate the feature F(Q) of point set Q based on the normal vector of point set Q: F(Q) = {F(q): q∈Q};

[0025] For each point p in point set P, find the point q in point set Q that has the smallest distance from point p. p For each point q in point set Q, find the point p in point set P that has the smallest distance to point q. q Thus, we obtain the set of point pairs K1 = {(p,q} p ),(q,p q The distance between two points is the difference in their features.

[0026] Perform a mutuality test on each pair of points in the pair set K1, and add the points that pass the mutuality test to the pair set K2;

[0027] Perform a multivariability test on each pair of points in the pair set K2, and add the points that pass the multivariability test to the pair set K3;

[0028] Establish the first objective function E1:

[0029]

[0030] Where, vector Let P be the three-dimensional coordinates of point P; let T be the vector. q for:

[0031]

[0032] (x q ,u q ,z q Let q be the three-dimensional coordinates of point q, and T be the 4×4 pose transformation matrix; l p,q This represents the distance between points p and q in the point pair (p,q) of the set of point pairs K3;

[0033] The initial pose transformation matrix T0 is calculated using the morphological noise-removed vehicle-side ground point cloud data and the morphological noise-removed road-side ground point cloud data.

[0034] Based on the initial pose transformation matrix T0, Gauss-Newton iterations are used. When the first objective function is less than a preset threshold or the number of iterations is reached, the final 4×4 pose transformation matrix T is obtained.

[0035]

[0036] Wherein, coefficient b 14 b 24 and b 34 This represents the translation value in three dimensions, with coefficient b. 12 The coefficient b represents the angle of rotation. 13 The pitch angle is represented by the coefficient b. 23 Indicates the angle of yaw.

[0037] Furthermore, a multivariability test is performed on each point in the point-pair set K2, and points that pass the multivariability test are added to the point-pair set K3; including:

[0038] For any point (p1, q1) in the set of point pairs K2, if for any two points in the set of point pairs K2 other than (p1, q1): (p, q2) and (p3, q3), the following holds true:

[0039]

[0040] Where τ = 0.9; i = 1, 2, 3; j = 1, 2, 3;

[0041] Then, if (p1,q1) passes the multivariability test, (p1,q1) is added to the point pair set K3.

[0042] Furthermore, using morphologically noise-removed vehicle-side ground point cloud data and morphologically noise-removed road-side ground point cloud data, the initial pose transformation matrix T0 is calculated; including:

[0043] Using the roadside ground point cloud data with morphological noise removed as the source end and the vehicle-side ground point cloud data with morphological noise removed as the target end, the source end and the target end are matched using a global point cloud matching method to obtain the three-degree-of-freedom pose transformation matrix F from the source end to the target end.

[0044] The ground plane model parameters of the vehicle-end ground are calculated based on the ground point cloud data with morphological noise removed. Based on the ground plane model parameters of the vehicle-end ground, any vehicle-end ground point is obtained, and its three-dimensional coordinates are (x1, y1, z1).

[0045] The ground plane model parameters of the roadside ground are calculated based on the roadside ground point cloud data after removing morphological noise; any roadside ground point is obtained based on the ground plane model parameters of the roadside ground, and its three-dimensional coordinates are (x2, y2, z2).

[0046] Calculate the rotation matrix R between the vehicle-end ground point and the road-end ground point:

[0047]

[0048] Calculate the initial value T0 of the pose transformation matrix:

[0049]

[0050] Furthermore, using morphologically noise-removed roadside ground point cloud data as the source and morphologically noise-removed vehicle-side ground point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a three-degree-of-freedom pose transformation matrix from the source to the target; including:

[0051] The noise-free vehicle-side ground point cloud data is denoted as point set M = {m}, where m represents a point in the road-side ground point cloud data; the noise-free road-side ground point cloud data is denoted as point set N = {n}, where n represents a point in the road-side ground point cloud data.

[0052] Calculate the normal vectors of point set M and point set N;

[0053] Calculate the feature F(M) of point set M based on the normal vectors of point set M: F(M) = {F(m): m ∈ M}; Calculate the feature F(N) of point set N based on the normal vectors of point set N: F(N) = {F(n): n ∈ N};

[0054] For each point m in point set M, find the point n with the smallest distance from point m in point set N. m For each point n in the point set N, find the point m in the point set N that has the smallest distance from point n. n Thus, we obtain the set of point pairs O1 = {(m,n)} m ),(n,m n The distance between two points is the difference in their features.

[0055] Perform a mutuality test on each pair of points in the point pair set O1, and put the points that pass the mutuality test into the point pair set O2; perform a multivariability test on each pair of points in the point pair set O2, and put the points that pass the multivariability test into the point pair set O3.

[0056] Establish the second objective function E2:

[0057]

[0058] Where, vector Let m be the three-dimensional coordinates of point m; let T be the vector. n for:

[0059]

[0060] (x n ,y n ,1) are the three-dimensional coordinates of point n; l m,n This represents the distance between points m and n in the pair (m,n) of the set of point pairs O3.

[0061] Using Gauss-Newton iterations, when the second objective function is less than a preset threshold or the number of iterations is reached, the final three-degree-of-freedom pose transformation matrix F is obtained:

[0062]

[0063] Wherein, coefficient a 13 and a 23 Represents the two-dimensional translation value; coefficient a 12 Indicates the angle of rotation.

[0064] Furthermore, the ground plane model parameters of the vehicle-end surface are calculated based on the morphological noise-removed point cloud data; including:

[0065] Step S1: Randomly select three points from the vehicle-end ground point cloud data after removing morphological noise, and calculate the ground plane model parameters A, B, C and D according to the plane equation Ax+By+Cz+D=0;

[0066] Step S2: Calculate the distance from the remaining points in the vehicle-mounted ground point cloud data to the plane, compare each distance with the set threshold, mark points less than the threshold as inner points, and mark the rest as outer points;

[0067] Step S3: Count the number of interior points; Proceed to step S1; Continue until the number of iterations exceeds the threshold, then proceed to step S4;

[0068] Step S4: Obtain the ground plane model parameters corresponding to the maximum number of interior points.

[0069] Secondly, embodiments of this application provide a registration device for heterogeneous point cloud fusion under road-vehicle cooperation, comprising:

[0070] The acquisition unit is used to acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data of the same target scene;

[0071] The semantic segmentation unit is used to segment vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data using the Polarseg semantic segmenter; and to segment road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data using the Polarseg semantic segmenter.

[0072] The first processing unit is used to process the vehicle-end ground point cloud data and the road-end ground point cloud data respectively using the radius density clustering method to obtain vehicle-end ground point cloud data and road-end ground point cloud data with morphological noise removed.

[0073] The second processing unit is used to process the vehicle-side non-ground point cloud data and the road-side non-ground point cloud data to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed.

[0074] The fusion unit is used to fuse the vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed to obtain fused vehicle-side point cloud data; and to fuse the road-side ground point cloud data with morphological noise removed and the road-side non-ground point cloud data with outliers removed to obtain fused road-side point cloud data.

[0075] The computing unit is used to match the source end and the target end using the fused road end point cloud data as the source end and the fused vehicle end point cloud data as the target end, and obtain the pose transformation matrix from the source end to the target end using the global point cloud matching method.

[0076] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0077] Fourthly, according to an embodiment of this application, a computer-readable storage medium is provided, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to implement the method of the embodiment of this application.

[0078] This application improves the registration accuracy of heterogeneous 3D point clouds. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0080] Figure 1 A flowchart illustrating the registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment;

[0081] Figure 2 A functional structure diagram of the registration device for heterogeneous point cloud fusion under road-vehicle cooperation provided in the embodiments of this application;

[0082] Figure 3 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0084] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0085] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0086] Perception is a crucial element in autonomous driving. By perceiving the surrounding environment, vehicles can make better decisions and plan routes. LiDAR sensors offer advantages such as long-range perception, high stability, and resistance to environmental factors (poor lighting conditions). Therefore, they are frequently used in autonomous vehicles. However, relying solely on the vehicle's own LiDAR sensors is insufficient for achieving accurate, all-around perception.

[0087] Existing vehicle-to-infrastructure (V2I) solutions address the limited radar perception of vehicles by designing complex V2I matching methods to homogeneous point clouds acquired by similar sensors on either the vehicle or the roadside. For example, multi-view image and map-based interactive perception fusion methods map sensor observation data to a semantic description of the vehicle environment, matching homogeneous point cloud data acquired by the original radar sensors of multiple vehicles, and combining homogeneous point cloud data acquired by multiple radar sensors at different spatial locations. Homogeneous point cloud data refers to point cloud data acquired by radar sensors of the same type. Most existing solutions align homogeneous point cloud data from different viewpoints. Currently, there is no solution for registering heterogeneous point clouds.

[0088] To address this, this application provides a heterogeneous 3D point cloud registration method for vehicle-road cooperation. This method integrates heterogeneous point clouds acquired by radar sensors at both ends of the vehicle and the road, and can directly match two frames of heterogeneous point clouds at both ends of the vehicle and the road to obtain a pose transformation matrix, thereby integrating the perception information at the road end and solving the problem of limited radar at the single vehicle end.

[0089] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0090] like Figure 1 As shown in the figure, this application provides a registration method for heterogeneous point cloud fusion under road-vehicle cooperation, including the following steps:

[0091] Step 101: Acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data of the same target scene;

[0092] Among them, the vehicle-side 3D point cloud data is collected by the vehicle-side mechanical radar sensor, while the road-side 3D point cloud data is collected by the road-side solid-state radar sensor, which is set near the streetlights or traffic lights at the intersection.

[0093] Step 102: Use the Polarseg semantic segmenter to segment the vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data; use the Polarseg semantic segmenter to segment the road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data.

[0094] The Polarseg semantic segmenter is a semantic segmenter that represents the input image using polar coordinates and combines it with a plasma evolution algorithm for 3D spectral matching. The plasma evolution algorithm obtains a globally optimized model during spectral fitting, and post-processing steps enable more accurate segmentation results. The specific steps of Polarseg semantic segmentation include: representing the input image using polar coordinates; performing 3D spectral matching using the plasma evolution algorithm; and constructing connected regions using post-processing steps to form the final semantic segmentation result.

[0095] Step 103: Use the radius density clustering method to process the vehicle-side ground point cloud data and the road-side ground point cloud data respectively to obtain vehicle-side ground point cloud data and road-side ground point cloud data with morphological noise removed.

[0096] Due to the differences in the working principles of mechanical and solid-state radars, different radar shapes are generated. The Radial-Density clustering method is used to remove the circular noise generated by mechanical radar and the dense noise points near the solid-state radar in the ground point cloud at the vehicle end. The Radial-Density clustering method categorizes points in space according to their different distributions using clustering techniques. The specific steps are as follows:

[0097] Step A1: First, select a center point and construct an initial cluster based on this point;

[0098] Step A2: Use Euclidean distance to assign the other points among the M points to this cluster;

[0099] Step A3: If the cluster radius exceeds a set threshold or the internal distance exceeds a set threshold, split the cluster;

[0100] Step A4: Repeat steps A1, A2 and A3 above until the clustering is stable.

[0101] Step 104: Process the vehicle-side non-ground point cloud data and the road-side non-ground point cloud data to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed.

[0102] In this embodiment, the step includes:

[0103] The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data of the vehicle to obtain the target prediction results.

[0104] Based on the vehicle-side target prediction results, a statistical outlier removal method is used to remove outliers from the vehicle-side non-ground point cloud data.

[0105] The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data at the roadside to obtain the target prediction results at the roadside.

[0106] Based on the roadside target prediction results, the statistical outlier removal method is used to remove outliers in the non-ground point cloud data of the roadside.

[0107] For each point in the vehicle-side non-ground point cloud data after removing outliers, several points are added in its neighborhood to make the density of each target point cloud close to the density of the corresponding target point cloud in the road-side non-ground point cloud data; thereby reducing the density difference between vehicle-side and road-side point cloud instance information.

[0108] When the target is a car, the step further includes:

[0109] Connect the outer contour points of the vehicle in the non-ground point cloud data at the roadside to form a line, which corresponds to the outer contour line of the vehicle in the non-ground point cloud data at the roadside.

[0110] Step 105: Fuse the vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed to obtain fused vehicle-side point cloud data; fuse the road-side ground point cloud data with morphological noise removed and the road-side non-ground point cloud data with outliers removed to obtain fused road-side point cloud data.

[0111] Step 106: Using the fused road end point cloud data as the source end and the fused vehicle end point cloud data as the target end, the global point cloud matching method is used to match the source end and the target end to obtain the pose transformation matrix from the source end to the target end.

[0112] In this embodiment, the step includes:

[0113] The fused vehicle endpoint cloud data is denoted as point set P = {p}, where p represents a point in the fused vehicle endpoint cloud data; the fused road endpoint cloud data is denoted as point set Q = {q}, where q represents a point in the fused road endpoint cloud data.

[0114] Calculate the normal vectors of point set P and point set Q;

[0115] Calculate the feature F(P) of point set P based on the normal vector of point set P: F(P) = {F(p): p∈P}; Calculate the feature F(Q) of point set Q based on the normal vector of point set Q: F(Q) = {F(q): q∈Q};

[0116] For each point p in point set P, find the point q in point set Q that has the smallest distance from point p. p For each point q in point set Q, find the point p in point set P that has the smallest distance to point q. q Thus, we obtain the set of point pairs K1 = {(p,q} p ),(q,p q The distance between two points is the difference in their features.

[0117] Perform a mutuality test on each pair of points in the pair set K1, and add the points that pass the mutuality test to the pair set K2;

[0118] Perform a multivariability test on each pair of points in the pair set K2, and add the points that pass the multivariability test to the pair set K3;

[0119] Establish the first objective function E1:

[0120]

[0121] Where, vector Let P be the three-dimensional coordinates of point P; let T be the vector. q for:

[0122]

[0123] (x q ,y q ,z q Let q be the three-dimensional coordinates of point q, and T be the 4×4 pose transformation matrix; l p,q This represents the distance between points p and q in the point pair (p,q) of the set of point pairs K3;

[0124] The initial pose transformation matrix T0 is calculated using the morphological noise-removed vehicle-side ground point cloud data and the morphological noise-removed road-side ground point cloud data.

[0125] Based on the initial pose transformation matrix T0, Gauss-Newton iterations are used. When the first objective function is less than a preset threshold or the number of iterations is reached, the final 4×4 pose transformation matrix T is obtained.

[0126]

[0127] Wherein, coefficient b 14 b 24 and b 34 This represents the translation value in three dimensions, with coefficient b. 12 The coefficient b represents the angle of rotation. 13 The pitch angle is represented by the coefficient b. 23 Indicates the angle of yaw.

[0128] Specifically, for each pair of points in the point pair set K1, a mutuality test is performed, and points that pass the mutuality test are added to the point pair set K2; including:

[0129] For any point (p, q) in the set of point pairs K1, keep it in the set of point pairs K2 if and only if F(q) is the nearest neighbor of F(p) in F(Q) and F(p) is the nearest neighbor of F(q) in F(P).

[0130] Specifically, a multivariability test is performed on each point in the point pair set K2, and points that pass the multivariability test are added to the point pair set K3; including:

[0131] For any point (p1, q1) in the set of point pairs K2, if for any two points in the set of point pairs K2 other than (p1, q1): (p, q2) and (p3, q3), the following holds true:

[0132]

[0133] Where τ = 0.9; i = 1, 2, 3; j = 1, 2, 3;

[0134] Then, if (p1,q1) passes the multivariability test, (p1,q1) is added to the point pair set K3.

[0135] Specifically, the initial pose transformation matrix T0 is calculated using morphologically noise-removed vehicle-side ground point cloud data and morphologically noise-removed road-side ground point cloud data; including:

[0136] Using the roadside ground point cloud data with morphological noise removed as the source end and the vehicle-side ground point cloud data with morphological noise removed as the target end, the source end and the target end are matched using a global point cloud matching method to obtain the three-degree-of-freedom pose transformation matrix F from the source end to the target end.

[0137] The ground plane model parameters of the vehicle-end ground are calculated based on the ground point cloud data with morphological noise removed. Based on the ground plane model parameters of the vehicle-end ground, any vehicle-end ground point is obtained, and its three-dimensional coordinates are (x1, y1, z1).

[0138] The ground plane model parameters of the roadside ground are calculated based on the roadside ground point cloud data after removing morphological noise; any roadside ground point is obtained based on the ground plane model parameters of the roadside ground, and its three-dimensional coordinates are (x2, y2, z2).

[0139] Calculate the rotation matrix R between the vehicle-end ground point and the road-end ground point:

[0140]

[0141] Calculate the initial value T0 of the pose transformation matrix:

[0142]

[0143] In this application, converting roadside point cloud data to vehicle-side data requires six degrees of freedom (x, y, z, r, p, y), where (x, y, z) are the three-dimensional coordinates representing the difference between the origin of the roadside coordinate system and the origin of the vehicle-side coordinate system; r, p, and y are the rotation angle, yaw angle, and pitch angle between the two coordinate systems, respectively. Since the vector m on the vehicle-side ground point cloud... vehic Vector m on the ground point cloud at the road end infrastructure By making them coplanar, the ground point clouds at both ends of the road can be overlapped. This allows 3 of the 6 degrees of freedom (x, y, z, r, p, y) to be fixed, thus requiring only 3 degrees of freedom to be chosen: x, y, and r.

[0144] Therefore, using morphologically noise-removed roadside ground point cloud data as the source and morphologically noise-removed vehicleside ground point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a three-degree-of-freedom pose transformation matrix from the source to the target; including:

[0145] The noise-free vehicle-side ground point cloud data is denoted as point set M = {m}, where m represents a point in the road-side ground point cloud data; the noise-free road-side ground point cloud data is denoted as point set N = {n}, where n represents a point in the road-side ground point cloud data.

[0146] Calculate the normal vectors of point set M and point set N;

[0147] Calculate the feature F(M) of point set M based on the normal vectors of point set M: F(M) = {F(m): m ∈ M}; Calculate the feature F(N) of point set N based on the normal vectors of point set N: F(N) = {F(n): n ∈ N};

[0148] For each point m in point set M, find the point n with the smallest distance from point m in point set N. m For each point n in the point set N, find the point m in the point set N that has the smallest distance from point n. n Thus, we obtain the set of point pairs O1 = {(m,n)} m ),(n,m n The distance between two points is the difference in their features.

[0149] Perform a mutuality test on each pair of points in the point pair set O1, and put the points that pass the mutuality test into the point pair set O2; perform a multivariability test on each pair of points in the point pair set O2, and put the points that pass the multivariability test into the point pair set O3.

[0150] Establish the second objective function E2:

[0151]

[0152] Where, vector Let m be the three-dimensional coordinates of point m; let T be the vector. n for:

[0153]

[0154] (x n ,y n ,1) are the three-dimensional coordinates of point n; l m,n This represents the distance between points m and n in the pair (m,n) of the set of point pairs O3.

[0155] Using Gauss-Newton iterations, when the second objective function is less than a preset threshold or the number of iterations is reached, the final three-degree-of-freedom pose transformation matrix F is obtained:

[0156]

[0157] Wherein, coefficient a 13 and a 23 Represents the two-dimensional translation value; coefficient a 12 Indicates the angle of rotation.

[0158] This includes calculating the ground plane model parameters of the vehicle-end ground based on the morphological noise-removed ground point cloud data; including:

[0159] Step S1: Randomly select three points from the vehicle-end ground point cloud data after removing morphological noise, and calculate the ground plane model parameters A, B, C and D according to the plane equation Ax+By+Cz+D=0;

[0160] Step S2: Calculate the distance from the remaining points in the vehicle-mounted ground point cloud data to the plane, compare each distance with the set threshold, mark points less than the threshold as inner points, and mark the rest as outer points;

[0161] Step S3: Count the number of interior points; Proceed to step S1; Continue until the number of iterations exceeds the threshold, then proceed to step S4;

[0162] Step S4: Obtain the ground plane model parameters corresponding to the maximum number of interior points.

[0163] Based on the above embodiments, this application provides a registration device for heterogeneous point cloud fusion under road-vehicle cooperation, see reference. Figure 2 As shown, the registration device 200 for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment includes at least:

[0164] The acquisition unit 201 is used to acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data of the same target scene;

[0165] Semantic segmentation unit 202 is used to segment vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data using the Polarseg semantic segmenter; and to segment road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data using the Polarseg semantic segmenter.

[0166] The first processing unit 203 is used to process the vehicle-end ground point cloud data and the road-end ground point cloud data respectively using the radius density clustering method to obtain vehicle-end ground point cloud data and road-end ground point cloud data with morphological noise removed.

[0167] The second processing unit 204 is used to process the vehicle-side non-ground point cloud data and the road-side non-ground point cloud data to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed.

[0168] The fusion unit 205 is used to fuse the vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed to obtain fused vehicle-side point cloud data; and to fuse the road-side ground point cloud data with morphological noise removed and the road-side non-ground point cloud data with outliers removed to obtain fused road-side point cloud data.

[0169] The computing unit 206 is used to match the source end and the target end using the fused road end point cloud data as the source end and the fused vehicle end point cloud data as the target end using the global point cloud matching method to obtain the pose transformation matrix from the source end to the target end.

[0170] It should be noted that the principle of the registration device 200 for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the registration device 200 for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0171] Based on the above embodiments, this application also provides an electronic device, see below. Figure 3 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored on the memory 302 and capable of running on the processor 301. When the processor 301 executes the computer program, it implements the registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment.

[0172] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0173] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0174] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0175] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 3 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0176] It should be noted that, Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0177] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application. Specifically, the executable program can be built into or installed in the electronic device 300, so that the electronic device 300 can implement the registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application by executing the built-in or installed executable program.

[0178] The registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the registration method for heterogeneous point cloud fusion under road-vehicle cooperation provided in this application embodiment.

[0179] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0180] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0181] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0182] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A registration method for heterogeneous point cloud fusion under road-vehicle cooperative operation, characterized in that, include: Acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data for the same target scene; The Polarseg semantic segmenter is used to segment vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data; the Polarseg semantic segmenter is used to segment road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data. The radius density clustering method was used to process the vehicle-side ground point cloud data and the road-side ground point cloud data respectively to obtain vehicle-side ground point cloud data and road-side ground point cloud data with morphological noise removed. The vehicle-side non-ground point cloud data and the road-side non-ground point cloud data are processed to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed. The vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed are fused to obtain the fused vehicle-side point cloud data. The roadside ground point cloud data with morphological noise removed and the roadside non-ground point cloud data with outliers removed are fused to obtain the fused roadside point cloud data. Using the fused roadside point cloud data as the source and the fused vehicle point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a pose transformation matrix from the source to the target; including: The merged vehicle endpoint cloud data is denoted as a point set. ,in, The points represent the vehicle endpoint cloud data after fusion; the points represent the road endpoint cloud data after fusion. ,in, Points representing the merged road end cloud data; Calculate the normal vectors of point set P and point set Q; Calculate the characteristics of point set P based on its normal vector. ： According to the point set Normal vector calculation point set Features ： ; For point sets Each point in From point set Get and point The point with the smallest distance For point sets Each point in From point set Get and point The point with the smallest distance This yields a set of point pairs. The distance between two points is the difference in their features. Point-to-set Each pair of points undergoes a mutuality test, and points that pass the mutuality test are added to the pair set. middle; Point-to-set Perform a multivariability test on each pair of points, and add the points that pass the multivariability test to the pair set. middle; Establish the first objective function : Where, vector , For point Three-dimensional coordinates; vector for: For point The three-dimensional coordinates for The pose transformation matrix; Represents a set of point pairs Midpoint Pair midpoint and points The distance; The initial pose transformation matrix is ​​calculated using morphologically noise-removed vehicle-side ground point cloud data and morphologically noise-removed road-side ground point cloud data. ; Based on the initial pose transformation matrix The algorithm uses Gauss-Newton iterations. When the first objective function is less than a preset threshold or the number of iterations is reached, the final result is obtained. pose transformation matrix : Among them, coefficient Represents the translation values ​​in three dimensions, coefficients Indicates the angle of rotation, coefficient Indicates the pitch angle, coefficient Indicates the angle of yaw.

2. The method according to claim 1, characterized in that, The vehicle-side non-ground point cloud data and road-side non-ground point cloud data are processed to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed; including: The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data of the vehicle to obtain the target prediction results. Based on the vehicle-side target prediction results, a statistical outlier removal method is used to remove outliers from the vehicle-side non-ground point cloud data. The pre-trained PointPillars 3D target detector is used to process the non-ground point cloud data at the roadside to obtain the target prediction results at the roadside. Based on the roadside target prediction results, the statistical outlier removal method is used to remove outliers in the non-ground point cloud data of the roadside. For each point in the vehicle-side non-ground point cloud data after removing outliers, add several points in its neighborhood to make the density of each target point cloud close to the density of the corresponding target point cloud in the road-side non-ground point cloud data.

3. The method according to claim 2, characterized in that, When the target is a car, the method further includes: Connect the outer contour points of the vehicle in the non-ground point cloud data at the roadside to form a line, which corresponds to the outer contour line of the vehicle in the non-ground point cloud data at the roadside.

4. The method according to claim 1, characterized in that, Point-to-set Perform a multivariability test on each point, and add the points that pass the multivariability test to a set of point pairs. In; including: For a set of point pairs any point ( , If for a set of point pairs Any two that are different from ( , ) points: ( , ), ( , All of them satisfy: in, ; ; ; Then determine ( , Through a diversity test, ( , Add to a set of point pairs middle.

5. The method according to claim 1, characterized in that, The initial pose transformation matrix is ​​calculated using morphologically noise-removed vehicle-side ground point cloud data and morphologically noise-removed road-side ground point cloud data. ;include: Using morphologically noise-removed roadside ground point cloud data as the source and morphologically noise-removed vehicle-side ground point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a three-degree-of-freedom pose transformation matrix from the source to the target. ; Based on the morphologically noise-removed point cloud data of the vehicle-end ground, the ground plane model parameters of the vehicle-end ground are calculated. Then, any point on the vehicle-end ground is obtained based on these parameters, and its three-dimensional coordinates are: ; The ground plane model parameters of the roadside surface are calculated based on the roadside ground point cloud data after removing morphological noise; the three-dimensional coordinates of any roadside ground point are obtained based on the ground plane model parameters of the roadside surface. ; Calculate the rotation matrix between the vehicle-end ground point and the road-end ground point. : Calculate the initial values ​​of the pose transformation matrix. : 。 6. The method according to claim 5, characterized in that, Using morphologically noise-removed roadside ground point cloud data as the source and morphologically noise-removed vehicle-side ground point cloud data as the target, a global point cloud matching method is used to match the source and target, resulting in a three-degree-of-freedom pose transformation matrix from the source to the target; including: The noise-removed ground point cloud data of the vehicle end is denoted as a point set. ,in, Points represent the roadside ground point cloud data; the roadside ground point cloud data after removing morphological noise is denoted as a point set. ,in, Points representing roadside ground point cloud data; Compute point set normal vector and point set The normal vector; Based on point set Normal vector calculation point set Features ： According to the point set Normal vector calculation point set Features ： ; For point sets Each point in From point set Get and point The point with the smallest distance For point sets Each point in From point set Get and point The point with the smallest distance This yields the set of point pairs. The distance between two points is the difference in their features. Point-to-set Each pair of points undergoes a mutuality test, and points that pass the mutuality test are added to the pair set. In the middle; pairs of points and sets Perform a multivariability test on each pair of points, and add the points that pass the multivariability test to the pair set. middle; Establish a second objective function : Where, vector , For point Three-dimensional coordinates; vector for: For point 3D coordinates; Represents a set of point pairs Midpoint Pair midpoint and points The distance; Using Gauss-Newton iterations, the final three-degree-of-freedom pose transformation matrix is ​​obtained when the second objective function is less than a preset threshold or the number of iterations is reached. : Among them, coefficient Represents the two-dimensional translation value; coefficient Indicates the angle of rotation.

7. The method according to claim 5, characterized in that, The ground plane model parameters of the vehicle-end surface are calculated based on the morphological noise-removed point cloud data; including: Step S1: Randomly select three points from the vehicle-end ground point cloud data after removing morphological noise, and calculate the ground plane model parameters A, B, C and D according to the plane equation Ax+By+Cz+D=0; Step S2: Calculate the distance from the remaining points in the vehicle-mounted ground point cloud data to the plane, compare each distance with the set threshold, mark points less than the threshold as inner points, and mark the rest as outer points; Step S3: Count the number of interior points; Proceed to step S1; Continue until the number of iterations exceeds the threshold, then proceed to step S4; Step S4: Obtain the ground plane model parameters corresponding to the maximum number of interior points.

8. A registration device for heterogeneous point cloud fusion under road-vehicle cooperation, characterized in that, include: The acquisition unit is used to acquire heterogeneous vehicle-side 3D point cloud data and road-side 3D point cloud data of the same target scene; The semantic segmentation unit is used to segment vehicle-side 3D point cloud data into vehicle-side ground point cloud data and vehicle-side non-ground point cloud data using the Polarseg semantic segmenter; and to segment road-side 3D point cloud data into road-side ground point cloud data and road-side non-ground point cloud data using the Polarseg semantic segmenter. The first processing unit is used to process the vehicle-end ground point cloud data and the road-end ground point cloud data respectively using the radius density clustering method to obtain vehicle-end ground point cloud data and road-end ground point cloud data with morphological noise removed. The second processing unit is used to process the vehicle-side non-ground point cloud data and the road-side non-ground point cloud data to obtain vehicle-side non-ground point cloud data with outliers removed and road-side non-ground point cloud data with outliers removed. The fusion unit is used to fuse the vehicle-side ground point cloud data with morphological noise removed and the vehicle-side non-ground point cloud data with outliers removed to obtain the fused vehicle-side point cloud data. The roadside ground point cloud data with morphological noise removed and the roadside non-ground point cloud data with outliers removed are fused to obtain the fused roadside point cloud data. The computational unit, using fused roadside point cloud data as the source and fused vehicle point cloud data as the target, employs a global point cloud matching method to match the source and target ends, obtaining the pose transformation matrix from the source to the target end. Specifically, it is used for: The merged vehicle endpoint cloud data is denoted as a point set. ,in, The points represent the vehicle endpoint cloud data after fusion; the points represent the road endpoint cloud data after fusion. ,in, Points representing the merged road end cloud data; Calculate the normal vectors of point set P and point set Q; Calculate the characteristics of point set P based on its normal vector. ： According to the point set Normal vector calculation point set Features ： ; For point sets Each point in From point set Get and point The point with the smallest distance For point sets Each point in From point set Get and point The point with the smallest distance This yields a set of point pairs. The distance between two points is the difference in their features. Point-to-set Each pair of points undergoes a mutuality test, and points that pass the mutuality test are added to the pair set. middle; Point-to-set Perform a multivariability test on each pair of points, and add the points that pass the multivariability test to the pair set. middle; Establish the first objective function : Where, vector , For point Three-dimensional coordinates; vector for: For point The three-dimensional coordinates for The pose transformation matrix; Represents a set of point pairs Midpoint Pair midpoint and points The distance; The initial pose transformation matrix is ​​calculated using morphologically noise-removed vehicle-side ground point cloud data and morphologically noise-removed road-side ground point cloud data. ; Based on the initial pose transformation matrix The algorithm uses Gauss-Newton iterations. When the first objective function is less than a preset threshold or the number of iterations is reached, the final result is obtained. pose transformation matrix : Among them, coefficient Represents the translation values ​​in three dimensions, coefficients Indicates the angle of rotation, coefficient Indicates the pitch angle, coefficient Indicates the angle of yaw.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-7.

Images