A method and system for constructing a vector map by fusing multi-source data

Abstract

The present invention relates to a method and system for constructing a vector map by fusing multi-source data, including: obtaining camera information, point cloud information, and vehicle status information during vehicle driving; extracting lane line 2D semantic information and traffic sign 2D semantic information from the camera information; simultaneously extracting curb 3D vector map elements and pole 3D vector map elements from the point cloud information; fusing the lane line 2D semantic information, traffic sign 2D semantic information with the point cloud information and the external parameter relationship and converting them into lane line 3D vector map elements and traffic sign 3D vector map elements, and forming a single-frame 3D vector map with the curb 3D vector map elements and pole 3D vector map elements; S4. Fusing the vehicle status information and the single-frame 3D vector map to obtain a multi-frame fusion vector of the curb and lane line and a multi-frame fusion vector of the traffic sign and pole, that is, obtaining the vector map. The present invention can quickly construct a high-precision vector map.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and particularly relates to a method and system for constructing a vector map by fusing multi-source data. Background Art

[0002] An autonomous driving system mainly consists of two major modules: perception and decision-making. The perception system is responsible for tasks such as target detection and tracking, map construction, and vehicle self-positioning. Among them, a high-precision map provides rich prior information for the vehicle, such as road topology, traffic signs, etc., which can greatly improve the effect of environmental perception. It not only provides a basis for trajectory planning and decision-making control, but also provides prior knowledge for positioning, target detection, and tracking tasks. The commonly used map types in autonomous driving systems include point cloud maps, raster maps, and vector maps. Point cloud maps record the three-dimensional coordinate points on the surfaces of objects in the environment, providing rich geometric information, especially suitable for precise modeling of complex environments and obstacle detection. However, point cloud maps have a large amount of data, requiring high storage and processing capabilities, and have limited support for path planning and decision-making when lacking semantic information. Raster maps divide the environment into regular grids and assign occupancy states to each grid, simplifying the environmental representation. Its advantage lies in that the simplified data structure is conducive to rapid path planning and obstacle avoidance, but the resolution limits the accuracy of representing detailed features, and the accuracy is affected by the grid size. Vector maps represent roads and environmental features with mathematical curves, providing precise road geometries and topologies, which are crucial for navigation and positioning. Vector map data volume is relatively small, facilitating update and maintenance. Therefore, how to quickly construct a high-precision vector map is regarded as an important task in the perception module of autonomous driving. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method and system for constructing a vector map by fusing multi-source data, which can quickly construct a high-precision vector map.

[0004] In a first aspect, a method for constructing a vector map by fusing multi-source data according to the present invention includes the following steps:

[0005] S1. Obtain camera information, point cloud information, and vehicle state information during the vehicle driving process;

[0006] S2. Extract lane line 2D semantic information and traffic sign 2D semantic information from the camera information; at the same time, extract road edge 3D vector map elements and pole 3D vector map elements from the point cloud information;

[0007] S3. Integrate the lane line 2D semantic information, traffic sign 2D semantic information with the point cloud information and the extrinsic parameter relationship, and convert them into lane line 3D vector map elements and traffic sign 3D vector map elements, and form a single-frame 3D vector map with the curb 3D vector map elements and the pole 3D vector map elements;

[0008] S4. Integrate the vehicle state information and the single-frame 3D vector map to obtain the multi-frame fusion vector of the curb and lane lines and the multi-frame fusion vector of the traffic signs and poles, that is, obtain the vector map.

[0009] Optionally, in S2, it specifically includes:

[0010] S21. Use the YOLOv8 network model to extract the traffic sign 2D semantic information from the camera information, and use the CLRNet network model to extract the lane line 2D semantic information from the camera information;

[0011] S22. Use an improved ground point cloud segmentation algorithm to obtain the curb 3D vector map elements from the point cloud information, and use a rule-based point cloud pole extraction algorithm to obtain the pole 3D vector map elements from the point cloud information according to geometric constraints. YOLOv8 performs excellently in object detection with its high efficiency and accuracy, and can quickly identify traffic signs. CLRNet focuses on the recognition of lane lines and improves the accuracy of lane line detection. The improved ground point cloud segmentation algorithm can more accurately segment the ground point cloud to extract curb information. The rule-based point cloud pole extraction algorithm improves the accuracy of pole extraction through geometric constraints.

[0012] Optionally, in S3, the lane line 3D vector map elements and the curb 3D vector map elements form a combination of curb and lane line 3D vector map elements; the traffic sign 3D vector map elements and the pole 3D vector map elements form a combination of traffic sign and pole 3D vector map elements;

[0013] The combination of curb and lane line 3D vector map elements and the combination of traffic sign and pole 3D vector map elements form the single-frame 3D vector map. This combination method can comprehensively describe the key elements in the road environment and provide a basis for subsequent multi-frame fusion and map construction.

[0014] Optionally, S3 includes:

[0015] S31. Coarse segmentation of point cloud based on extrinsic parameter projection, removal of noise points based on Euclidean clustering, and recovery of traffic sign corner points under geometric constraints to obtain the 3D vector map elements of the traffic sign; this series of steps can accurately extract traffic sign information from point cloud data and recover its corner point coordinates, improving the accuracy of the 3D vector map of traffic signs.

[0016] S32. Robust segmentation of ground point cloud based on ground slope change and vectorization of lane lines based on ground grid constraints to obtain the 3D vector map elements of the lane lines. This series of steps can accurately extract lane line information from point cloud data, improving the accuracy of the 3D vector map of lane lines.

[0017] Optionally, S31 specifically includes:

[0018] S311. Coarse segmentation of point cloud based on extrinsic parameter projection:

[0019] Combining the extrinsic parameter relationship of the sensor, a surjective relationship is established between the camera coordinate system and the lidar coordinate system, so that the 2D semantic information of the traffic sign is integrated into the point cloud information; the coarsely segmented point cloud clusters on the traffic sign are screened out from the mapped point cloud data; this step obtains the conversion relationship between the camera coordinate system and the lidar coordinate system, providing a basis for subsequent point cloud processing.

[0020] S312. Removal of noise points based on Euclidean clustering:

[0021] Perform Euclidean clustering on the coarsely segmented point cloud clusters, and divide the point cloud into multiple clustered point cloud clusters according to a preset distance threshold;

[0022] Perform principal component analysis on each clustered point cloud cluster, and calculate its eigenvalues and eigenvectors;

[0023] Comprehensively calculate the feature score according to the eigenvalues and the number of points in the clustered point cloud cluster, select the clustered point cloud cluster with the highest feature score as the final point cloud of the traffic sign, and remove the remaining clustered point cloud clusters as noise points; this step effectively removes noise points, improves the accuracy of the traffic sign point cloud, and improves the calculation efficiency.

[0024] S313. Recovery of traffic sign corner points under geometric constraints:

[0025] Using the eigenvector of the clustered point cloud cluster obtained by principal component analysis, determine the normal vector of the clustered point cloud cluster, and combine the centroid of the clustered point cloud cluster to determine the plane equation of the traffic sign in the lidar coordinate system;

[0026] Convert the corner points of the traffic sign bounding box in the camera coordinate system to the lidar coordinate system;

[0027] By solving the system of equations composed of the plane equation of the traffic sign in the radar coordinate system and the line equation connecting the camera optical center and the corner points of the traffic sign, the actual corner point coordinates of the traffic sign in the radar coordinate system are obtained. This step realizes the precise positioning of the traffic sign in the radar coordinate system.

[0028] Optionally, the S32 specifically includes:

[0029] S321. Ground point cloud segmentation robust to ground slope changes:

[0030] With the origin of the radar coordinate as the center, according to the maximum distance, minimum distance and fixed angle of the preset field of view range, the field of view range is divided into multiple fan-shaped ring regions and sliced into sub-fan-shaped rings with increasing width in sequence to form a point cloud container.

[0031] Judge each point cloud container, count the number of points in the point cloud container, and the Z-axis height features of all points. The Z-axis height features include the minimum value, maximum value in the Z-axis direction and the maximum Z-axis difference. Determine the ideal ground height, distance from the ideal ground threshold, obstacle determination threshold and point cloud number threshold according to the calibration information.

[0032] Set the category of the point cloud container. According to the relationship between the number of points in the point cloud container, Z-axis height features, ideal ground height, distance from the ideal ground threshold, obstacle determination threshold and point cloud number threshold, judge the point cloud container as the corresponding category through the preset point cloud container type judgment rule.

[0033] Perform dimensionality reduction processing on the ground points, fit a straight line reflecting the ground height change according to the dimensionality-reduced ground points, traverse each point in the point cloud, calculate the distance from each point to the straight line, and judge whether the point is a ground point or a non-ground point according to the distance. Finally, obtain the ground point cloud marked as ground points and the non-ground point cloud marked as non-ground points. This step realizes the effective segmentation of the ground point cloud and provides a basis for the subsequent lane line vectorization.

[0034] S322. Lane line vectorization based on ground grid constraints:

[0035] Use the ground point cloud for plane fitting to obtain the spatial plane expression.

[0036] Divide the grid according to the point cloud segmentation method in S321, determine the X-axis and Y-axis coordinates of the grid vertices, and substitute them into the spatial plane expression to obtain the Z-axis coordinates of the grid vertices.

[0037] For the common vertices of the grid, calculate the final Z value of the common vertex according to the number of ground points in the grids related to it and the determined Z value of the common vertex.

[0038] Using the extrinsic parameter relationship between the radar and the camera, project the grid onto the plane in the camera coordinate system, associate the detected key points of the lane line with the corresponding grid, obtain the positions of the key points of the lane line in the radar coordinate system, and finally obtain the 3D vector map elements of the lane line. This step realizes the precise positioning of the lane line in the radar coordinate system.

[0039] Optionally, S4 includes:

[0040] S41. Obtain the multi-frame fusion vector of traffic signs and poles:

[0041] Combined with the vehicle state information, convert the 3D vector map elements of traffic signs and poles to the world coordinate system;

[0042] Maintain a tracking queue for each individual object, and add the objects associated with the single-frame recovery to the queue;

[0043] When outputting the final vectorization result, eliminate the queues with the number of objects less than the preset number threshold in the tracking queue;

[0044] Use the confidence of each object in the tracking queue for weighted average to obtain the final position of the object; among them, the confidence is calculated based on the position information and shape information of the object in the world coordinate system;

[0045] When the confidence of the object is higher than the preset confidence threshold, it is considered that the match is valid, and the object with the highest confidence is selected for matching, and a new queue is created for the objects with invalid matches;

[0046] For the situation where the same object is vectorized multiple times and tracked, perform confidence-weighted average vectorization; multiply the confidence by the center coordinates of the object that has been tracked in the world coordinate system, and finally obtain the multi-frame fusion vector of traffic signs and poles; this step realizes the precise positioning and tracking of traffic signs and poles in the world coordinate system.

[0047] S42. Obtain the multi-frame fusion vector of road edges and lane lines:

[0048] Use the DBSCAN algorithm to cluster the key points belonging to the same linear element to obtain the clustered set;

[0049] Segment the points in the set at a fixed distance to obtain the segmented set;

[0050] Use the weighted least squares method to perform cubic polynomial curve fitting on all points in each subset to obtain the curve fitted by the subset;

[0051] Judge through the fitting error. When the fitting error is greater than the preset error threshold, re-divide the subset and re-fit the curve until the fitting errors of all subsets are less than the preset error threshold;

[0052] Obtain the current state element expression of the subset where each fitting error meets the requirements, determine the elevations of the lane lines and the road edges, and finally obtain the multi-frame fusion vector of the road edges and the lane lines. This step realizes the precise fitting and positioning of the road edges and the lane lines in the world coordinate system.

[0053] Optionally, the confidence level is calculated based on the position information and shape information of the object in the world coordinate system. Among them, the calculation formulas include the position confidence formula, the shape confidence formula, and the comprehensive confidence calculation formula;

[0054] The position confidence formula takes into account the difference in the center coordinates of the tracked object and the object to be tracked in the world coordinate system;

[0055] The shape confidence formula takes into account the difference in the shape features of the tracked object and the object to be tracked;

[0056] The comprehensive confidence level is obtained by weighted calculation of the position confidence level and the shape confidence level. Obtaining the comprehensive confidence level by weighted calculation of the position confidence level and the shape confidence level improves the accuracy and reliability of object matching. Confidence level calculation can reflect the stability and accuracy of the object in the world coordinate system.

[0057] Optionally, the cubic polynomial curve fitting process uses the weighted least squares method, takes into account the confidence level of each point, and obtains the coefficients of the fitting curve through iterative solution;

[0058] The current state element expression is determined according to the coefficients of the cubic polynomial obtained by fitting and the start point and end point of the subset. Using the weighted least squares method for cubic polynomial curve fitting and taking into account the confidence level of each point improves the accuracy of the fitting curve. The weighted least squares method can reduce the influence of points with larger errors on the fitting result and improve the reliability of the fitting curve.

[0059] In a second aspect, a vector map construction system for multi-source data fusion according to the present invention includes a memory and a controller. A computer-readable program is stored in the memory. When the computer-readable program is called by the controller, it can execute the steps of the vector map construction method for multi-source data fusion according to the present invention.

[0060] The unexpected beneficial effects of the present invention are as follows:

[0061] The vector map construction method and system for multi-source data fusion proposed by the present invention realize the efficient and precise construction of the vector map by fusing camera information, point cloud information, and vehicle state information, providing important technical support for fields such as autonomous driving. Its unexpected beneficial effects are mainly reflected in the following aspects:

[0062] 1. Precise restoration of the spatial positions of traffic signs and lane lines:

[0063] The present invention realizes the precise restoration of the spatial positions of these traffic elements by fusing the 2D semantic information and point cloud information of traffic signs and lane lines. It overcomes the problem of insufficient accuracy caused by relying solely on a single data source in traditional methods, improving the accuracy and reliability of vector maps. Additionally, the precise spatial position information provides the autonomous driving system with more reliable environmental perception capabilities, helping the vehicle make more accurate decisions and plans.

[0064] 2. Efficient detection of poles and curbs:

[0065] The present invention utilizes the rich spatial features in point cloud information to achieve the efficient detection of poles and curbs. By deeply analyzing the spatial distribution and geometric features of point cloud data, the present invention can accurately identify and extract these important traffic elements. Additionally, the efficient and accurate detection ability not only improves the integrity of vector maps but also provides the autonomous driving system with more abundant environmental information, helping to enhance the safety and driving stability of the vehicle.

[0066] 3. Application of multi-frame fusion vectorization strategy:

[0067] The present invention introduces a multi-frame fusion vectorization strategy, combined with vehicle state information, effectively overcoming problems such as visual differences in sensing information, sparse changes in point clouds, misdetection, and missed detection during vehicle driving. Through the fusion processing of multi-frame data, the present invention can generate multi-frame vectorization results with high consistency and concise expression. This globally consistent vector map not only improves the accuracy and reliability of the map but also provides the autonomous driving system with a more stable environmental model, helping to enhance the driving performance and user experience of the vehicle.

[0068] 4. Providing lightweight data support for downstream algorithms of autonomous driving:

[0069] The vector map constructed by the present invention provides more lightweight data support for downstream algorithms of autonomous driving. Compared with traditional raster maps or image data, vector maps have the advantages of simple data structure, small storage space, and high processing efficiency. The lightweight data support not only reduces the computational burden of the autonomous driving system but also improves the development, testing, and operation efficiency of the system. This makes it easier to deploy and apply autonomous driving technology in actual engineering, promoting the rapid development and popularization of autonomous driving technology.

[0070] In summary, the method for constructing a vector map by fusing multi-source data proposed by the present invention has significant beneficial effects, providing important technical support and guarantee for fields such as autonomous driving. Brief Description of the Drawings

[0071] Figure 1It is a flowchart of a method for constructing a vector map by fusing multi-source data described in an embodiment of the present application.

[0072] Figure 2 It is a flowchart of S31 in an embodiment of the present application.

[0073] Figure 3 It is a schematic diagram of the principle of rough segmentation of point cloud based on external parameter projection in S31 of an embodiment of the present application.

[0074] Figure 4 It is a flowchart of S32 in an embodiment of the present application.

[0075] Figure 5 It is a schematic diagram of the basis for judging the type of point cloud container in S32 of an embodiment of the present application.

[0076] Figure 6 It is a schematic diagram of the type of ground point container and the process of screening ground points in S32 of an embodiment of the present application.

[0077] Figure 7 It is a schematic diagram of the projection of key points of lane lines based on ground grids in S32 of an embodiment of the present application.

[0078] Figure 8 It is a flowchart of the multi-frame fusion vectorization of traffic signs and poles in S41 of an embodiment of the present application.

[0079] Figure 9 It is a flowchart of the multi-frame fusion vectorization of lane lines and road edges in S41 of an embodiment of the present application.

[0080] Figure 10 It is a block diagram of the principle of a system for constructing a vector map by fusing multi-source data described in an embodiment of the present application. Detailed implementation manners

[0081] The following further describes in detail the specific implementation manners of the present invention with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0082] In an embodiment of the present application, a method for constructing a vector map by fusing multi-source data can be executed by any computing device (such as a PC, a mobile terminal, or a server, etc.) with computing, processing, and storage functions.

[0083] As Figure 1 shown, in an embodiment of the present application, a method for constructing a vector map by fusing multi-source data includes the following steps:

[0084] S1. Obtain camera information, point cloud information, and vehicle status information during the vehicle's driving process.

[0085] S2. Extract the lane line 2D semantic information and traffic sign 2D semantic information from the camera information; meanwhile, extract the curb 3D vector map elements and pole 3D vector map elements from the point cloud information.

[0086] S3. Integrate the lane line 2D semantic information, traffic sign 2D semantic information with the point cloud information and the extrinsic parameter relationship, and convert them into lane line 3D vector map elements and traffic sign 3D vector map elements, which together with the curb 3D vector map elements and pole 3D vector map elements form a single-frame 3D vector map.

[0087] S4. Integrate the vehicle state information and the single-frame 3D vector map to obtain the multi-frame fusion vectors of the curb and lane lines and the multi-frame fusion vectors of the traffic signs and poles, that is, obtain the vector map.

[0088] In the embodiment of the present application, for S1. When obtaining the camera information, point cloud information and vehicle state information during vehicle driving, the specific implementation method is as follows:

[0089] Deploy cameras, radars and inertial measurement units (IMUs) on the experimental vehicle to collect the camera information, point cloud information and vehicle state information during vehicle driving respectively, and send them to the computing device for operation and storage. Regarding the types of vehicles, cameras, radars and IMUs, the embodiments of the present application do not make specific limitations.

[0090] In the embodiment of the present application, for S2. Extract the lane line 2D semantic information and traffic sign 2D semantic information from the camera information; meanwhile, extract the curb 3D vector map elements and pole 3D vector map elements from the point cloud information; specifically include:

[0091] S21. Use the YOLOv8 network model to extract the traffic sign 2D semantic information from the camera information. Use the CLRNet network model to extract the lane line 2D semantic information from the camera information.

[0092] S22. Use the improved ground point cloud segmentation algorithm to obtain the curb 3D vector map elements from the point cloud information, and use the rule-based point cloud pole extraction algorithm to obtain the elements of the pole 3D vector map from the point cloud information according to geometric constraints.

[0093] In the embodiment of the present application, in S3, the lane line 3D vector map elements and the curb 3D vector map elements form a combination of curb and lane line 3D vector map elements; the traffic sign 3D vector map elements and the pole 3D vector map elements form a combination of traffic sign and pole 3D vector map elements.

[0094] In a possible embodiment, S3 specifically includes the following steps:

[0095] S31. Coarse segmentation of point cloud based on extrinsic parameter projection, removal of noise points based on Euclidean clustering, and recovery of traffic sign corner points under geometric constraints to obtain 3D vector map elements of traffic signs;

[0096] S32. Robust segmentation of ground point cloud based on ground slope change and vectorization of lane lines based on ground grid constraint to obtain 3D vector map elements of lane lines.

[0097] As Figure 2 shown, in a possible embodiment, S31 specifically includes the following steps:

[0098] S311. Coarse segmentation of point cloud based on extrinsic parameter projection:

[0099] Combining the extrinsic parameter relationship of the sensor, a surjective relationship is established between the camera coordinate system and the lidar coordinate system, enabling the integration of 2D semantic information of traffic signs into the point cloud information, and screening out the coarsely segmented point cloud clusters on traffic signs. Among them, the principle of coarse segmentation of point cloud based on extrinsic parameter projection is as Figure 3 shown. In the figure, F1 is the quadrangular pyramid determined by the camera information in the three-dimensional space, F2 is the point cloud information on the traffic sign board, and F3 is the noise point cloud introduced due to parallax.

[0100] The coarse segmentation of point cloud based on extrinsic parameter projection is expressed as the following formula (1):

[0101] (1)

[0102] Formula (1) represents the projection process of the point cloud from the lidar coordinate system to the camera coordinate system.

[0103] In formula (1), represents the two-dimensional representation of the i th point in the point cloud in the camera coordinate system; represents the three-dimensional coordinates of the i th point in the point cloud in the lidar coordinate system; R , t represents the extrinsic parameter matrix, R represents the rotation matrix (the rotation of the camera coordinate system relative to the lidar coordinate system), t represents the translation vector (the internal geometric and optical characteristics of the camera (such as focal length, principal point, etc.). R and t constitute the extrinsic parameter matrix, which is used to describe the conversion relationship between the camera coordinate system and the lidar coordinate system; K represents the intrinsic parameter matrix.

[0104] is used to convert the point in the lidar coordinate system to the camera coordinate system through the extrinsic parameter matrix [R,t], and then project it onto the image plane through the intrinsic parameter matrix K to obtain the point in homogeneous coordinate form. Among them, Represents the abscissa (x - coordinate) of the projected point on the image plane. In the camera coordinate system, this value usually corresponds to the horizontal position of the pixel in the image. Represents the ordinate (y - coordinate) of the projected point on the image plane. In the camera coordinate system, this value usually corresponds to the vertical position of the pixel in the image. 1 is an additional coordinate in homogeneous coordinate representation, used to extend two - dimensional image coordinates to three - dimensional homogeneous coordinates. Homogeneous coordinates are very useful in computer vision and graphics because they can simplify the calculation of many geometric transformations (such as rotation, translation, scaling, etc.).

[0105] According to the bounding box of the traffic sign 2D semantic information in the camera coordinate system, determine whether each point in the point cloud belongs to the traffic sign. If it belongs to the traffic sign, it is marked as the corresponding traffic sign; if it does not belong to the traffic sign, it is marked as a background point. Traverse all the points in the point cloud, and complete the rough segmentation task of the point cloud based on the external parameter projection according to the marked labels.

[0106] S312, Noise point removal based on Euclidean clustering:

[0107] Since there is usually a perspective difference in the installation positions of the radar and the camera, there are noise points in the rough - segmented point cloud clusters, which affect the acquisition of the depth information of the traffic signs. To accurately recover the depth information of the traffic signs from the point cloud, the noise points need to be filtered.

[0108] Specifically, project the point cloud in three - dimensional space onto the X - Y plane, perform Euclidean clustering on the rough - segmented point cloud clusters. During the Euclidean clustering process, divide the point cloud into multiple clustered point cloud clusters according to a preset distance threshold.

[0109] Perform principal component analysis (PCA) on each clustered point cloud cluster to obtain the principal component representation of this clustered point cloud cluster. Here, one clustered point cloud cluster is taken as an example for introduction, and other clustered point cloud clusters are processed in the same way. The centroid of the clustered point cloud cluster is expressed as the following formula (2):

[0110] (2)

[0111] In formula (2), Represents the centroid of the clustered point cloud cluster; Represents the three - dimensional coordinates of the centroid of the clustered point cloud cluster; Represents the i' th point in the clustered point cloud cluster; Represents the number of points in the clustered point cloud cluster.

[0112] Center each point in the clustered point cloud cluster, and the centering is expressed as the following formula (3):

[0113] (3)

[0114] In formula (3), represents the i' th point in the centered clustered point cloud cluster. represents the three-dimensional coordinates of the i' th point in the centered clustered point cloud cluster.

[0115] Use the centered points to construct a covariance matrix, and the covariance matrix is as formula (4) below:

[0116] (4)

[0117] In formula (4), Cov represents the covariance matrix.

[0118] Perform singular value decomposition on the covariance matrix to obtain eigenvalues , and , and their corresponding eigenpost-vectors , and .

[0119] Perform feature scoring on each cluster of the clustered point cloud clusters (that is, comprehensively calculate the feature score based on the eigenvalues and the number of points in the clustered point cloud clusters), and select the clustered point cloud cluster with the highest feature score as the final point cloud of the traffic sign, and eliminate the remaining clustered point cloud clusters as noise points.

[0120] Among them, use the eigenvalues obtained by PCA and combine with the number of points in the clustered point cloud cluster to calculate the feature score, and the feature score is as formula (5) below:

[0121] (5)

[0122] In formula (5), represents the feature score of the PCA part; represents the feature score of the clustering part; N represents the number of points in the coarsely segmented point cloud cluster; represents the feature score; represents the adjustment coefficient, which is used to adjust the weights of the feature score of the PCA part and the feature score of the clustering part in the feature score.

[0123] S313, Traffic sign corner point recovery under geometric constraints:

[0124] Overcome the problem of incomplete reconstruction of the traffic sign shape information caused by the sparsity of the point cloud information through the traffic sign corner point recovery under geometric constraints, so as to obtain the complete shape and position of the traffic sign in three-dimensional space.

[0125] Specifically, the following introduces the clustered point cloud clusters determined as traffic signs obtained in S312:

[0126] The eigenvector of the clustered point cloud cluster obtained by principal component analysis , the normal vector of the clustered point cloud cluster can be determined , combined with the centroid of the clustered point cloud cluster , the plane equation of the traffic sign in the radar coordinate system can be determined, and this plane equation is as follows formula (6):

[0127] (6)

[0128] Where: A represents the component of the normal vector in the x axis direction, which reflects the inclination degree of the plane in the x axis direction. B represents the component of the normal vector in the y axis direction, which reflects the inclination degree of the plane in the y axis direction. C represents the component of the normal vector in the z axis direction, which reflects the inclination degree of the plane in the z axis direction. D represents the constant term of the plane equation.

[0129] Convert the corner points of the traffic sign bounding box in the camera coordinate system to the radar coordinate system, and the conversion formula is as follows formula (7):

[0130] (7)

[0131] In formula (7), represents the j th bounding box corner point of the traffic sign in the radar coordinate system. represents the three-dimensional coordinates of the j th bounding box corner point of the traffic sign in the radar coordinate system; represents the inverse matrix of the intrinsic parameter matrix; represents the transpose of the rotation matrix.

[0132] According to the camera optical center point coordinates in the radar coordinate system and the traffic sign bounding box corner point coordinates in the radar coordinate system, a straight line in three-dimensional space (i.e., the connection equation between the camera optical center and the traffic sign corner point) can be determined, and this connection equation between the camera optical center and the traffic sign corner point is as follows formula (8):

[0133] (8)

[0134] In formula (8), represents the camera optical center point in the radar coordinate system, Represents the three-dimensional coordinates of the camera optical center point in the radar coordinate system. , , , Are all coefficients of the straight-line equation in the radar coordinate system, and these coefficients are determined by fitting the straight line between the camera optical center point and the corner points of the traffic sign.

[0135] Substitute formula (8) into formula (6) to obtain formula (9):

[0136] (9)

[0137] Solve formula (9) to obtain the intersection parameters of the space straight line and the plane, and the intersection parameters are as follows in formula (10):

[0138] (10)

[0139] In formula (10), Are the intersection parameters.

[0140] Substitute formula (10) back into formula (8) to obtain the actual corner point coordinates of the traffic sign in the radar coordinate system, and the actual corner point coordinates are as follows in formula (11):

[0141] (11)

[0142] In formula (11), Represents the actual corner point coordinates.

[0143] Obtaining the actual corner point coordinates can recover the true spatial position of the traffic sign in the radar coordinate system under single-frame data.

[0144] S32. Based on the robust ground point cloud segmentation for ground slope changes and the lane line vectorization based on ground grid constraints, obtain the lane line 3D vector map elements. The process of lane line single-frame vectorization in this application example is as Figure 4 shown.

[0145] When specifically implemented, S32 can be implemented through the following S321 to S322:

[0146] S321. Through the robust ground point cloud segmentation for ground slope changes, segment the ground point cloud from the point cloud information. Specifically:

[0147] With the radar coordinate origin as the center, the maximum distance of its field of view is , the minimum distance is , and at a fixed angle , divide the field of view range into multiple fan-shaped ring regions, and then from to The field of view range is divided into sub-sector rings with increasing width in sequence to form a point cloud container, completing the point cloud segmentation task. The segmentation result is as Figure 6 shown.

[0148] Specifically, each point cloud container is judged, and the number of points in the point cloud container is counted n , and all points are Z The minimum value in the axis direction is , the maximum value is , the maximum difference in the Z axis is , the ideal ground height is determined according to the calibration information , the distance from the ideal ground threshold , the obstacle determination threshold , and the point cloud number threshold Figure 5 . It is set that the point cloud container has 4 categories, including: Unknown, Class A, Class B, and Class C. The basis for judging the type of the point cloud container is as , , , , , , and . According to the relationship between

[0149] (12)

[0150] According to the point cloud container type judgment formula, the point cloud container can be judged into 4 types, including: Unknown container, Class A container, Class B container, and Class C container. The Unknown container is judged that the number of point cloud points in the point cloud container is insufficient, so the category of the point cloud points in the Unknown container cannot be judged. The Class A container is judged as a ground container, the Class B container is judged as a container containing obstacles, and the Class C container is judged as a complex container. Therefore, the point cloud points in the Class A container can be directly marked as ground points, and the points in the Class B container need to be traversed. According to the following formula (13), the category of the point cloud points in the point cloud container is judged:

[0151] (13)

[0152] In formula (13), represents the z value of the i''-th point in the traversed point cloud container, represents the z value of the ideal ground. The processing of the Class C container depends on the surrounding sub-sector rings for processing. As Figure 6 shown, in the example of the present invention, for the convenience of description, Represents a Class C container, and the surrounding sub-sector rings are represented as , , and , and set the point cloud containers and After discrimination, the ground points in the point cloud containers and are .

[0153] Specifically, for the ground points perform dimensionality reduction, and the dimensionality reduction process is as follows in formula (14):

[0154] (14)

[0155] In formula (14), represents the i''-th ground point after dimensionality reduction. represents the first dimensional parameter of the i''-th ground point after dimensionality reduction; the ground points after dimensionality reduction can fit the corresponding straight line , where and respectively represent the coefficients of the straight line equation in the ground coordinate system and are obtained through fitting. This straight line reflects the ground height change situation in the direction from the inside (close to the radar coordinate origin) to the outside (far from the radar coordinate origin) near . Traverse each point in

[0156] (15)

[0157] If the calculation result is , then this point is marked as a ground point, otherwise it is marked as a non-ground point. Finally, the ground point cloud marked as ground points and the non-ground point cloud marked as non-ground points are obtained.

[0158] S322, Lane line vectorization based on ground grid constraints:

[0159] Use the ground point cloud to perform plane fitting to obtain the spatial plane expression, and restore the three-dimensional positions of the key points of the lane line to avoid the problem of significant reconstruction errors of the key points caused by the absolute ground assumption.

[0160] Specifically, divide the grid according to the point cloud segmentation method in S321. As Figure 6 shown, the X axis and Y axis coordinates of the grid vertices can be determined, and substituting them into the spatial plane expression can obtain the ZAxis coordinates, perform the same operation for all grids. For the processing of the common vertices of the grids, the following formula (16):

[0161] (16)

[0162] In formula (16), M represents the number of grids related to the common vertex of the grid, represents the th number of ground points in the grid, represents the value of the common vertex determined by the z th grid, represents the average value of the z values of the common vertices determined by the th grid.

[0163] Using the extrinsic parameter relationship between the radar and the camera, project the grid onto the plane in the camera coordinate system, and associate the detected key points of the lane line with the corresponding grid, as Figure 7 shown. The position of the key points of the lane line in the radar coordinate system can be obtained, and finally the 3D vector map element of the lane line is obtained.

[0164] S4. Integrate the vehicle state information and the single-frame 3D vector map to obtain the multi-frame fusion vector of the road edge and the lane line and the multi-frame fusion vector of the traffic sign and the pole, and finally obtain the vector map; specifically including:[[]]

[0165] S41. Obtain the schematic diagram of the process of multi-frame fusion vectorization of traffic signs and poles, as Figure 8 shown.

[0166] To ensure the continuity of the system output during vehicle driving and overcome the problem of poor positioning accuracy of the single-frame vectorization result. Combining the vehicle state information, the 3D vector map elements of traffic signs and poles are combined and transformed into the world coordinate system. To avoid fluctuations in the position in the world coordinate system caused by errors introduced by vehicle state information, sensors, and calibration, for each individual object, a tracking queue is maintained. The queue will add the objects associated with the single-frame recovery. When outputting the final vectorization result, the queues with the number of objects less than a certain threshold in the tracking queue will be excluded, and then the weighted average of the confidence levels of each object in the tracking queue is used to obtain the final position of the object.

[0167] Specifically, the confidence level is calculated based on the position information of the object in the world coordinate system, as the following formula (17):

[0168] (17)

[0169] In formula (17), S represents the similarity score between the object to be tracked and the tracked object, that is, the confidence level Indicates the position confidence Indicates the shape confidence Indicates the coefficient for adjusting the proportion of the shape confidence. The position confidence and the shape confidence are as shown in the following formulas (18) and (19):

[0170] (18)

[0171] (19)

[0172] In formula (18), Indicates the center coordinates of the tracked object in the world coordinate system Indicates the center coordinates of the object to be tracked in the world coordinate system Indicates the situation where multiple objects are placed one above the other in the unified position

[0173] In formula (19), Indicates the shape feature of the tracked object Indicates the shape feature of the object to be tracked. If the confidence is higher than the preset confidence threshold, it is considered that the match is valid, and the object with the highest confidence is selected for matching. For the objects with invalid matches, a new queue is created

[0174] For the same object that is vectorized and tracked multiple times, it is necessary to perform confidence weighted average vectorization on it, as shown in the following formula (20):

[0175] (20)

[0176] In formula (20), Indicates the confidence after weighted average V Indicates the number of objects in the queue Indicates the p th object's confidence. Finally, the confidence is multiplied by the center coordinates of the tracked object in the world coordinate system to finally obtain the multi-frame fusion vector of the traffic sign and the pole

[0177] S42. The process schematic diagram for obtaining the multi-frame fusion vector of the road edge and the lane line is as Figure 9 shown

[0178] To overcome the different degrees of interference in the acquisition of camera information and point cloud information during vehicle driving, and finally obtain a concise and complete vectorized expression of the lane line and the road edge

[0179] Specifically, use the DBSCAN algorithm to cluster the key points belonging to the same linear element to obtain the clustered set , where in Represent the coordinates of the key points for single-frame vectorization, represent the confidence of this point. For the set C of points, segment them according to a fixed distance d to obtain the segmented set , where the subset , k = 1, 2, …, W ; where represent the coordinates of the key points after single-frame vectorization segmentation, represent the confidence of this point. Use the weighted least squares method to perform a cubic polynomial curve fitting on all points in each subset to obtain the curve fitted for this subset , as shown in formula (21) below:

[0180] (21)

[0181] In formula (21), represent the coefficients of the cubic polynomial curve .

[0182] Iteratively solve formula (21) to obtain the curve fitted for the subset and , where represent the fitting error. Make a judgment based on the fitting error. When the fitting error is greater than the error threshold, the subset needs to be re-divided and the curve re-fitted until the fitting errors of all subsets are less than the error threshold.

[0183] Furthermore, obtain the current state element expression of each subset whose fitting error meets the requirements, as shown in formula (22) below:

[0184] (22)

[0185] In formula (22), represent the coefficients of the cubic polynomial obtained by fitting, and respectively represent the start point and end point of this subset, that is , , and determine the elevations of the lane line and the road edge, and finally obtain the multi-frame fusion vector of the road edge and the lane line. Among them, represent the start point coordinates of the multi-frame fusion vector of the road edge and the lane line, represent the coordinates of the end point of the multi-frame fusion vector of the road edge and the lane line.

[0186] Finally, complete the construction of the vector map.

[0187] In an embodiment of the present application, a vector map construction system for multi-source data fusion includes a memory and a controller. A computer-readable program is stored in the memory. When the computer-readable program is called by the controller, it can execute the steps of the vector map construction method for multi-source data fusion in the embodiment of the present application.

[0188] As Figure 10 shown below, the computer-readable program can be divided into an input module, an image 2D target detection module, a single-frame 3D vector map construction module, a single-frame 3D vector map module, a point cloud 3D target detection module, a multi-frame fusion vectorization module, and an output module according to functional modules. Among them, the input module is respectively connected to the image 2D target detection module, the single-frame 3D vector map construction module, the point cloud 3D target detection module, and the multi-frame fusion vectorization module. The image 2D target detection module is connected to the single-frame 3D vector map construction module. The single-frame 3D vector map construction module and the point cloud 3D target detection module are respectively connected to the single-frame 3D vector map module. The single-frame 3D vector map module is connected to the multi-frame fusion vectorization module. The multi-frame fusion vectorization module is connected to the output module.

[0189] The input module is used to obtain camera information, point cloud information, and vehicle state information.

[0190] The image 2D target detection module is used to extract lane line 2D semantic information and traffic sign 2D semantic information from the camera information.

[0191] The point cloud 3D target detection module includes an improved ground point cloud segmentation sub-module and a rule-based point cloud rod-shaped object extraction sub-module, and is used to extract curb 3D vector map elements and rod-shaped object 3D vector map elements from the point cloud information.

[0192] The single-frame 3D vector map module fuses the lane line 2D semantic information, the traffic sign 2D semantic information with the point cloud information and the external parameter relationship, and converts them into lane line 3D vector map elements and traffic sign 3D vector map elements.

[0193] The single-frame 3D vector map module includes a traffic sign and rod-shaped object 3D vector map element combination sub-module and a curb and lane line 3D vector map element combination sub-module, and is used to form a single-frame 3D vector map with the lane line 3D vector map elements, the traffic sign 3D vector map elements, the curb 3D vector map elements, and the rod-shaped object 3D vector map elements.

[0194] The multi-frame fusion vectorization module is used to fuse the vehicle state information and the single-frame 3D vector map to obtain a curb and lane line multi-frame fusion vector and a traffic sign and rod-shaped object multi-frame fusion vector, that is, to obtain a vector map.

[0195] The output module is used to output the vector map.

[0196] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. It should be noted that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the technical principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for constructing a vector map by multi-source data fusion, characterized in that, It includes the following steps: S1. Obtain camera information, point cloud information, and vehicle status information during vehicle driving; S2. Extract lane line 2D semantic information and traffic sign 2D semantic information from the camera information; at the same time, extract curb 3D vector map elements and pole 3D vector map elements from the point cloud information; S3. Integrate the lane line 2D semantic information, traffic sign 2D semantic information with the point cloud information and the extrinsic parameter relationship, and convert them into lane line 3D vector map elements and traffic sign 3D vector map elements, and form a single-frame 3D vector map with the curb 3D vector map elements and the pole 3D vector map elements; S4. Integrate the vehicle status information and the single-frame 3D vector map to obtain a multi-frame fusion vector of the curb and lane lines and a multi-frame fusion vector of traffic signs and poles, that is, obtain a vector map; The S3 includes: S31. Coarse segmentation of point cloud based on extrinsic parameter projection, removal of noise points based on Euclidean clustering, and recovery of traffic sign corner points under geometric constraints to obtain the traffic sign 3D vector map elements; S32. Robust ground point cloud segmentation based on ground slope change and lane line vectorization based on ground grid constraint to obtain the lane line 3D vector map elements; The S32 specifically includes: S321. Robust ground point cloud segmentation based on ground slope change: Taking the radar coordinate origin as the center, divide the field of view range into multiple fan-shaped ring regions according to the maximum distance, minimum distance, and fixed angle of the preset field of view range, and cut them into sub-fan-shaped rings with increasing width order to form a point cloud container; Judge each point cloud container, count the number of points in the point cloud container, and the Z-axis height characteristics of all points. The Z-axis height characteristics include the minimum value, maximum value, and Z-axis maximum difference in the Z-axis direction. Determine the ideal ground height, distance from the ideal ground threshold, obstacle determination threshold, and point cloud number threshold according to the calibration information; Set the category of the point cloud container, and judge the point cloud container as the corresponding category through the preset point cloud container type judgment rule according to the relationship between the number of points in the point cloud container, Z-axis height characteristics, ideal ground height, distance from the ideal ground threshold, obstacle determination threshold, and point cloud number threshold; Perform dimensionality reduction processing on the ground points, fit a straight line reflecting the ground height change according to the dimensionality-reduced ground points, traverse each point in the point cloud, calculate the distance from each point to the straight line, and determine whether the point is a ground point or a non-ground point according to the distance, and finally obtain the ground point cloud marked as ground points and the non-ground point cloud marked as non-ground points; S322. Lane line vectorization based on ground grid constraint: Use the ground point cloud for plane fitting to obtain a spatial plane expression; Divide the grid according to the point cloud segmentation method in S321, determine the X-axis and Y-axis coordinates of the grid vertices, and substitute them into the spatial plane expression to obtain the Z-axis coordinates of the grid vertices; For the grid common vertices, calculate the final Z value of the common vertices according to the number of ground points in the grids related to them and the determined Z value of the common vertices; Using the extrinsic parameter relationship between the radar and the camera, project the grid onto the plane in the camera coordinate system, associate the detected lane line key points with the corresponding grid, obtain the positions of the lane line key points in the radar coordinate system, and finally obtain the 3D vector map elements of the lane line; The S4 includes: S41. Obtain the multi-frame fusion vector of traffic signs and poles: Combined with vehicle state information, convert the 3D vector map elements of traffic signs and poles into the world coordinate system; Maintain a tracking queue for each individual object, and add the objects associated with the single-frame recovery to the queue; When outputting the final vectorization result, eliminate the queues with the number of objects in the tracking queue less than the preset number threshold; Use the confidence of each object in the tracking queue for weighted averaging to obtain the final position of the object; wherein, the confidence is calculated based on the position information and shape information of the object in the world coordinate system; When the confidence of the object is higher than the preset confidence threshold, it is considered that the match is valid, and the object with the highest confidence is selected for matching, and a new queue is created for the objects with invalid matches; For the situation where the same object is vectorized multiple times and tracked, perform confidence-weighted average vectorization; multiply the confidence by the center coordinates of the tracked object in the world coordinate system, and finally obtain the multi-frame fusion vector of traffic signs and poles; S42. Obtain the multi-frame fusion vector of road edges and lane lines: Use the DBSCAN algorithm to cluster the key points belonging to the same linear element to obtain the clustered set; Segment the points in the set at a fixed distance to obtain the segmented set; Use the weighted least squares method to perform cubic polynomial curve fitting on all the points in each subset to obtain the curve fitted by the subset; Judge through the fitting error. When the fitting error is greater than the preset error threshold, re-divide the subset and re-fit the curve until the fitting errors of all subsets are less than the preset error threshold; Obtain the linear element expression of each subset whose fitting error meets the requirements, determine the elevation of the lane line and the road edge, and finally obtain the multi-frame fusion vector of the road edge and the lane line.

2. The method for constructing a vector map by multi-source data fusion according to claim 1, characterized in that In the S2, it specifically includes: S21. Use the YOLOv8 network model to extract the 2D semantic information of the traffic signs from the camera information, and use the CLRNet network model to extract the 2D semantic information of the lane lines from the camera information; S22. Use the improved ground point cloud segmentation algorithm to obtain the 3D vector map elements of the road edge from the point cloud information, and use the rule-based point cloud pole extraction algorithm to obtain the elements of the 3D vector map of the pole from the point cloud information according to geometric constraints.

3. The method for constructing a vector map by multi-source data fusion according to claim 1, characterized in that In the S3, the 3D vector map elements of the lane line and the 3D vector map elements of the road edge form a combination of 3D vector map elements of the road edge and the lane line; the 3D vector map elements of the traffic signs and the 3D vector map elements of the poles form a combination of 3D vector map elements of the traffic signs and the poles; The combination of 3D vector map elements of the road edge and the lane line and the combination of 3D vector map elements of the traffic signs and the poles form the single-frame 3D vector map.

4. The method for constructing a vector map by multi-source data fusion according to claim 3, characterized in that The S31 specifically includes: S311. Coarse segmentation of point cloud based on extrinsic parameter projection: Combining the extrinsic parameter relationship of the sensor, a surjective relationship is established between the camera coordinate system and the lidar coordinate system, so that the 2D semantic information of traffic signs is integrated into the point cloud information; the coarse segmentation point cloud clusters on the traffic signs are screened out from the mapped point cloud data; S312. Removal of noise points based on Euclidean clustering: Perform Euclidean clustering on the coarse segmentation point cloud clusters, and divide the point cloud into multiple clustering point cloud clusters according to a preset distance threshold; Perform principal component analysis on each clustering point cloud cluster, and calculate its eigenvalues and eigenvectors; Comprehensively calculate the feature score according to the eigenvalues and the number of points in the clustering point cloud cluster, and select the clustering point cloud cluster with the highest feature score as the final point cloud of the traffic sign, and remove the remaining clustering point cloud clusters as noise points; S313. Restoration of traffic sign corner points under geometric constraints: Using the eigenvector of the clustering point cloud cluster obtained by principal component analysis, determine the normal vector of the clustering point cloud cluster, and combine the centroid of the clustering point cloud cluster to determine the plane equation of the traffic sign in the radar coordinate system; Convert the corner points of the traffic sign bounding box in the camera coordinate system to the radar coordinate system; By solving the system of equations composed of the plane equation of the traffic sign in the radar coordinate system and the line equation connecting the camera optical center and the traffic sign corner points, obtain the actual corner point coordinates of the traffic sign in the radar coordinate system.

5. The method for constructing a vector map by multi-source data fusion according to claim 1, characterized in that The confidence level is calculated based on the position information and shape information of the object in the world coordinate system, where the calculation formulas include the position confidence formula, the shape confidence formula, and the comprehensive confidence formula; The position confidence formula considers the difference in the center coordinates of the tracked object and the object to be tracked in the world coordinate system; The shape confidence formula considers the difference in the shape characteristics of the tracked object and the object to be tracked; The comprehensive confidence level is calculated by weighting the position confidence level and the shape confidence level.

6. The method for constructing a vector map by multi-source data fusion according to claim 1, characterized in that The cubic polynomial curve fitting process uses the weighted least squares method, considering the confidence level of each point, and obtaining the coefficients of the fitting curve through iterative solution; The linear element expression is determined according to the coefficients of the cubic polynomial obtained by fitting and the start point and end point of the subset.

7. A vector map construction system for multi-source data fusion, characterized in that, It includes a memory and a controller. When the computer-readable program stored in the memory is called by the controller, it can execute the steps of the method for constructing a vector map by multi-source data fusion according to any one of claims 1 to 6.

Images