A semantic map construction method based on vehicle-road cooperative perception system

Through the semantic map construction method of vehicle-road collaborative perception system, the problem of limiting the perception range of driverless cars in harsh environments is solved by using image and point cloud instance matching and LeGO-LOAM algorithm, and real-time detection of global traffic participants is realized and large-scale semantic map construction improves the ability of autonomous driving and the robustness of maps.

CN114882182BActive Publication Date: 2024-11-05SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210428195.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-22
Publication Date
2024-11-05
Estimated Expiration
2042-04-22

AI Technical Summary

Technical Problem

It is difficult for existing driverless cars to achieve full target detection, trajectory prediction and driving intention decisions under environmental conditions such as occlusion and harsh weather, resulting in limited perception range and the inability to build accurate vehicle planning and control maps.

Method used

Using a semantic map construction method based on vehicle-road collaborative perception system, the roadside perception device and vehicle-side lidar are combined, and the image and point cloud instance matching algorithm, RangeNet++ network and LeGO-LOAM algorithm are used to detect and integrate traffic participant data in real time. , build a global point cloud semantic map.

Benefits of technology

Real-time detection and recognition of all traffic participants on the road, build a large-scale semantic map, improve the perception and autonomous driving capabilities of driverless cars, provide rich semantic information and robust map construction, suitable for regional All vehicles in use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114882182B_ABST
    Figure CN114882182B_ABST
Patent Text Reader

Abstract

The present invention discloses a semantic map construction method based on a vehicle-road cooperative perception system, which relates to the field of intelligent driving technology and solves the technical problem of a small perception range of a single vehicle and a large error in the semantic map in unmanned driving technology. The key points of the technical solution are to match and fuse the traffic participant data detected by the vehicle-side and roadside perception devices using the idea of ​​vehicle-road cooperative perception; to semantically segment the point cloud using the RangeNet++ network, and then to establish a point cloud map using the LeGO‑LOAM algorithm. Detecting and identifying all traffic participants on the road can not only build a point cloud semantic map for a single vehicle in real time, but also fuse the point cloud semantic maps of each vehicle to form a large-scale semantic map for all vehicles traveling in the area. It has good application prospects, and the map algorithm constructed by this method has a simple structure, rich semantic information, a large mapping range, small errors, and strong robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, specifically to unmanned driving vehicle-road collaborative perception technology, and in particular to a semantic map construction method based on a vehicle-road collaborative perception system. Background Art

[0002] The acquisition and information processing of vehicle data is one of the key technologies to achieve autonomous driving of driverless cars. Single-vehicle intelligent autonomous driving is easily affected by environmental conditions such as occlusion and bad weather, and has difficulties in full-scale target detection, trajectory prediction, and driving intention "game". However, vehicle-road collaborative autonomous driving can greatly expand the perception range of a single vehicle, improve its perception capabilities, introduce new intelligent elements represented by high-dimensional data, and realize group intelligence through information interaction, collaborative perception, and collaborative decision-making control. It can fundamentally solve the technical bottlenecks encountered by single-vehicle intelligent autonomous driving and improve autonomous driving capabilities.

[0003] At present, all countries have made the development of vehicle-road cooperative systems a current research hotspot. The US vehicle-road cooperative system (Vehicle Infrastructure Integration, VII) is a special joint organization composed of the US Federal Highway Administration, AASH-TO, state transportation departments, the Automobile Industry Alliance, ITS American, etc., which realizes the integration of vehicles and road facilities through information and communication technology. The development focus of Japan's Smartway plan is to integrate the functions of various ITS in Japan and establish a common platform for on-board units, so that roads and vehicles can become Smartways and Smartcars through two-way transmission of IT consultations, so as to reduce traffic accidents and alleviate traffic congestion. The main content of the EU eSafety plan is: to make full use of advanced information and communication technologies, accelerate the research and development and integrated application of safety systems, and provide comprehensive safety solutions for road traffic. In May 2021, Baidu, in conjunction with the Institute of Intelligent Industry of Tsinghua University, formally proposed the Apollo Air plan, proposing V2X vehicle-road-cloud cooperative technology, which can provide a global perspective, provide ubiquitous connection technology and end-to-end application services for autonomous driving and intelligent transportation, and provide full-domain perception capabilities for urban transportation. It can be foreseen that the application and promotion of vehicle-road cooperative intelligent technology has become an inevitable choice for the development of modern road traffic.

[0004] How to build a semantic map through the vehicle-road collaborative perception system so that cars no longer rely on GPS and can plan, control and make decisions for vehicles correctly and timely is an urgent problem that needs to be solved. Summary of the invention

[0005] The present application provides a semantic map construction method based on a vehicle-road cooperative perception system, the technical purpose of which is to improve the perception range of a single vehicle and construct a large-scale semantic map for use by all vehicles traveling in the area.

[0006] The above technical objectives of this application are achieved through the following technical solutions:

[0007] A semantic map construction method based on a vehicle-road cooperative perception system, comprising:

[0008] S1: The roadside perception device detects traffic participants in real time based on the image and point cloud instance matching algorithm, obtains the roadside traffic participant data, and obtains the road line boundary information;

[0009] S2: converting the road-side traffic participant data into a structured data set T1(m) and sending it to the vehicle side; wherein m represents the total number of road-side traffic participants;

[0010] S3: The vehicle-side LiDAR collects point cloud data and performs real-time semantic segmentation of the point cloud through the RangeNet++ network;

[0011] S4: performing post-processing operations on the semantically segmented point cloud, converting the post-processed point cloud into a depth map, obtaining the vehicle-side traffic participant data and converting it into a structured data set T2(n); wherein n represents the total number of vehicle-side traffic participants;

[0012] S5: The vehicle side performs data integration and data matching fusion on the structured data set T1(m) and the structured data set T2(n) to obtain the full amount of traffic participant data;

[0013] S6: The point cloud after semantic segmentation is mapped using the LeGO-LOAM algorithm to obtain a point cloud map. All traffic participant data is then visualized in the point cloud map to obtain a global point cloud semantic map.

[0014] The beneficial effects of the present application are as follows: a semantic map construction method based on a vehicle-road cooperative perception system is proposed for driverless cars, and the idea of ​​vehicle-road cooperative perception is used to match and fuse the traffic participant data detected by the vehicle-side and roadside perception devices; the RangeNet++ network is used to perform semantic segmentation on the point cloud, and then a point cloud map is established through the LeGO-LOAM algorithm.

[0015] This vehicle-road cooperative system detects and identifies all traffic participants on the road. It can not only build a point cloud semantic map for a single vehicle in real time, but also integrate the point cloud semantic maps of each vehicle to form a large-scale semantic map for all vehicles traveling in the area. It has good application prospects. Experiments have shown that the map algorithm constructed by the method proposed in this application has a simple structure, rich semantic information, a large mapping range, small errors, and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the method described in this application;

[0017] Figure 2 Matching fusion algorithm flow chart for traffic participants;

[0018] Figure 3 Schematic diagram of the vehicle-road cooperative perception system. DETAILED DESCRIPTION

[0019] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 A flowchart of the method described in this application, such as Figure 1 As shown, the method includes:

[0021] S1: The roadside perception device detects traffic participants in real time based on the image and point cloud instance matching algorithm, obtains roadside traffic participant data, and acquires road line boundary information.

[0022] Specifically, the image and point cloud instance matching algorithm includes:

[0023] First, the image is segmented based on the YOLACT algorithm, and the point cloud is perspective projected. Then, instance matching, target point cloud clustering and three-dimensional model fitting are performed on the image after instance segmentation and the point cloud after perspective projection. The detection structure is fused, the bounding box is generated, and the road-side traffic participant data is obtained.

[0024] Road boundary information can be added by combining GPS and physical measurement because the roadside sensing device is a fixed-point device.

[0025] S2: Convert the road-side traffic participant data into a structured data set T1(m) and send it to the vehicle side; wherein m represents the total number of road-side traffic participants.

[0026] Specifically, the roadside perception device can communicate with the vehicle through the TCP communication protocol.

[0027] Structured data set T1(m)=(t1 1 ,t1 2 ,...,t1m ), where t1 m Represents the data of the mth road-end traffic participant, t1 m The parameters are expressed as (x1 m ,y1 m ,z1 m ) represents the center point coordinates of the mth road end traffic participant; l1 m ,ω1 m 、h1 m Respectively represent the length, width, and height of the bounding box of the mth road-end traffic participant; represents the direction angle of the mth road-end traffic participant; c1 m Indicates the category of the mth detected target at the road end.

[0028] S3: The vehicle-side LiDAR collects point cloud data and performs real-time semantic segmentation of the point cloud through the RangeNet++ network.

[0029] S4: Perform post-processing operations on the semantically segmented point cloud, convert the post-processed point cloud into a depth map, obtain the vehicle-side traffic participant data and convert it into a structured data set T2(n); where n represents the total number of vehicle-side traffic participants.

[0030] The post-processing operations of the semantically segmented point cloud include: compensating for missing data in the point cloud and removing ground points, then converting the point cloud into a depth map, clustering the point cloud and fitting a three-dimensional model, and finally obtaining the vehicle-side traffic participant data, which is converted into a structured data set T2(n); where T2(n) = (t2 1 ,t2 2 ,...,t2 n ), where t2 n represents the nth vehicle-side traffic participant data, t2 n The parameters are expressed as (x2 n ,y2 n ,z2 n ) represents the center point coordinates of the nth vehicle-side traffic participant; l2 n ω2 n 、h2 n Respectively represent the length, width, and height of the bounding box of the nth vehicle-side traffic participant; represents the direction angle of the nth vehicle-side traffic participant; c2 n Indicates the category of the nth detected target on the vehicle side.

[0031] The point cloud data is collected by LiDAR and converted into an undirected graph of size 1800*64 with an angular resolution of 0.2°, namely μmax =1800, k max = 64. Each point p in the point cloud μ,k Determined by the rotation index μ and the beam index k. μ,k Represents point p μ,k The measurement distance is, then in the point cloud of the adjacent beam, if point p μ,k r of the upper beam point μ,k R of the lower beam point μ,k The value is small, then the point p μ,k It is the key point, because when scanning the ground normally, the distance obtained by the upper beam scanning is definitely farther than the distance of the lower beam, so the key point can be preliminarily obtained.

[0032] Through the distance H r , height H z and beam index H i The distribution density of the point cloud is estimated to achieve preliminary clustering of key points in order to expand the key points; the adjacent key points of each column are clustered into a container and the height and beam index range of each container are calculated. If the number of key points in the container is less than 2, the container is discarded. Otherwise, the non-key points in each container within the beam index range are judged to decide whether to include the non-key points in the container and convert them into key points. The judgment function is:

[0033]

[0034] Among them, H z represents the height, i represents point i, H i Represents the total beam index of the point cloud, z i represents the coordinate of point i on the z-axis, p height Indicates the height of the point cloud.

[0035] Find a bunch of invalid points surrounded by key points. The invalid points are caused by the laser radar beam hitting an object with low reflectivity (such as car glass or black objects), which makes the beam unable to return to the sensor, causing the beam to be invalid at that moment. If the number of invalid points in a container is less than 30, and the Euclidean distance of the key points around the invalid points does not exceed 1.3m, these invalid points are simulated by linear fitting to compensate for the missing data of the point cloud.

[0036] In addition, the ground points are removed through the semantic segmentation results of the RangeNet++ network.

[0037] The above converts the point cloud into a depth map, clusters the point cloud and fits the three-dimensional model, and finally obtains the vehicle-side traffic participant data, which is converted into a structured data set T2(n), including:

[0038] Map the 3D point cloud (x, y, z) to a 2D depth image (μ, ν). The mapping formula is:

[0039]

[0040] in: represents the width of the 2D depth map, h represents the height of the 2D depth map; f up Indicates the upper viewing angle range of the radar in the vertical direction, f down Indicates the downward viewing angle range of the radar in the vertical direction; f = f up +f down Represents the total viewing angle range in the vertical direction; γ represents the distance from the point to the radar.

[0041] The point cloud is clustered using the BFS clustering algorithm combined with the breakpoints of the point cloud depth map. Finally, the convex hull contour approximation algorithm is used to fit the 2D bounding box from the target point cloud based on the clustering results. The 3D bounding box is obtained by combining the height information of the point cloud, and the 3D bounding box structured dataset T2(n) of the vehicle-side traffic participants is obtained.

[0042] S5: The vehicle side performs data integration and data matching fusion on the structured data set T1(m) and the structured data set T2(n) to obtain the full amount of traffic participant data.

[0043] Specifically, data integration includes the coordinate system of roadside perception equipment and vehicle-side, data model unification and classification and grading unification.

[0044] Data matching and fusion include: matching the repeatedly detected road-side traffic participant data with the vehicle-side traffic participant data, and then fusing the missing vehicle-side traffic participant data. The specific process is as follows: Figure 2 As shown, including:

[0045] S511: First, the coordinate systems of the road-side structured dataset T1(m) and the vehicle-side structured dataset T2(n) are converted into the world coordinate system. The converted T1(m) and T2(n) are respectively expressed as:

[0046] T1(m)=[x ωm ,y ωm ,z ωm ,l ωm ,ω ωm ,h ωm ,φ ωm ,c ωm ];

[0047] T2(n)=[x ωn ,y ωn ,z ωn ,l ωn ,ω ωn ,hωn ,φ ωn ,c ωn ].

[0048] Then the problem of data matching and fusion is transformed into the similarity problem of two matrices. The information dimensions of structured data include spatial position, shape, direction and semantic labels.

[0049] S512: According to the spatial position coordinates (x ωn ,y ωn ,z ωn ) constructs the KD data structure, using the spatial position coordinates (x ωm ,y ωm ,z ωm ) as the benchmark, based on the KD tree search algorithm, search in T2(n) for the first data t1 in the structured data set T1(m) 1 The two nearest vehicle-side traffic participants are then compared with the similarity between the roadside sensing device and the vehicle-side traffic participant data.

[0050] S513: Compared with the Euclidean distance, the Mahalanobis distance can solve the variance and correlation between different dimensions, specifically:

[0051] Calculate the mean of each dimension of the vector:

[0052] Compute the covariance between every two dimensions, where x w ,y w The covariance calculation formula between is:

[0053]

[0054] Finally, we can get the 8*8 covariance matrix ∑ and calculate its inverse matrix ∑ -1 , calculate t1 1 The Mahalanobis distance between the two nearest vehicle-side traffic participants is expressed as:

[0055]

[0056] like If the value is less than the preset threshold D, it is considered that there are repeated detections of traffic participants. The closest vehicle-side traffic participant is matched with the road-side traffic participant. If the match is consistent, it is confirmed that there are repeated detections of traffic participants. If the match is inconsistent, the road-side traffic participant data t1 is discarded. 1 Add to the vehicle-side structured data set T2(n); if MD is greater than the preset threshold D, it is considered that there is no repeated detection, and the road-side traffic participant data t1 1Added to the vehicle-side structured dataset T2(n).

[0057] Repeat steps S511 to S513 until all data in the road-side structured data set T1(m) are matched, and obtain the final vehicle-side structured data set T2(n'), n'≥n.

[0058] S6: The point cloud after semantic segmentation is mapped using the LeGO-LOAM algorithm to obtain a point cloud map, and then the data of all traffic participants are visualized in the point cloud map to obtain a global point cloud semantic map. The global point cloud semantic map includes all traffic participants with semantic labels, road line boundaries, and the point cloud map constructed by the LeGO-LOAM algorithm.

[0059] Figure 3 Schematic diagram of the vehicle-road cooperative perception system. Figure 3 As shown, in a large-scale vehicle-road perception collaborative system with multiple vehicles and multiple roads, the roadside shared server integrates the traffic participants of each roadside perception device in the road network, sends it to the cars traveling on the road network, matches and integrates it with the vehicle-side data, and each vehicle-side then feeds back the fusion results to the roadside shared server for data matching and fusion, thereby realizing real-time detection of all traffic participants in the road network.

[0060] The cars driving on the road network build local point cloud semantic maps, and match and fuse the point cloud maps offline to form a global point cloud semantic map. The global point cloud semantic map is combined with the fusion results of the roadside shared server to obtain a real-time semantic map of the road network for all vehicles to use.

[0061] The above are exemplary embodiments of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A semantic map construction method based on a vehicle-road cooperative perception system, characterized in that: include: S1: The roadside perception device detects traffic participants in real time based on the image and point cloud instance matching algorithm, obtains the roadside traffic participant data, and obtains the road line boundary information; S2: converting the road-side traffic participant data into a structured data set T1(m) and sending it to the vehicle side; wherein m represents the total number of road-side traffic participants; S3: The vehicle-side LiDAR collects point cloud data and performs real-time semantic segmentation of the point cloud through the RangeNet++ network; S4: performing post-processing operations on the semantically segmented point cloud, converting the post-processed point cloud into a depth map, obtaining the vehicle-side traffic participant data and converting it into a structured data set T2(n); wherein n represents the total number of vehicle-side traffic participants; S5: The vehicle side performs data integration and data matching fusion on the structured data set T1(m) and the structured data set T2(n) to obtain the full amount of traffic participant data; S6: Use the LeGO-LOAM algorithm to map the semantically segmented point cloud to obtain a point cloud map, and then visualize all traffic participant data in the point cloud map to obtain a global point cloud semantic map; Wherein, in the step S5, the data integration includes the coordinate system of the roadside sensing device and the vehicle end, the data model unification and the classification and grading unification; The data matching and fusion includes: matching the repeatedly detected road-side traffic participant data and vehicle-side traffic participant data, and then fusing the missing vehicle-side traffic participant data; The data matching and fusion specifically includes: S511: First, the coordinate systems of the structured data set T1(m) and the structured data set T2(n) are converted into the world coordinate system. The converted T1(m) and T2(n) are respectively expressed as: S512: According to the spatial position coordinates (x ωn ,y ωn , z ωn ) constructs the KD data structure, using the spatial position coordinates (x ωm ,y ωm , z ωm ) as a benchmark, based on the KD tree search algorithm, search in T2(n) for the two vehicle-side traffic participants that are closest to the first data t11 in T1(m); S513: Calculate the Mahalanobis distance between t11 and the two nearest vehicle-side traffic participants, expressed as: like If MD is less than the preset threshold D, it is considered that there are repeated detections of traffic participants, and the closest vehicle-side traffic participants are matched with the road-side traffic participants. If the matches are consistent, it is confirmed that there are repeated detections of traffic participants. If the matches are inconsistent, the road-side traffic participant data t11 is added to the vehicle-side structured data set T2(n); if MD is greater than the preset threshold D, it is considered that there are no repeated detections, and the road-side traffic participant data t11 is added to the vehicle-side structured data set T2(n); S514: Repeat steps S511 to S513 until all data in the road-side structured data set T1(m) are matched, and the final vehicle-side structured data set T2(n′) is obtained, where n′≥n.

2. The semantic map construction method according to claim 1, characterized in that: In step S1, the image and point cloud instance matching algorithm includes: First, the image is segmented based on the YOLACT algorithm, and the point cloud is perspective projected. Then, instance matching, target point cloud clustering and three-dimensional model fitting are performed on the image after instance segmentation and the point cloud after perspective projection. The detection structure is fused, the bounding box is generated, and the road-side traffic participant data is obtained.

3. The semantic map construction method according to claim 1, characterized in that: In step S2, the structured data set T1(m)=(t11, t12, ..., t1 m ), where t1 m Represents the data of the mth road-end traffic participant, t1 m The parameters are expressed as (x1 m , y1 m , z1 m ) represents the center point coordinates of the mth road end traffic participant; l1 m ,ω1 m 、h1 m Respectively represent the length, width, and height of the bounding box of the mth road-end traffic participant; represents the direction angle of the mth road-end traffic participant; c1 m Indicates the category of the mth detected target at the road end.

4. The semantic map construction method according to claim 1, characterized in that: In step S4, the post-processing operation of the semantically segmented point cloud includes: compensating for missing data in the point cloud, removing ground points, converting the point cloud into a depth map, clustering the point cloud and fitting a three-dimensional model, and finally obtaining the vehicle-side traffic participant data, and converting the vehicle-side traffic participant data into a structured data set T2(n); wherein T2(n)=(t21, t22, ..., t2 n ), where t2 n represents the data of the nth vehicle-side traffic participant, t2 n The parameters are expressed as (x2 n , y2 n , z2 n ) represents the center point coordinates of the nth vehicle-side traffic participant; l2 n ω2 n 、h2 n Respectively represent the length, width, and height of the bounding box of the nth vehicle-side traffic participant; represents the direction angle of the nth vehicle-side traffic participant; c2 n Indicates the category of the nth detected target on the vehicle side.

5. The method for constructing a semantic map according to claim 4, wherein: The compensation for missing data of the point cloud includes: obtaining key points, expanding the key points, clustering adjacent key points in each column into a container, and discarding the container if the number of key points in the container is less than 2. Otherwise, the non-key points in each container within the beam index range are judged to determine whether to include the non-key points in the container and convert them into key points. The judgment function is: Get the invalid points surrounded by key points. If the number of invalid points in the container is less than 30 and the Euclidean distance of key points around the invalid points does not exceed 1.3m, simulate these invalid points by linear fitting to compensate for the missing data of the point cloud. Removing ground points includes: removing ground points through semantic segmentation results of RangeNet++ network; Among them, H z represents the height, i represents point i, H i represents the beam index at point i, z i represents the coordinate of point i on the z-axis, p height Indicates the height of the point cloud.

6. The method for constructing a semantic map according to claim 4, wherein: The point cloud is converted into a depth map, the point cloud is clustered and a three-dimensional model is fitted, and finally the vehicle-side traffic participant data is obtained, and the vehicle-side traffic participant data is converted into a structured data set T2(n), including: Map the 3D point cloud (x, y, z) to a 2D depth image (μ, v). The mapping formula is: in: represents the width of the 2D depth map, h represents the height of the 2D depth map; f up Indicates the upper viewing angle range of the radar in the vertical direction, f down Indicates the downward viewing angle range of the radar in the vertical direction; f = f up +f down Indicates the total viewing angle range in the vertical direction; γ indicates the distance from the point to the radar; The point cloud is clustered using the BFS clustering algorithm, and the clustering results are fitted with a three-dimensional model to obtain a 3D bounding box, and then the vehicle-side traffic participant data and its structured data set T2(n) are obtained.