Method for infrastructure-assisted real-time hd mapping for autonomous driving

Abstract

A method and system for infrastructure assisted real-time HD mapping is provided. The method includes constructing geometry to project data to a bird's eye view (BEV) space; performing topology estimation based on the trajectory; and carrying out map fusion at the vehicle end. Construction of geometry includes execution of BEV projection, feature extraction, BEV instance segmentation and map vectorization. Performing BEV projection includes obtaining a cumulative point cloud and a vehicle trajectory, projecting the point cloud and trajectory to a ground plane in a BEV view, and rasterizing the points to generate a two-dimensional (2D) grid. Performing feature extraction includes extracting one or more features for each cell, and connecting the extracted features to form a feature map in which a trajectory direction vector occupies two channels. Performing BEV instance segmentation includes configuring a two-dimensional convolutional neural network (2D CNN) having a UNet-like structure to perform semantic instance segmentation using feature maps extracted in BEV.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202480073210.5 (22) Application Date 2024.09.27 (30) Priority Data 63 / 586,109 2023.09.28 US (85) PCT International Application Entering National Phase Date 2026.05.19 (86) PCT International Application Application Data PCT / CN2024 / 121699 2024.09.27 (87) PCT International Application Publication Data WO2025 / 067409 EN 2025.04.03 (71) Applicant: The Chinese University of Hong Kong Address: Sha Tin, New Territories, Hong Kong, China (72) Inventors: Xing Guoliang, He Yuze (74) Patent Agency: Beijing Gaowo Law Firm 11569 Patent Attorney Yan Yue (51) Int.Cl. G08G 1 / 0969 (2006.01) (54) Invention Title: Real-time HD Map Construction Method for Infrastructure-Assisted Autonomous Driving (57) Abstract: A method and system for real-time HD map construction for infrastructure assistance are provided. The method includes: constructing geometry to project data into a bird's-eye view (BEV) space; performing topology estimation based on trajectories; and performing map fusion at the vehicle end. Constructing geometry includes performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization. Performing BEV projection includes acquiring accumulated point clouds and vehicle trajectories, projecting the point clouds and trajectories onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional (2D) mesh. Performing feature extraction includes extracting one or more features for each cell and concatenating the extracted features to form a feature map in which trajectory direction vectors occupy two channels. Performing BEV instance segmentation includes configuring a two-dimensional convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV. Claims (2 pages), Description (22 pages), Drawings (13 pages), CN 122228534 A, 2026.06.16, CN 1 22 22 85 34 A. 1. A method for constructing a real-time HD map for infrastructure assistance, comprising: constructing geometry to project data onto a bird's-eye view BEV space; performing topology estimation based on trajectory; and performing map fusion at the vehicle end. 2. The method of claim 1, wherein constructing geometry comprises: performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization. 3. The method of claim 2, wherein performing BEV projection comprises: acquiring an accumulated point cloud and vehicle trajectory, projecting the point cloud and trajectory onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional 2D mesh.4. The method of claim 2, wherein performing feature extraction comprises: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, wherein a trajectory direction vector occupies two channels in the feature map. 5. The method of claim 2, wherein performing BEV instance segmentation comprises: configuring a 2D convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV. 6. The method of claim 5, wherein the 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output. 7. The method of claim 2, wherein performing map vectorization comprises: vectorizing the map to generate a sparse and compact representation of the HD map. 8. The method of claim 1, wherein performing topology estimation based on trajectory comprises: establishing and updating the map topology based on the instance segmentation results of accurate vehicle trajectories and lanes obtained from the constructed geometry. 9. The method of claim 1, wherein performing map fusion at the vehicle end comprises: merging the HD map received from the constructed geometry and the topology estimation with the onboard HD map. 10. The method of claim 9, wherein performing map fusion at the vehicle end includes: performing map element mapping, map alignment, and map element refitting. 11. A computer program product comprising: a non-transitory computer-executable storage device having computer-readable program instructions thereon, which, when executed by a computer, cause the computer to perform a method for infrastructure-assisted real-time HD map construction, the computer-executable program instructions including: constructing geometry to project data onto a bird's-eye view BEV space; performing topology estimation based on trajectories; and performing map fusion at the vehicle end. 12. The computer program product of claim 11, wherein constructing geometry includes: performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization. 13. The computer program product of claim 12, wherein performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectories, projecting the point cloud and trajectories onto a ground plane in a BEV view, and rasterizing the points to generate a two-dimensional 2D mesh. 14. The computer program product of claim 12, wherein performing feature extraction comprises: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, wherein a trajectory direction vector occupies two channels in the feature map.15. The computer program product of claim 12, wherein performing BEV instance segmentation comprises: configuring a two-dimensional convolutional neural network (2D CNN) having a UNet-like structure to perform semantic instance segmentation using feature maps extracted from the BEV. 16. The computer program product of claim 15, wherein the 2D CNN is configured to take the feature maps as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output. 17. The computer program product of claim 12, wherein performing map vectorization comprises: vectorizing the map to generate a sparse and compact representation of the HD map. 18. The computer program product of claim 11, wherein performing topology estimation based on trajectory comprises: establishing and updating the map topology based on the instance segmentation results of accurate vehicle trajectories and lanes obtained from the constructed geometry. 19. The computer program product of claim 11, wherein performing on-vehicle map fusion comprises: merging the HD map received from the constructed geometry and the topology estimation with the on-vehicle HD map. 20. The computer program product of claim 19, wherein map fusion at the vehicle end comprises: performing map element mapping, map alignment, and map element refitting. Claims 2 / 2 Page 3 CN 122228534 A Real-time HD Map Construction Method for Infrastructure Assistance in Automated Driving Cross-Reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 586,109, filed September 28, 2023, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0002] 1 Introduction

[0003] Automated driving systems promise to revolutionize the transportation industry. High-definition (HD) maps are an integral part of enabling these systems, facilitating perception and navigation of automated vehicles in their operating environment. HD maps comprise two core components: geometry and topology

[28] . In this context, geometry encompasses the location and semantics of static physical assets associated with roads, such as lanes, road boundaries, lane dividers, and crosswalks. It is important to emphasize that the HD maps considered employ a vectorized representation. Vectorization uses geometric primitives (including lines, curves, and polygons) to encode road geometry, rather than relying on raw point clouds. Topology, as a second core component, describes lane connectivity, illustrating how lanes or groups of lanes connect to each other, influenced by predetermined traffic rules and real-time road conditions. By providing a comprehensive and detailed representation of the environment, HD maps enable autonomous vehicles to understand the scene, map optimal routes, and make context-aware decisions.

[0004] Existing methods for building HD maps can be divided into two main schemes: offline and online. Offline construction typically involves labor-intensive data collection using specialized survey vehicles (each costing over $200,000) equipped with a combination of high-end sensors (such as cameras, LiDAR, GPS, IMU, and radar) [3, 42]. The use of SLAM (Simultaneous Localization and Mapping) technology [43, 53] helps to create globally consistent maps, followed by manual or semi-automatic map annotation.

[0005] Online HD map construction methods. HDMapNet

[27] was the first study to introduce the problem of learning HD semantic maps. It encodes features from a single-frame LiDAR point cloud and / or images from surrounding cameras and predicts semantic map elements in a bird's-eye view. STSU [9] proposed an end-to-end method that simultaneously extracts local road network maps and detects objects given only images from a front-facing camera. VectorMapNet

[29] uses the Transformer module to predict a set of sparse polylines in a bird's-eye view to model the geometry of HD maps. These studies all take onboard sensor observations as input and are therefore limited by physical obstacles (such as obstacle occlusion) and sensing range.

[0006] Infrastructure-assisted vehicle perception. Recent studies have shown the potential of infrastructure in enhancing perception for autonomous driving. VI-Eye

[21] proposed a semantic-based point cloud registration method for merging vehicle point clouds with infrastructure point clouds. VIPS

[44] uses graph matching algorithms to fuse object detection results from infrastructure and vehicles. Michael et al.[7] used infrastructure to detect, track and predict vehicle motion to build an environment model that was transmitted to the vehicle for motion planning. In these studies, infrastructure was mainly used for dynamic object perception. VI-Map differs from these attempts in that it aims to use infrastructure to build accurate and up-to-date HD maps on the infrastructure side, thereby assisting in the generation of onboard HD maps.

[0007] Major players in the autonomous driving industry, such as TomTom

[47] , HERE

[46] , and Baidu [3], have adopted this approach because it can generate very detailed and comprehensive map information. However, this existing practice is too costly. In addition, the constructed maps are prone to becoming obsolete between successive mapping iterations [24, 38].This predicament stems from the inherent impracticality of maintaining up-to-date HD maps covering large areas, given the vast scale of road networks and the frequent changes in lane connectivity

[37] . Consequently, most current offline HD maps focus on highways, resulting in a severe lack of coverage of urban roads [6].

[0008] To address the challenges inherent in offline approaches, recent technological advances [9, 22, 27, 29, 30] have explored the concept of online HD map learning. This approach aims to estimate local HD maps in real time based on observations from onboard sensors, such as point clouds from LiDAR or images captured by surrounding cameras. While this strategy reduces reliance on global offline HD maps and offers the potential for more cost-effective and scalable HD map construction, it is not without its limitations. These approaches are subject to inherent limitations stemming from onboard sensors and dynamic road conditions. These challenges encompass the constraints of limited sensor field of view (due to factors such as occlusion and rapid movement) and variations in sensor data quality. Furthermore, online HD maps often lack road topology due to the complexity of inferring such logical information in real time. Consequently, existing online map construction schemes may be fragile and incomplete.

[0009] Embodiments of the present invention relate to a method and system for constructing HD maps for autonomous driving.

[0010] According to an embodiment of the present invention, a method for constructing real-time HD maps with infrastructure assistance is provided. The method includes: constructing geometry to project data into a bird-eye-view (BEV) space; performing topology estimation based on trajectories; and performing map fusion at the vehicle end. Constructing geometry includes performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization. Furthermore, performing BEV projection includes acquiring accumulated point clouds and vehicle trajectories, projecting the point clouds and trajectories onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional (2D) grid. Performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which trajectory direction vectors occupy two channels. Performing BEV instance segmentation includes: configuring a two-dimensional convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV. The 2D CNN is configured to take feature maps as input and generate pixel-wise masks for each individual road element, thus obtaining instance masks as output. Furthermore, map vectorization is performed to generate a sparse and compact representation of the HD map. Trajectory-based topology estimation involves building and updating the map topology based on accurate vehicle trajectories and lane instance segmentation results obtained from the constructed geometry.Map fusion at the vehicle end includes merging an HD map received from geometry construction and topology estimation with an onboard HD map. Furthermore, map fusion at the vehicle end includes performing map element mapping, map alignment, and map element refitting.

[0011] In another embodiment of the invention, a computer program product is provided, including a non-transitory computer-executable storage device containing computer-readable program instructions that, when executed by a computer, cause the computer to perform a method for infrastructure-assisted real-time HD map construction, the computer-executable program instructions including: constructing geometry to project data into a bird's-eye view (BEV) space; performing topology estimation based on trajectories; and performing map fusion at the vehicle end. Constructing geometry includes performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization. Furthermore, performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectories, projecting the point cloud and trajectories onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional (2D) mesh. Performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which the trajectory direction vector occupies two channels. Performing BEV instance segmentation includes: configuring a two-dimensional convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV. The 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output. In addition, performing map vectorization includes: quantizing the map to generate a sparse and compact representation of the HD map. Performing topology estimation based on trajectory includes: establishing and updating the map topology based on the instance segmentation results of accurate vehicle trajectories and lanes obtained from the constructed geometry. Performing map fusion at the vehicle end includes: merging the HD map received from the constructed geometry and topology estimation with the vehicle-mounted HD map. In addition, performing vehicle-mounted map fusion includes: performing map element correspondence, map alignment, and map element refitting. Brief Description of the Drawings

[0012] This patent application or application document contains at least one color drawing. Upon request and payment of the necessary fees, the Patent Office will provide a copy of the published text of this patent or patent application with color drawings.

[0013] Figure 1 illustrates infrastructure-assisted HD map construction according to an embodiment of the present invention.

[0014] Figures 2A and 2B illustrate vehicle behavior with topologies extracted from an outdated HD map (Figure 2A) and a new HD map (Figure 2B) according to an embodiment of the present invention. Figure 2A shows that the topology of the outdated HD map fails to reflect vehicle malfunctions, with the right-hand side showing a vehicle failing to make a right-turn decision and causing traffic congestion. Figure 2B shows that the topology of the new HD map depicts blocked and passable lanes, with the right-hand side showing a vehicle changing lanes into a passable lane ahead of schedule.

[0015] Figure 3 is a schematic diagram illustrating the clearer perception brought about by infrastructure point cloud accumulation according to an embodiment of the present invention.

[0016] Figure 4 is an illustration of objects extracted from vehicle trajectories according to an embodiment of the present invention (from left to right: vehicle trajectory, trajectory attributes, and corresponding map elements).

[0017] Figure 5 illustrates the system architecture of VI-Map according to an embodiment of the present invention.

[0018] Figure 6 illustrates the design of HD map geometry construction on roadside infrastructure according to an embodiment of the present invention.

[0019] Figure 7 illustrates the design of HD map fusion on vehicles according to an embodiment of the present invention.

[0020] Figures 8A-8C illustrate a real-world test platform deployed for VI-Map data collection and system evaluation according to an embodiment of the present invention, wherein Figure 8A shows a modified vehicle, Figure 8B shows mobile roadside infrastructure, and Figure 8C shows 18 test locations in two cities.

[0021] Figures 9A and 9B show Town 5 maps used in the CARLA dataset according to an embodiment of the present invention, wherein Figure 9A shows the location of roadside infrastructure, and Figure 9B shows the HD map parsed from an OpenDRIVE file.

[0022] Figure 10 shows the HD map IoU score when a vehicle passes through a road segment containing infrastructure according to an embodiment of the present invention.

[0023] Figure 11 shows that VI-Map extends the range of online HD maps according to an embodiment of the present invention.

[0024] Figure 12 shows that VI-Map improves ride comfort according to an embodiment of the present invention.

[0025] Figure 13 shows that VI-Map improves traffic efficiency according to an embodiment of the present invention.

[0026] Figure 14 shows the HD map construction results of different methods in a real-world scenario according to an embodiment of the present invention.

[0027] Figures 15A and 15B illustrate the topology estimation performance of different update strategies according to embodiments of the present invention under different road types, wherein Figure 15A shows the response and success time, and Figure 15B shows the success rate.

[0028] Figure 16 shows the variation of HD map IoU score of fused maps generated by different map merging methods according to embodiments of the present invention with vehicle positioning error.

[0029] Figure 17 illustrates the runtime latency of each step of VIMap on a vehicle under different road types according to an embodiment of the present invention. Specification 3 / 22 pages 6 CN 122228534 A Detailed Description

[0030] Embodiments of the present invention relate to a method and system for infrastructure-assisted real-time HD map construction.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. As used herein, unless the context clearly indicates otherwise, the singular forms “a”, “an”, and “the” are intended to also include the plural forms. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the said features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0032] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant technical context and the context of this disclosure, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0033] When the term “about” is used herein in conjunction with numerical values, it should be understood that the value can be in the range of 90% to 110% of that value, i.e., the value can be + / - 10% of said value. For example, “about 1 kg” means from 0.90 kg to 1.1 kg.

[0034] Intelligent roadside infrastructure or roadside units (RSUs) equipped with sensors and computing units provide a superior solution for HD map construction. Notably, a key property unique to roadside infrastructure is its ability to continuously observe road segments in a static state. This capability overcomes the shortcomings of both offline and online methods. First, compared to offline methods, uninterrupted continuous observation enables infrastructure to update dynamically evolving HD maps in a timely manner. Second, compared to online technologies, the wide field of view and statically accumulated observations provide infrastructure with comprehensive, unobstructed, and high-quality sensor data.

[0035] According to embodiments of the present invention, a system and method for infrastructure-assisted real-time HD map construction (named VI-Map) are provided, aiming to utilize roadside infrastructure to create and maintain HD maps for autonomous driving. In particular, the infrastructure utilizes its own sensors (such as LiDAR or 2D / 3D cameras) to construct and refresh HD maps.Then, the vehicle integrates this map with its own HD map in real time to enhance / update the vehicle's scene understanding (as shown in Figure 1). The core idea of ​​VI-Map is to use the unique, static, and cumulative observations of roadside infrastructure to achieve accurate and up-to-date HD map construction. Specifically, VI-Map first extracts multiple (e.g., 5) carefully designed concise bird's-eye view (BEV) features from the dense point cloud and accurate vehicle trajectories captured by the infrastructure, and then uses them for efficient map geometry construction. Next, VI-Map uses the latest vehicle trajectories to estimate and update the current map topology. Finally, VI-Map employs a new three-stage map fusion algorithm to merge the HD map of the infrastructure with the vehicle's HD map. It is worth noting that VI-Map is not intended to replace existing HD map construction methods. Instead, it provides a key complementary HD map construction paradigm for autonomous driving by leveraging the increasingly prevalent intelligent roadside infrastructure.

[0036] VI-Map offers several key advantages: (i) it transforms large amounts of unstructured, cumulative 3D data into structured, compact, and concise 2D bird's-eye view (BEV) features. These features can be processed by efficient 2D CNNs, greatly reducing computational overhead and enabling practical deployment on edge devices of infrastructure. (ii) VI-Map generates vectorized HD maps on infrastructure. This vectorized representation is very lightweight, minimizing communication overhead between infrastructure and vehicles. In addition, the representation is compatible with the widely adopted industry standard HD map data format OpenDRIVE

[36] . (iii) VI-Map does not require precise location of roadside infrastructure or time synchronization between vehicles and infrastructure. This greatly reduces the barriers to widespread deployment of roadside infrastructure and adoption of the solution. (iv) VI-Map allows for simple, fast, and flexible deployment because it allows roadside infrastructure nodes to build their own HD maps independently and then merge them into a continuous global vehicle-mounted HD map specification 4 / 22 pages 7 CN 122228534 A. Therefore, VI-Map can work with mobile RSUs deployed on complex and rapidly changing road segments where the need for new HD map updates is urgent. Therefore, VI-Map can provide patches for global HD map updates of important road segments, complementing existing offline and online map building schemes. This provides a highly scalable architecture for the construction of HD maps assisted by the infrastructure for autonomous driving.

[0037] VI-Map was implemented on a real-world test platform that included modified passenger vehicles and mobile RSUs deployed on 18 different road segments in two cities, covering up to five types of roads. Two new datasets were collected using this test platform and a leading autonomous driving simulation platform, respectively.The results show that, compared with online map building methods, VI-Map can expand the local HD map range of vehicles by 3 times and improve map accuracy by 2.8 times, while generating only 42 milliseconds of map fusion latency on the vehicle. At the application level, VI-Map improves traffic efficiency of problem road sections by more than 5 times and improves ride comfort index by 3.9 times. Video demonstration of VI-Map on a real-world test platform can be obtained at https: / / youtu.be / p2RO65R5Ezg.

[0038] Embodiments of the present invention utilize roadside infrastructure to enhance real-time HD map building for autonomous vehicles. It utilizes the unique cumulative observations of roadside infrastructure to build and maintain accurate and up-to-date HD maps. Then, the HD map is fused with the vehicle-mounted HD map in real time to form a more comprehensive and up-to-date HD map. The present invention has wide applications in autonomous driving and intelligent transportation systems, covering multiple fields such as smart cities, infrastructure-assisted autonomous driving, cooperative perception, and online HD map building.

[0039] According to embodiments of the present invention, the system and method for infrastructure-assisted real-time HD map building for autonomous driving includes three key components. (1) Geometric construction. The module first extracts five carefully designed concise bird's-eye view (BEV) features from the dense point cloud and accurate vehicle trajectories captured by the infrastructure, and then uses them for efficient map geometry construction. (2) Topology estimation. The module uses the latest vehicle trajectories to estimate and update the current map topology. The resulting accurate vectorized map geometry, together with the new topology, forms a concise infrastructure HD map. (3) Map fusion. The module is executed inside the vehicle. It receives the infrastructure HD map and merges it with the onboard HD map. The module cleverly utilizes the semantic, proximity, and directional attributes of the vectorized HD map to facilitate the rapid integration of infrastructure and vehicle HD maps, thereby providing real-time HD map support for autonomous driving.

[0040] The system and method for infrastructure-assisted real-time HD map construction were tested in an autonomous driving simulator and five real-world road segments. The experiments showed that the system and method for infrastructure-assisted real-time HD map construction can construct HD maps at the decimeter level (up to 0.3 meters) and achieve real-time (up to 42 milliseconds latency) map fusion between driving vehicles and roadside infrastructure. Compared to state-of-the-art online HD map building methods, this represents a significant improvement in map accuracy and coverage, by 2.8 times and 3 times, respectively. The prototype was validated experimentally and tested on simulators and five real-world road sections, achieving high map building accuracy in both cases.

[0041] A new HD map building paradigm is provided that leverages roadside infrastructure to enhance real-time HD map building for autonomous driving.Systems and methods for infrastructure-assisted real-time HD map construction utilize unique observations of roadside infrastructure to build accurate HD maps at the infrastructure end, and then fuse them with in-vehicle HD maps in real time to form a more comprehensive and up-to-date HD map.

[0042] Materials and Methods

[0043] 2 Motivation Case Study

[0044] This section first presents a case study to understand the limitations of existing HD maps on autonomous vehicles (Section 2.1). Then, key insights into the advantages that infrastructure provides in generating timely and high-quality HD maps are presented, highlighting the potential of infrastructure-assisted HD map construction (Section 2.2). Specification 5 / 22 pages 8 CN 122228534 A

[0045] 2.1 Limitations of Existing In-Vehicle HD Maps

[0046] Offline HD maps. Autonomous vehicles rely on up-to-date HD maps for both global path planning and local behavioral decisions and motion planning. However, various road conditions (such as road construction, congestion, and accidents) cause road geometry and topology (i.e., lane connectivity) to change constantly, thus requiring frequent updates to HD maps to ensure the safety and efficiency of autonomous driving [24, 38].

[0047] Vehicles using outdated HD maps may make incorrect behaviors and motion planning decisions that do not conform to the current road conditions, resulting in suboptimal or even dangerous driving performance. Using CARLA

[16] as an example, CARLA is a popular driving simulator that has been used to develop industrial autonomous driving systems such as Apollo [2] and Autoware.

[0048] In this case study, a faulty vehicle blocked the rightmost lane, causing a change in road topology, and the right lane was no longer connected to other lanes. Figures 2A and 2B illustrate the behavior of vehicles using outdated HD maps and new HD maps, respectively. Using outdated HD maps, vehicles experience sudden braking and sharp turns, resulting in speed reduction or even congestion. In contrast, the new HD map allows vehicles to perceive lane conditions in advance and make better decisions, such as choosing the unobstructed left lane, thus navigating affected road sections more smoothly and quickly (average speed 1.6 m / s vs. 8.1 m / s). It should be noted that the decisions made by CARLA's autonomous driving agent in the case study may not be optimal. Nevertheless, even with off-the-shelf autonomous driving agent implementations, significant benefits of maintaining real-time HD maps have been observed.

[0049] Online HD Maps. Online map building methods use onboard sensors to acquire up-to-date HD maps. However, such maps are susceptible to the limited sensor range of vehicles and uncertain sensor data quality. For visual illustration, online map building methods were run on real vehicle LiDAR data collected from city streets

[27] . Detailed results / visualizations can be found in the video in the abstract.In particular, occlusion of LiDAR by surrounding vehicles results in incomplete and fragmented maps. Evaluations show that when half of the road is occluded, the integrity of online generated maps may be less than 25%, failing to meet the stringent safety requirements of autonomous driving.

[0050] 2.2 Advantages of Roadside Infrastructure

[0051] As described in Section 2.1, offline HD maps provide complete but outdated perception, while the latest online HD maps built solely by vehicles may be inaccurate. This study addresses these issues by utilizing infrastructure-assisted HD map building and updating. Roadside infrastructure enables continuous observation of road segments while stationary, which is ideal for local HD map building because it offers two key advantages: higher perception quality and the ability to capture real-time topology changes. Roadside infrastructure, including off-the-shelf RSUs [12, 13, 20], is becoming increasingly prevalent. This trend presents an opportunity to leverage roadside infrastructure to improve HD map building for autonomous vehicles.

[0052] Complete and clear perception. Sensors mounted on infrastructure provide a wider field of view and a longer perception range, and are less susceptible to occlusion compared to sensors mounted on vehicles. Furthermore, by accumulating sensor data over a period of time, more detailed and accurate observations of the road can be obtained. The left and right images of Figure 3 show a single-frame point cloud and a 10-second overlay point cloud of the infrastructure, respectively. The accumulated point cloud from the infrastructure can provide more detailed and accurate information (such as lane lines and road boundaries), which is crucial for autonomous driving.

[0053] Accurate trajectory observation. By using LiDAR for vehicle detection and tracking, the infrastructure can obtain continuous and accurate trajectories (with decimeter-level accuracy

[33] ), which is unique to the infrastructure and cannot be obtained by other methods (such as GPS of vehicles). The key observation is that these new and accurate trajectories provide valuable information for inferring the geometry and topology of the road. The results are illustrated using a typical real-world intersection as shown in Figure 4. Trajectory density distinguishes lanes and helps to find lane dividers and road boundaries. The trajectory direction is consistent with the map topology, indicating the directed connectivity between lanes. The trajectory direction variance reflects the difference in driving directions at the same location, which is higher near intersections. Therefore, it can be used to infer potential pedestrian crossings on page 6 / 22 of the specification, CN 122228534 A. These observations can be integrated to achieve real-time and highly lightweight map building and updating.

[0054] 3 Design of VI-MAP

[0055] 3.1 System Overview

[0056] A system and method for infrastructure-assisted real-time HD map building (named VI-Map) that utilizes roadside infrastructure to provide real-time HD map building for autonomous vehicles. Figure 5 shows an overview of VI-Map.Specifically, VI-Map utilizes unique data collected from roadside infrastructure (including accumulated point clouds and precise vehicle trajectories) to build and maintain HD maps. It is noteworthy that these two data types are unique to roadside infrastructure.

[0057] VI-Map comprises three key components. First, to effectively handle the large volume of unstructured and heterogeneous 3D point cloud and trajectory data, the geometry construction (Section 3.2) module is configured to project these data types into BEV space, producing a streamlined, structured, and unified 2D BEV representation. Then, the geometry construction module extracts specific features from both data types, refining valuable information tailored for generating map geometry. The geometry construction module utilizes relatively new but large volumes of point cloud and trajectory data for high-precision geometric prediction. In contrast, topology estimation (Section 3.3) uses newly arrived trajectories for topology inference. An update strategy is designed to identify trajectory changes and trigger topology updates. The resulting precise vectorized map geometry, along with the new topology, constitutes a concise infrastructure HD map. Finally, the map fusion performed inside the vehicle (Section 3.4) receives the infrastructure HD map and merges it with the vehicle HD map. The map fusion module cleverly utilizes the semantic, proximity, and directional attributes of the vectorized HD map to facilitate the rapid integration of infrastructure and vehicle HD maps, thereby providing real-time HD map support for autonomous driving. In addition, the vectorized HD map itself is very lightweight, resulting in minimal communication overhead and strong adaptability to changing communication conditions between infrastructure and vehicles.

[0058] 3.2 Geometry Construction

[0059] This module is designed to generate the geometric part of the HD map on the infrastructure using two key data sources obtained from the infrastructure (cumulative point cloud and vehicle trajectory). In this design, map geometry is defined as a vectorized representation of four types of road elements: road boundaries, lane dividers, lanes, and pedestrian crossings. These elements cover the most common elements in HD maps and are consistent with existing online map construction methods. Specifically, line-based elements (such as road boundaries and lane dividers) are represented as splines, while region-based elements (such as lanes and pedestrian crossings) are represented as polygons.

[0060] Geometry construction presents the following challenges: (i) point clouds and trajectories are two heterogeneous data types that may be difficult to process simultaneously. (ii) Processing multi-frame accumulated 3D point clouds on edge devices with limited computing resources is a challenge. These challenges are addressed by projecting both data types onto a bird's-eye view (BEV) and rasterizing them. Thus, (i) both unstructured LiDAR points and trajectories are converted into regular grids, enabling the simultaneous learning of features from both inputs to infer geometry; and (ii) this image-like BEV representation can be efficiently processed by highly lightweight 2D CNNs, eliminating the need for resource-intensive 3D point cloud DNNs.This method enables efficient map building on mobile GPUs (Graphics Processing Units).

[0061] Figure 6 illustrates the pipeline of the geometry building module, which includes four steps: BEV projection, feature extraction, instance segmentation, and vectorization. In the initial step, two types of accumulated data are projected onto the BEV space. From this projection, five carefully crafted features are extracted, which are tailored for inferring map geometry. These features are then fed into a 2D CNN for instance segmentation. This process produces map element instances represented as a grid map. Finally, the grid-based representation of the map elements is vectorized to create a vectorized HD map.

[0062] BEV Projection. First, the process of acquiring the accumulated point cloud and accurate vehicle trajectory is described. For each infrastructure point cloud frame, a 3D multi-object tracking method such as AB3DMOT (A Baseline for 3D Multi-Object Tracking, specification 7 / 22 pages 10 CN 122228534 A) is configured to detect and track vehicles. Frames without vehicle detection or tracking are accumulated into a static point cloud, while trajectories are saved for the tracked vehicles. The point cloud and trajectories are then projected onto the ground plane in the BEV view. Points in the point cloud greater than 0.5m above the ground are filtered out, as they may contain points of trees or buildings irrelevant to map building. Both LiDAR points and trajectories are projected onto the ground plane, resulting in a set of 2D points (x, y), where (x, y) represent the coordinates of each point. Finally, the points generating the 2D mesh are rasterized. Specifically, the points are distributed to pixel locations (u, v), with a grid size of (H, W). Rasterization is represented as u = [(x - xmin) / α], v = [(y - ymin) / α]. The grid height and width are represented as H = [(xmax - xmin) / α], W = [(ymax - ymin) / α], where [·] is the rounding operation, and α represents the rasterization resolution. For example, when α = 10, the cell size is 0.1m × 0.1m. After the previous step, BEV grid representations of the point cloud and trajectory are obtained, respectively, where each cell may contain a different number of points. Each point contains some features, which will be designed and extracted in the next step.

[0063] Feature extraction. Multiple (e.g., five) features are selected from the LiDAR points and trajectory points for HD map construction. The features include, but are not limited to, height, intensity, density, orientation mean, and orientation variance.For LiDAR points, the maximum distance to the ground plane in each cell is calculated as the height feature (∈ R1), which helps infer road boundaries, as curbs are typically 0.2–0.3 m higher than the ground. The average intensity (∈ R1) of points in each cell is also calculated. High intensity indicates the presence of ground markings (such as lane dividers and pedestrian crossings), as the paint used for ground marking printing is typically reflective. For trajectory points, as described in Section 2.2, multiple (e.g., three) features are calculated, including but not limited to density (∈ R1), orientation mean (∈ R2), and orientation variance (∈ R1). Density is calculated by counting the number of trajectory points in each cell. 2D orientation vectors are used instead of angles to represent orientation, improving the smoothness of the representation space. Orientation variance is defined as where is the average of orientation vectors {}, and is the L2 norm of the average orientation. In summary, a total of five features are extracted for each cell and concatenated to form a feature map with shape (H, W, 6), where the trajectory orientation vectors occupy two channels.

[0064] BEV Instance Segmentation. A 2D CNN with a UNet-like structure is used to perform semantic instance segmentation using feature maps extracted from BEVs. Prediction is made for four types of road elements: lanes, road boundaries, lane dividers, and pedestrian crossings. The CNN is trained by combining a weighted cross-entropy loss [3 9] and an instance clustering loss [1 5], which is represented as follows. The CNN takes a feature map of shape (H,, 6) as input and generates a pixel-wise mask for each individual road element, thus outputting an instance mask. The instance segmentation results are also used as input to the topology estimation module.

[0065] Map Vectorization. Instance segmentation maps for each category of road elements cannot be used directly by vehicles because they are incompatible with most autonomous driving frameworks that typically use vectorized representations of HD maps. Therefore, the maps are further vectorized to generate a sparse and compact representation of the HD map, minimizing the amount of map data and thus reducing the communication overhead during the transmission of infrastructure HD maps to vehicles.

[0066] The standard HD map specification in ASAM OpenDRIVE is adopted to model road elements as spline curves (for boundaries and dividers) or polygons (for lanes and crosswalks). Each instance is first extracted as a set of pixel locations on the segmentation map. To fit each boundary and divider, a cubic function (u) that minimizes the mean square error is found: . This solution can be easily found by least squares regression. For lanes and crosswalks, the fitted minimum bounding rectangle is used as the geometric representation.

[0067] 3.3 Topology Estimation

[0068] This module establishes and updates the map topology by utilizing the precise vehicle trajectories and lane instance segmentation results from Section 3.2 (page 8 / 22, CN 122228534 A). A graph is used to represent the topology, following the definitions in OpenDRIVE. In this context, the nodes and edges of the graph correspond to lanes and the connections between lanes, respectively. The design of this module is based on a key observation: precise vehicle trajectories can be used to infer map topology and identify topology changes, because there is a strong correlation between lane connectivity and trajectories, allowing inference from one to another. Furthermore, since trajectories are continuous, new trajectory data can be used to estimate and update the map topology in a timely manner.

[0069] The principle of topology construction is as follows: if two lanes are traversed by the same trajectory, they are considered connected. Connectivity is directional, described by the direction of the trajectory. As described in Section 2.1, the road topology is dynamic. Therefore, an update strategy is designed to determine when an update is triggered when the topology changes. This update strategy detects topology changes in both time and space dimensions and performs cross-validation to reduce erroneous updates. Specifically, in the time dimension, it is assumed that for each lane i, the arrival times t of k vehicle trajectories follow a Poisson distribution. The continuous arrival times are discretized and defined as the number of time units. Each time unit lasts for 1 second. The probability mass function for each lane i is defined as: . The rate parameter for each lane i is dynamically updated based on the latest historical observation set of the arrival times of k trajectories within a predefined time interval. In the experiment, the time interval is set to one hour, which is a common update interval in many traffic flow monitoring and prediction studies. Then, a test with a significance level of 0.05 is used to determine whether the historical observations of each lane i follow the current Poisson distribution. In the spatial dimension, three trajectory features of the most recent k trajectories are calculated, namely trajectory density, directional mean, and directional variance. Then, the cross-entropy loss l between the current trajectory features and the features within the one-hour time interval is calculated. The map topology is updated only when p>0.05 and l exceeds a specified threshold.

[0070] 3.4 Map Fusion

[0071] This module runs on the vehicle and is configured to merge the HD map received from the infrastructure with the onboard HD map. This step involves solving the map fusion problem, namely finding the coordinate transformation between two maps and using this transformation to integrate them. Existing map fusion techniques are mainly developed for scenarios such as multi-robot collaborative SLAM, where the merged map is an occupancy grid map. However, due to the difference in map types—merging two vectorized HD maps—existing methods cannot directly adapt to the context.

[0072] To this end, a three-stage map fusion algorithm is provided to merge infrastructure HD maps and vehicle HD maps. Figure 7 shows the pipeline of this module.First, the semantic attributes and spatial proximity of map elements are used to establish their correspondence. Then, directional characteristics are used to determine the transformation between the two maps based on the established correspondence. Then, the two HD maps are precisely aligned using this transformation. Finally, each pair of corresponding map elements is refitted into a unified representation, and the final merged HD map is created

[0073] . First stage: Map element correspondence. To merge two HD maps, each with multiple road elements, it is necessary to first find the element correspondence between infrastructure and vehicles. First, an initial transformation is used to transform the infrastructure HD map from its infrastructure coordinates to vehicle coordinates. This initial transformation can be easily derived based on the infrastructure and vehicle poses obtained from GPS and IMU data. Due to GPS errors, this initial transformation may be inaccurate and will be optimized in the next stage. Then, within the vehicle HD map, a direct method is used, that is, each map element is matched with the corresponding element in the infrastructure HD map. This matching is established based on shared semantic labels and proximity. The corresponding element is the closest element in the infrastructure HD map that has the same semantic label. The distance between two elements is defined as the distance between their nearest endpoints. One fact is that elements in an infrastructure HD map may correspond to multiple elements in a vehicle HD map, especially when occlusion from the vehicle's viewpoint causes fragmentation of map elements. It is worth noting that semantic labels of map elements are used to improve the accuracy and efficiency of matching. Specifically, elements belonging to the same semantic category are specifically selected as potential counterparts. This simple and effective method has shown commendable results in establishing correspondences, even when the initial transformation is inaccurate.

[0074] Second stage: Map alignment. After finding all corresponding element pairs, the errors in the initial transformation are eliminated by optimizing the alignment between all element pairs. Note that alignment is performed using only linear elements (splines) because it reveals more location information. The optimal rigid transformation is found by minimizing the novel direction-aware chamfer distance between two spline curves. Specifically, the splines are first transformed back to points by sampling at fixed intervals. Then the tangent direction (,) of each point is calculated. Combining the location (u, v), a tuple is assigned to each point, and two sets of tuples, {u} and {v}, are obtained from the infrastructure and vehicles. Next, in each iteration, tuple correspondences are first determined by searching for the nearest tuple in each {v}. The distance is defined as the Euclidean distance between two tuples. Then, a weighted root mean square estimate of the rigid transformation is used, , where is a hyperparameter that controls the degree to which the estimate should be biased towards points with similar orientations. All estimated transformations are averaged over each corresponding element, and this average is used to transform the splines on the infrastructure.Then, the above process is repeated for multiple iterations. In one embodiment, convergence is found to be achieved in five iterations. Finally, a fine transformation is obtained to align map elements from infrastructure to vehicles.

[0075] Third stage: map element refitting. Each pair is merged into a union map element. For linear elements, sampling points are densely taken on both elements and the fitting process described in Section 3.2 is repeated. For regional features, the union of a pair of corresponding elements is simply taken. An integrated HD map aligned with the vehicle's viewpoint is obtained for potential downstream tasks.

[0076] 4 Test platform and dataset

[0077] In this section, the test platform for VI-Map and the dataset collected for evaluation are provided.

[0078] Test platform. VI-Map is implemented on both real-world settings and the CARLA simulator for extensive data collection and evaluation. Figures 8A-8C show real-world settings that include modified vehicles and custom mobile poles as roadside infrastructure units. Each pole is equipped with two Livox AVIA LiDARs at a height of 5 meters.

[31] The Livox HAP LiDAR

[32] was mounted approximately 1.7 meters above the vehicle. The mobile pole was equipped with an NVIDIA Jetson Orin and an 802.11ac WiFi router for wireless communication with the vehicle. The test vehicle, as shown in Figures 8A-8C, carried another Orin to run the online HD map baseline building

[27] and VI-Map map fusion code. Both the vehicle and the mobile pole were equipped with a NEO-M8T GPS

[49] and an HWT9052-485 IMU

[52] to estimate pose.

[0079] This paper presents a cost analysis of the mobile RSU, providing valuable insights into its practical application in real-world scenarios. Table 1 shows a comparison of the hardware and cost of the RSUs used in existing studies[25, 48] and VI-Map. Each RSU mainly includes several components such as sensors, computing units, and communication units. It is worth noting that the cost of RSUs has the potential to be further reduced, especially as the price of LiDAR continues to decline over time. To ensure comprehensive road coverage, a deployment distance of approximately 50 meters between two adjacent RSUs is recommended. Table 1. Hardware and cost comparison of RSUs used in existing studies [25, 48] and VI-Map 10 / 22 pages 13 CN 122228534 A

[0080] Existing public datasets for autonomous driving [8, 18] only provide vehicle-centric point cloud data and cannot be used to evaluate VI-Map. Therefore, two new datasets were collected, one from a real-world testing platform and one generated by the CARLA simulator

[16] . Both datasets contain point clouds from infrastructure and vehicles over constant time periods.The real-world datasets were collected from 18 road segments in two cities, covering various road types (T-junctions, crossroads, curves, straight roads, etc.) and different numbers of lanes from single lane to six lanes. Table 2 summarizes the two datasets. Table 2. Summary of the two new datasets, where "infr." represents infrastructure and "veh." represents vehicles

[0081] Real-world datasets. During the collection of the datasets, the average collection time for infrastructure point clouds was 3 minutes (maximum approximately 10 minutes). The average vehicle speed was 25 km / h (maximum approximately 30 km / h). To obtain ground-based HD maps, the point clouds of each pair of infrastructure and vehicles were first registered, and then the fused point clouds were projected onto the BEV and rasterized into images with a resolution of 0.15 m / pixel. Next, the map geometry (polylines and polygons) was manually labeled in the images using the CVAT tool. HD maps were labeled for each pair of infrastructure-vehicle point clouds and individual vehicle point clouds if they had little overlap with the infrastructure point clouds. Independent ground-based HD maps of infrastructure and vehicles were obtained by cropping the fused HD maps. The map topology of each road segment was manually labeled as an n×n binary array, where n represents the number of lanes in the road segment, and the values ​​1 and 0 represent the connection and disconnection of two lanes, respectively. Thus, an infrastructure HD map and multiple vehicle HD maps were obtained for each road segment, which were used as training data for the geometry building module and the online map building baseline, respectively. Vehicles in the infrastructure point cloud were also labeled for training data of the tracking algorithm AB3DMOT.

[0082] CARLA dataset. The dataset was also rendered in CARLA

[16] . Roadside infrastructure was configured at different locations in Town 5 in the CARLA ecosystem. Each infrastructure and ego vehicle was equipped with a LiDAR with 32 channels and 360° field of view, consistent with mainstream autonomous driving datasets [8, 18]. Traffic flow was simulated using 200 vehicles roaming throughout the town. Map files provided by CARLA (in the .xodr file format defined by OpenDRIVE) were parsed to generate ground-based HD maps of the entire town. Then, by cropping the town map according to the pose of each subject, vehicle point clouds and local HD maps of each infrastructure are obtained for each frame. Figures 9A and 9B show the locations of all infrastructure and the global HD map. The CARLA dataset covers 50 road segments in a 439m × 509m town. Furthermore, CARLA is able to simulate events that cause changes in road topology, such as vehicle breakdowns or accidents, which would be dangerous in real-world simulations. Vehicle breakdowns were simulated for three different road types (including T-junctions, intersections, and two-lane straight roads).Different degrees of road marking blurring were also simulated, including but not limited to light (<20%), moderate (50%-60%) and severe (>90%).

[0083] 5 Evaluation Specification 11 / 22 pages 14 CN 122228534 A

[0084] 5.1 Evaluation Setup and Metrics

[0085] 5.1.1 Evaluation Setup. Based on the road coverage of the infrastructure point cloud, the height H and width W of the BEV grid (see Section 3.2) were set to 300. The geometric model, vehicle tracking model

[51] and baseline model

[27] of VI-Map were trained or fine-tuned using the two datasets introduced in Section 4. Training was performed on a server equipped with an Intel Xeon Silver 4210 CPU and an NVIDIA RTX 2080Ti GPU. A 10-fold validation method was used for the three models.

[0086] 5.1.2 IoU, CDP, CDL, CD, P, R. Following the evaluation methods in [19, 27, 29], and employing these widely accepted evaluation metrics, we assess the accuracy of map geometry. Intersection-over-union (IoU) and Chamfer distance (CD) are used as metrics for evaluating linear elements (such as boundaries and separators). For the area element pedestrian crossings, IoU, precision (P), and recall (R) are used as metrics. Specifically, IoU is an Eulerian metric used to measure the pixel semantic difference between the predicted map and the ground reality, denoted as IoU, where DP, DG, RH×W×D are dense representations of map elements (rasterized curves and polygons on a BEV grid), DP is the prediction, DG represents the ground reality, H and W are the height and width of the BEV grid, D is the number of map element categories, and |·| represents the size of the set. CD is a Lagrangian metric used to measure the spatial distance between vector geometries (curves or splines). CDDir is the directional chamfer distance, and CD is the bidirectional chamfer distance. Specifically, CDDir is defined as CDDir, where and are two sets of points sampled on the prediction curve and the ground reality curve. CDP represents the CD from prediction to label (equivalent to precision), while CDL represents the CD from label to prediction (equivalent to recall). CD is defined as CD(,) = CDDir(,) + CDDir(,). The precision and recall metrics for regional elements are defined as and respectively. CDP and CDL reflect the accuracy and completeness of the prediction map, respectively.

[0087] 5.1.3 Ride comfort, average transit time, and traffic throughput.The effectiveness of VI-Map was evaluated using these three user experience-related metrics. Ride comfort was quantified using the longitudinal and lateral acceleration of the vehicle, which are widely considered key metrics for evaluating ride comfort [23, 35]. Lower acceleration values ​​correspond to better ride comfort; average transit time is the average time it takes for a vehicle to pass through a road segment; traffic throughput is the number of vehicles passing through a road segment within a fixed time.

[0088] 5.1.4 Response time, success time, and success rate. These three metrics were used to evaluate the performance of the map topology update strategy. Response time is defined as - , where and are the times when the road topology changes and the infrastructure map topology is updated, respectively. Success time is defined as - , where is the time when the map topology is correctly updated. Since updates may be erroneous, may differ from . Success rate is the number of times the topology is correctly updated divided by the total number of updates.

[0089] 5.2 End-to-end evaluation

[0090] The end-to-end system performance of VI-Map was evaluated on real-world roads containing four road types, including two-way single-lane intersections, T-junctions, four-lane straight roads, and curves. Figure 10 shows the trajectory of a vehicle entering and exiting an intersection with moving poles on the roadside. Gray dots represent the vehicle point cloud along the trajectory. Colored dots represent the vehicle's GPS location, with different colors representing the IoU of the constructed HD map. At the beginning and end of the trajectory, only the onboard HD map is available, with an IoU of 40%. When the vehicle enters the infrastructure coverage area, the IoU of the fused HD map increases to 80%. The results indicate that VI-Map can benefit vehicles by utilizing accurate infrastructure maps, thereby helping vehicles safely navigate complex road sections. The end-to-end latency of VI-Map was also measured. VI-Map achieved an average end-to-end latency of 37ms and a maximum of 42ms, indicating that VI-Map can work in real time, as most autonomous vehicles have a sensor frame rate of 10 Hz. Furthermore, a gap in IoU was observed at the beginning and end of the trajectory (see page 12 / 22 of the manual, CN 122228534 A), even though both only had the vehicle's HD map. This is because the road markings at the starting position were severely worn, resulting in a very low IoU for the online map. The impact of incomplete road markings will be evaluated in Section 5.6.

[0091] 5.3 Benefits of VI-Map

[0092] In this section, the benefits of VI-Map for autonomous vehicles are evaluated. The main observations in this section are: (i) VI-Map’s accurate map geometry complements and extends the vehicle’s online HD map beyond the limitations of single-vehicle perception. (ii) VI-Map’s new map topology enables vehicles to make better behavioral decisions in the current road conditions, benefiting passengers, vehicles and traffic systems.

[0093] 5.3.1 Extending the range of online HD mapsThe range expansion provided by VI-Map was evaluated when the distance between the vehicle and infrastructure changed. Figure 11 shows that the range of the VI-Map-fused HD map can be doubled compared to using only the online HD map, even when the vehicle is very close to the infrastructure (e.g., <10 m). This is because infrastructure provides a wider field of view due to its height and is less likely to be obstructed than the vehicle. The expansion of the HD map range is the basis for VI-Map to benefit many downstream tasks and improve the safety of autonomous driving.

[0094] 5.3.2 Improved ride experience and traffic efficiency. VI-Map provides vehicles with up-to-date map topology to support downstream tasks such as decision making and motion planning. How map freshness affects the behavior of autonomous vehicles, and the impact and consequences of these behaviors, were evaluated. An intersection was constructed in CARLA where vehicles with hard-coded artificial intelligence (AI) used HD maps and surrounding information for autonomous driving. The following steps were performed. First, all autonomous vehicles driving in the town were equipped with offline HD maps provided by CARLA. Second, the vehicles were set to stop suddenly to simulate a lane congestion scenario, which caused the road topology to change and the offline HD map to become outdated. Record the data trajectory of the road segment before and after the topology change, including the infrastructure point cloud and the timestamps, accelerations, and positions of all vehicles passing through the road segment. Third, input the infrastructure point cloud into VI-Map to generate an updated HD map. Manually update the topology of the corresponding road segment in the CARLA map and load the new map onto all autonomous vehicles. Finally, let the vehicles drive using the new map and record the new data trajectory.

[0095] Ride comfort, average transit time, and traffic throughput were calculated. Figure 12 shows the average ride comfort under three conditions. The results show that driving with an outdated map results in a poor ride experience due to frequent starts, stops, and sharp turns. Compared with the outdated map, VI-Map significantly improves ride comfort by 3.9 times, even comparable to the comfort level in non-accident scenarios. This is because VI-Map can update the map in a timely manner, enabling vehicles to make better decisions, resulting in a smoother and more comfortable ride experience. Figure 13 shows the average vehicle transit time and traffic throughput of the road segment. Compared with the outdated map, VI-Map reduces the time for vehicles to pass through the problem road segment to one-fifth and increases traffic throughput by 2 times.

[0096] 5.4 Performance of VI-Map

[0097] 5.4.1 Geometry Construction. The geometry construction of VI-Map was evaluated on a real-world test platform.VI-Map was compared to two baselines: (i) a state-of-the-art online HD map building method called HDMapNet, which uses only vehicle data

[27] ; and (ii) a modified HDMapNet-based method in which the original point cloud of infrastructure was fused to the vehicles. Figure 14 shows an example of map building results at an intersection using the two baselines and VI-Map. The blue and red splines and the green rectangles are directly from the output of VI-Map. The orange arrows are manually labeled to show the topology of the map generated by the system, which is in its original form as a graph (Section 3.3). It was determined that the HD map generated by VI-Map provides a wider range compared to the method using only vehicles. In addition, VI-Map provides a topology and a more accurate HD map than the two baselines.

[0098] VI-Map was then evaluated on a real-world dataset. Table 3 shows the results using the metrics defined in Section 5.1.2. VI-Map outperformed the baselines on all metrics. In particular, VI-Map improved the IoU by 16%–35% compared to HDMapNet. VI-Map achieves decimeter-level map accuracy (i.e., CD) with an average error of 0.3m. The results also show that HDMapNet can benefit from the infrastructure point cloud (IoU improvement of 20%) thanks to the unobstructed view of the infrastructure. However, VI-Map outperforms HDMapNet using infrastructure data by a significant margin, producing an improvement of approximately 60% in CD. This is because the superposition of the two single-frame point clouds of infrastructure and vehicles is still too sparse to accurately predict the location of map elements. VI-Map utilizes accumulated point clouds and precise trajectories to reveal more details and clues to accurately locate map elements. Table 3. Geometric accuracy of HD maps on real-world datasets. Higher IoU (%) is better. Lower CD (m) is better.

[0099] It was observed that the infrastructure point cloud contributes more significantly to the improvement of HDMapNet on lane dividers compared to road boundaries and pedestrian crossings. This is because lane dividers occupy a relatively small area, and inference relies primarily on point intensity, while overlaying infrastructure point clouds makes the intensity more prominent within a specific small area. Furthermore, the difference between VI-Map's CDP and CDL (approximately 13%) is significantly smaller than that of the baselines. This is because the HD maps generated from the two baselines may be incomplete due to sparse point clouds and potential occlusion, while VI-Map's complete HD map ensures consistency between the two metrics. Additionally, compared to the Euclidean metric IoU, VI-Map demonstrates a greater improvement over the two baselines on the Lagrange metric CD (35%–26% improvement vs. 65%–60% improvement). This is because VI-Map generates continuous map elements and fits them into a vectorized shape.In addition, ablation studies were conducted to evaluate the effectiveness of trajectories in map geometry construction. Table 3 also shows the VI-Map without trajectory input, where IoU decreased by 7% and CD increased by 50%. This result supports the design that accurate trajectories contain features of inferred map geometry and can achieve better performance by combining them with point cloud features.

[0100] 5.4.2 Topology Estimation. The performance of the HD map topology update strategy was evaluated by examining the response time, success time, and success rate defined in Section 5.1.4. The update strategy in Section 3.3 was compared with two baselines: (i) fixed time period: updating the topology for a fixed time period T; (ii) fixed number of trajectories: updating the topology after observing a fixed number of additional trajectories n. For the former approach, T was set to the traffic light cycle length of the intersection and 60 seconds for straight roads. For the fixed number of trajectories approach, n was set to 3 × the number of lanes. It is worth noting that these values ​​were selected after extensive searching to determine the optimal performance parameters for each baseline approach. Because map update performance can vary with different road types and traffic conditions, three different road conditions were evaluated: a city center intersection, a suburban T-junction, and a four-lane straight road. Specifically, five simulations were performed for each of these three different road conditions, under the same conditions described in Section 5.3.2. Figures 15A-15B show the performance of VI-Map and the two baselines. In particular, Figure 15A shows the response time and success time of the three update methods. VI-Map provided the shortest success time in all road conditions, although its response was not the fastest. The two baselines responded quickly but took longer to obtain the correct topology update. VI-Map considers topology updates at the lane level, while the two baselines do not. Therefore, the trajectories collected under the two baseline strategies may be unevenly distributed among lanes, with some lanes possibly having no trajectories, which could lead to update failures. Figure 15B shows the success rate. VI-Map achieved an average success rate of 84%, significantly exceeding the 30%-44% of the two baselines. The update strategy integrates temporal and spatial changes to detect changes in the topology, and suppresses erroneous updates through cross-validation.

[0101] 5.4.3 Map Fusion. The map fusion method was then tested using a real-world dataset. The module runs on a vehicle and is responsible for merging the infrastructure map with the vehicle map. Two baseline methods were implemented by adapting the ideas of typical map merging methods in collaborative SLAM to this scenario: a probabilistic method [5, 26, 54] and an optimization method [4, 10, 34]. The vectorized map was rasterized into a grid map and the implementation of these methods was followed. Figure 16 shows the performance of the fused map obtained after applying these fusion methods.The curves in Figure 16 show the map IoU of the three fusion methods under different vehicle positioning errors. The green bars in Figure 16 show the distribution of GPS positioning errors. The results show that when the positioning error is greater than 8 m, VI-Map achieves an IoU improvement of more than 40% compared to the baseline. As the positioning error increases, the performance of the two baselines drops sharply, while VI-Map maintains a map IoU of more than 78% under all tested positioning errors. This is attributed to VI-Map treating the map as individual map elements and using the instance-level correspondence of these elements to accurately align and merge the two maps. In contrast, the two baseline methods treat the map as a whole and rely only on the initial transformation to align the two maps.

[0102] 5.5 System Overhead

[0103] 5.5.1 Vehicle Step Latency. The runtime latency of VI-Map at each step on the vehicle and the runtime latency of the entire map fusion process were measured. Since the runtime of map fusion may be affected by road complexity (i.e., the number of map elements), three different road types in the real world were tested. Figure 17 shows the runtime for each step, where the error bars represent the lowest 5% and 95% of the runtime across all frames under the same settings. The results show that the overall on-vehicle computation time of VI-Map is less than 50 ms for all road types, which meets the stringent real-time requirements of autonomous driving applications (prediction

[17] , decision making

[11] , and motion planning

[50] ). Thus, VI-Map achieves real-time HD map construction on vehicles with the assistance of roadside infrastructure. The matching step accounts for half of the total computation time because it involves calculating the distance between pairs of map elements. It is worth noting that this runtime can be further accelerated by computation on a GPU. The runtime of the transformation and alignment steps does not increase with road complexity because they are independent of the number of map elements.

[0104] 5.5.2 Communication overhead. The communication overhead of VI-Map is compared with methods for transmitting raw point cloud data (e.g., HDMapNet with infrastructure) or rasterized map data [4, 10, 26, 54]. Data transmission time was measured using the real-world testbed described in Section 4, where vehicles and infrastructure communicated via an 802.11ac Wi-Fi network with a bandwidth of 80 MHz. Average data volume and transmission time are shown in Table 4. VI-Map transmitted 71.2 KB of vectorized map representation (spline control points, polygon vertices, and topology) with an average cycle time of 13.6 ms. Compared to transmitting raw point cloud data, VI-Map reduced data volume and time by approximately 42 times and 40 times, respectively, and by 22 times compared to transmitting rasterized map data.Therefore, VI-Map can be deployed on a wide range of V2X (Vehicle to Everything) networks, even those with low communication bandwidth. Table 4. Average size and transmission time of data shared over an 802.11ac network.

[0105] 5.6 System Robustness

[0106] As observed in Section 5.2, online HD maps can be significantly affected by worn road markings. In this section, VI-Map was evaluated under different degrees of road marking incompleteness. Specifically, CARLA was used to mask different proportions of road marking points in the point cloud. Table 5 shows that as road marking incompleteness increases, all metrics of VI-Map decrease slightly and steadily, maintaining acceptable performance even under the most severe occlusion, and still outperforming the baseline in Table 3 without severe occlusion. Unlike online map building methods that rely solely on sensor data (point cloud), VI-Map additionally utilizes precise trajectory observations of infrastructure and extracts unique features valuable for map building. This design allows VI-Map to adapt to various road marking conditions because they do not affect trajectory features. Table 5. Accuracy of HD map geometry of VI-Map under different degrees of road marking incompleteness.

[0107] 6 Discussion

[0108] Scalability of VI-Map. The adaptability of VI-Map extends to a variety of road types and traffic scenarios. First, the BEV features used to infer the HD map are low-level features directly derived from the raw data without making any assumptions about road structure or number of lanes. Second, busy or sparse traffic conditions mainly affect the accumulation time of static point clouds and vehicle trajectories. However, it does not impair the accuracy of the constructed HD map. However, VI-Map has limited scalability in inferring 3D map elements with complex shapes (such as traffic lights, fire hydrants, or bus stops). This limitation is mainly attributed to projecting 3D data into a 2D BEV space and performing subsequent processing within that domain, which leads to information loss during dimensionality reduction. Therefore, this method may not produce optimal performance when dealing with complex 3D map elements because the details of their 3D geometry are lost during projection.

[0109] Limitations and Failure Cases. VI-Map may not achieve satisfactory performance under adverse weather conditions (such as rain, snow, and fog) due to significant noise in the LiDAR data. Furthermore, VI-Map may also encounter challenges in high-speed driving scenarios, as increased speed leads to a significant deterioration in point cloud quality, thus affecting VI-Map performance. Nevertheless, geometry construction, topology estimation, and map fusion can be modified and applied to other sensing modalities, such as 3D cameras.

[0110] VI-Map was developed to enhance in-vehicle HD maps by providing accurate and timely infrastructure HD maps, utilizing roadside infrastructure. VI-Map was implemented end-to-end, and experimental results show that VI-Map enhances existing HD map construction methods in terms of map geometric accuracy, map topological freshness, system robustness, and efficiency. The system and method for infrastructure-assisted real-time HD map construction can be applied to industries such as autonomous driving, intelligent transportation systems, and smart cities.

[0111] Example 1. A method for infrastructure-assisted real-time HD map construction, comprising: constructing geometry to project data into a bird's-eye view (BEV) space; performing topology estimation based on trajectory; and fusion of maps on the vehicle side.

[0112] Example 2. The method according to Example 1, wherein constructing geometry comprises: performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization.

[0113] Example 3. The method according to any of the preceding embodiments, wherein performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectory, projecting the point cloud and trajectory onto a ground plane in a BEV view, and rasterizing the points to generate a two-dimensional (2D) mesh.

[0114] Example 4. The method according to any of the preceding embodiments, wherein performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which trajectory vectors occupy two channels.

[0115] Example 5. The method according to any of the preceding embodiments, wherein performing BEV instance segmentation includes: configuring a two-dimensional convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV.

[0116] Example 6. The method according to any of the preceding embodiments, wherein the 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output.

[0117] Example 7. The method according to any of the preceding embodiments, wherein performing map vectorization includes: vectorizing the map to generate a sparse and compact representation of the HD map.

[0118] Example 8. The method according to any of the preceding embodiments, wherein performing topology estimation based on trajectory includes: establishing and updating the map topology based on the accurate vehicle trajectory and lane instance segmentation results obtained from the constructed geometry.

[0119] Example 9. The method according to any of the preceding embodiments, wherein performing map fusion at the vehicle end includes: merging the HD map received from the constructed geometry and topology estimation with the vehicle-mounted HD map.

[0120] Example 10. The method according to any of the preceding embodiments, wherein performing map fusion at the vehicle end includes: performing map element mapping, map alignment, and map element refitting.

[0121] Example 11. A computer program product, comprising: a non-transitory computer-executable storage device having computer-readable program instructions thereon, which, when executed by a computer, cause the computer to perform a method for infrastructure-assisted real-time HD map construction, the computer-executable program instructions including: constructing geometry to project data into a bird's-eye view (BEV) space; performing topology estimation based on trajectories; and performing map fusion at the vehicle end.

[0122] Example 12. The computer program product according to Example 11, wherein constructing geometry includes: performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization.

[0123] Example 13. A computer program product according to any of the preceding embodiments, wherein performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectory, projecting the point cloud and trajectory onto a ground plane in a BEV view, and rasterizing the points to generate a two-dimensional (2D) mesh.

[0124] Example 14. A computer program product according to any of the preceding embodiments, wherein performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which trajectory direction vectors occupy two channels.

[0125] Example 15. A computer program product according to any of the preceding embodiments, wherein performing BEV instance segmentation includes: configuring a two-dimensional convolutional neural network (2D CNN) with a UNet-like structure to perform semantic instance segmentation using the feature map extracted from the BEV.

[0126] Example 16. A computer program product according to any of the preceding embodiments, wherein the 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output.

[0127] Example 17. A computer program product according to any of the foregoing embodiments, wherein performing map vectorization includes: vectorizing the map to generate a sparse and compact representation of the HD map.

[0128] Example 18. A computer program product according to any of the foregoing embodiments, wherein performing topology estimation based on trajectory includes: establishing and updating the map topology based on the instance segmentation results of accurate vehicle trajectories and lanes obtained from the constructed geometry.

[0129] Example 19. A computer program product according to any of the foregoing embodiments, wherein performing map fusion at the vehicle end includes: merging the HD map received from the constructed geometry and topology estimation with the vehicle-mounted HD map.

[0130] Example 20. A computer program product according to any of the foregoing embodiments, wherein map fusion at the vehicle end includes: performing map element correspondence, map alignment and map element refitting.

[0131] All patents, patent applications, provisional applications and publications mentioned or cited herein, including all drawings and tables, are incorporated herein by reference in their entirety, provided that they do not contradict the explicit teachings of this specification.

[0132] It should be understood that the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes that would occur to those skilled in the art should be included within the spirit and scope of this application. Furthermore, any element or limitation of any invention or embodiment thereof disclosed herein may be combined with any and / or all other elements or limitations (alone or in any combination) or any other invention or embodiment thereof disclosed herein, and all such combinations are covered within the scope of the invention without limiting it.

[0133] References

[0134] [1] Autoware. 2023. The Autoware Foundation - open source for autonomous driving. https: / / www.autoware.org / . (2023).

[0135] [2] Baidu. 2023. Apollo. https: / / apollo.auto / . (2023).

[0136] [3] Zhibin Bao, Sabir Hossain, Haoxiang Lang and Xianke Lin. 2022. High-definition map generation technologies for autonomous driving: a review. arXiv preprint arXiv:2206.05400 (2022).

[0137] [4] Andreas Birk and Stefano Carpin. 2006. Merging occupancy grid maps from multiple robots. Proc. IEEE 94, 7 (2006), 1384–1397.

[0138] [5] Jose-Luis Blanco, Javier González-Jiménez and Juan-Antonio Fernández-Madrigal. 2013. A robust, multi-hypothesis approach to matching occupancy grid maps. Robotica 31, 5 (2013), 687–701.

[0139] [6] Lidar News Blog. 2022. HD Map Database Coverage Doubled. https: / / blog.lidarnews.com / hd-map-database-coverage-doubled / . (2022).

[0140] [7] Michael Buchholz, Johannes Müller, Martin Herrmann, Jan Strohbeck, Benjamin Völz, Matthias Maier, Jonas Paczia, Oliver Stein, Hubert Rehborn and Rüdiger-Walter Henn. 2021. Handling occlusions in automated driving using a multiaccess edge computing server-based environment model from infrastructure sensors. IEEE Intelligent Transportation Systems Journal 14, 3 (2021), 106–120.

[0141] [8] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. See IEEE / CVF Conference Proceedings on Computer Vision and Pattern Recognition, pp. 11621–11631.

[0142] [9] Yigit Baran Can, Alexander Liniger, Danda Pani Paudel and Luc Van Gool. 2021. Structured bird's-eye-view traffic scene understanding from onboard images. Proceedings of the IEEE / CVF International Conference on Computer Vision. 15661–15670.

[0143]

[10] Stefano Carpin. 2008. Fast and accurate map merging for multi-robot systems. Autonomous Robots 25 (2008), 305–316.

[0144]

[11] Xue-Mei Chen, Min Jin, Yi-song Miao and Qiang Zhang. 2017. Driving decision-making analysis of car-following for autonomous vehicle under complex urban environment. Journal of Central South University 24 (2017), 1476–1482.

[0145]

[12] Cohda. 2023. Cohda Wireless MK6C EVK RSU. https: / / www.cohdawireless.com / solutions / hardware / mk6c-evk / . (2023).

[0146]

[13] Commsignia. 2023. Commsignia Roadside Unit. https: / / www.commsignia.com / products / . (2023).

[0147]

[14] CVAT. 2023. CVAT Annotation tool. https: / / www.cvat.ai / . (2023).

[0148]

[15] Bert De Brabandere, Davy Neven and Luc Van Gool. 2017. Semantic instance segmentation for autonomous driving. See Proceedings of the IEEE Workshop on Computer Vision and Pattern Recognition. 7–9.

[0149]

[16] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. See Conference on robot learning. PMLR, 1–16.

[0150]

[17] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou et al. 2021. Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset. See IEEE / CVF International Conference on Computer Vision Proceedings. 9710–9719.

[0151]

[18] Andreas Geiger, Philip Lenz and Raquel Urtasun. 2012. Are we ready for autonomous driving? the kitti vision benchmark suite. See IEEE Conference on Computer Vision and Pattern Recognition 2012. IEEE, 3354–3361.

[0152]

[19] Nikhil Gosala, Kürsat Petek, Paulo L. J. Drews-Jr, Wolfram Burgard and Abhinav Valada. 2023. SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images. See IEEE / CVF Conference Proceedings on Computer Vision and Pattern Recognition. 14901–14910.

[0153]

[20] HARMAN. 2023. HARMAN Savari StreetWAVE RSU. https: / / 19 / 22 pages 22 CN 122228534 A car.harman.com / solutions / connectivity / harman-savari-streetwave. (2023).

[0154]

[21] Yuze He, Li Ma, Zhehao Jiang, Yi Tang and Guoliang Xing. 2021. VI-eye: semantic-based 3D point cloud registration for infrastructure-assisted autonomous driving. See Proceedings of the 27th International Conference on Mobile Computing and Networking. 573–586.

[0155]

[22] Namdar Homayounfar, Wei-Chiu Ma, Justin Liang, Xinyu Wu, Jack Fan and Raquel Urtasun. 2019. Dagmapper: Learning to map by discovering lane topology. See Proceedings of the IEEE / CVF International Conference on Computer Vision. 2911–2920.

[0156]

[23] Zhiyu Huang, Jingda Wu and Chen Lv. 2021. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning. IEEE Transactions on Intelligent Transportation Systems 23, 8 (2021), 10239–10251.

[0157]

[24] Kitae Kim, Soohyun Cho and Woojin Chung. 2021. HD map update for autonomous driving with crowdsourced data. IEEE Letters on Robotics and Automation 6, 2 (2021), 1895–1901.

[0158]

[25] Annkathrin Krämmer, Christoph Schöller, Dhiraj Gulati, Venkatnarayanan Lakshminarasimhan, Franz Kurz, Dominik Rosenbaum, Claus Lenz and Alois Knoll. 2019. Providentia–A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation. arXiv preprint arXiv:1906.06789 (2019).

[0159]

[26] Heon-Cheol Lee, Seung-Hwan Lee, Myoung Hwan Choi and Beom-Hee Lee. 2012. Probabilistic map merging for multi-robot RBPF-SLAM with unknown initial poses. Robotica 30, 2 (2012), 205–220.

[0160]

[27] Qi Li, Yue Wang, Yilun Wang and Hang Zhao. 2022. Hdmapnet: An online HD map construction and evaluation framework. See International Conference on Robotics and Automation 2022. IEEE, 4628–4634.

[0161]

[28] Rong Liu, Jinling Wang and Bingqi Zhang. 2020. High definition map for automated driving: Overview and analysis. Navigation Journal 73, 2 (2020), 324–341.

[0162]

[29] Yicheng Liu, Yue Wang, Yilun Wang and Hang Zhao. 2022. Vectormapnet: End-to-end vectorized HD map learning. arXiv preprint arXiv:2206.08920 (2022).

[0163]

[30] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus and Song Han. 2022. BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation. arXiv preprint arXiv:2205.13542 (2022). Instruction manual 20 / 22 pages 23 CN 122228534 A

[0164]

[31] Livox. 2023. Livox AVIA. https: / / www.livoxtech.com / avia. (2023).

[0165]

[32] Livox. 2023. Livox HAP. https: / / www.livoxtech.com / hap. (2023).

[0166]

[33] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé and Bastian Leibe. 2021. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision 129 (2021), 548–578.

[0167]

[34] Xin Ma, Rui Guo, Yibin Li and Weidong Chen. 2008. Adaptive genetic algorithm for occupancy grid maps merging. See The 7th World Conference on Intelligent Control and Automation, 2008. IEEE, 5716–5720.

[0168]

[35] Maximilian Naumann, Liting Sun, Wei Zhan and Masayoshi Tomizuka. 2020. Analyzing the Suitability of Cost Functions for Explaining and Imitating Human Driving Behavior based on Inverse Reinforcement Learning. See IEEE International Conference on Robotics and Automation 2020. 5481–5487. https: / / doi.org / 10.1109 / ICRA40945.2020.9196795.

[0169]

[36] ASAM OpenDRIVE. 2023. OpenDRIVE Format Specification. https: / / www.asam.net / standards / detail / opendrive / . (2023).

[0170]

[37] Teddy Ort, Liam Paull and Daniela Rus. 2018. Autonomous vehicle navigation in rural environments without detailed prior maps. See IEEE International Conference on Robotics and Automation 2018. IEEE, 2040–2047.

[0171]

[38] David Pannen, Martin Liebner, Wolfgang Hempel and Wolfram Burgard. 2020. How to keep HD maps for automated driving up to date. See IEEE International Conference on Robotics and Automation 2020. IEEE, 2288–2294.

[0172]

[39] Vasyl Pihur, Susmita Datta and Somnath Datta. 2007. Weighted rank aggregation of cluster validation measures: a Monte Carlo cross-entropy approach. Bioinformatics 23, 13 (2007), 1607–1615.

[0173]

[40] Nicholas G Polson and Vadim O Sokolov. 2017. Deep learning for short-term traffic flow prediction. Traffic Research Part C: Emerging Technologies 79 (2017), 1–17.

[0174]

[41] Olaf Ronneberger, Philipp Fischer and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. See Medical Image Computing and Computer-Aided Intervention—MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18. Springer, 234–241.

[0175]

[42] Heiko G Seif and Xiaolong Hu. 2016. Autonomous driving in the iCity—HD maps as a key challenge of the automotive industry. Engineering 2, 2 (2016), 159–162.

[0176]

[43] Tixiao Shan and Brendan Englot. 2018. Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. See 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems. IEEE, 4758–4765.

[0177]

[44] Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu and Zhenchao Ouyang. 2022. VIPS: real-time perception fusion for infrastructure-assisted autonomous driving. See Proceedings of the 28th International Conference on Mobile Computing and Networking. 133–146.

[0178]

[45] Brian L Smith and Michael J Demetsky. 1994. Short-term traffic flow prediction models—a comparison of neural network and nonparametric regression approaches. See Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Vol. 2. IEEE, 1706–1709.

[0179]

[46] HERE Technologies. 2018. HERE HD LiveMap, The Most Intelligent Sensor for Autonomous Driving. https: / / bit.ly / 2Woss4K. (2018).

[0180]

[47] TomTom. 2018. HD Maps - Highly Accurate Border-to-border Model of the Road. https: / / bit.ly / 2WrI1sd. (2018).

[0181]

[48] Manabu Tsukada, Takaharu Oi, Masahiro Kitazawa and Hiroshi Esaki. 2020. Networked roadside perception units for autonomous driving. Sensors 20, 18 (2020), 5320.

[0182]

[49] U-BLOX. 2023. NEO-M8T GPS. https: / / www.u-blox.com / en / product / neole-m8t-series. (2023).

[0183]

[50] Jingke Wang, Yue Wang, Dongkun Zhang, Yezhou Yang and Rong Xiong. 2020. Learning hierarchical behavior and motion planning for autonomous driving. See IEEE / RSJ International Conference on Intelligent Robots and Systems 2020. IEEE, 2235–2242.

[0184]

[51] Xinshuo Weng, Jianren Wang, David Held and Kris Kitani. 2020. Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metrics. arXiv preprint arXiv:2008.08063 (2020).

[0185]

[52] WIT-MOTION. 2023. HWT9052-485 IMU. https: / / www.wit-motion.cn / # / witmotion / product / detail?id=0e24e2a59ac94b14b3034a92e4338b2c. (2023).

[0186]

[53] Ji Zhang and Sanjiv Singh. 2014. LOAM: Lidar odometry and mapping in real-time. See Robotics: Science and Systems, Vol. 2. Berkeley, CA, 1–9.

[0187]

[54] Xun S Zhou and Stergios I Roumeliotis. 2006. Multi-robot SLAM with unknown initial correspondence: The robot rendezvous case. See IEEE / RSJ International Conference on Intelligent Robots and Systems, 2006. IEEE, 1785–1792.Instruction Manual Page 22 / 22 25 CN 122228534 A Figure 1 Figure 2A Figure 2B Instruction Manual Drawing Page 1 / 13 26 CN 122228534 A Figure 3 Figure 4 Instruction Manual Drawing Page 2 / 13 27 CN 122228534 A Figure 5 Instruction Manual Drawing Page 3 / 13 28 CN 122228534 A Figure 6 Instruction Manual Drawing Page 4 / 13 29 CN 122228534 A Figure 7 Instruction Manual Drawing Page 5 / 13 30 CN 122228534 A Figure 8A Figure 8B Instruction Manual Drawing Page 6 / 13 31 CN 122228534 A Figure 8C Instruction Manual Drawing Page 7 / 13 32 CN 122228534 A Figure 9A Figure 9B Instruction Manual Drawing Page 8 / 13 33 CN 122228534 A Figure 10 Figure 11 Instruction Manual Drawing Page 9 / 13 Page 34 CN 122228534 A Figure 12 Figure 13 Appendix 10 / 13 Page 35 CN 122228534 A Figure 14 Appendix 11 / 13 Page 36 CN 122228534 A Figure 15A Figure 15B Appendix 12 / 13 Page 37 CN 122228534 A Figure 16 Figure 17 Appendix 13 / 13 Page 38 CN 122228534 A.

Claims

1. A method for constructing real-time HD maps for infrastructure assistance, comprising: Construct geometry to project data onto the bird's-eye view BEV space; Topology estimation based on trajectory; as well as Map fusion is performed on the vehicle side.

2. The method according to claim 1, wherein, Geometry construction includes performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization.

3. The method according to claim 2, wherein, Performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectory, projecting the point cloud and trajectory onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional 2D mesh.

4. The method according to claim 2, wherein, Performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which the trajectory direction vector occupies two channels.

5. The method according to claim 2, wherein, Performing BEV instance segmentation involves configuring a 2D convolutional neural network (CNN) with a UNet-like structure to perform semantic instance segmentation using feature maps extracted from BEVs.

6. The method according to claim 5, wherein, The 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output.

7. The method according to claim 2, wherein, Performing map vectorization includes: vectorizing the map to generate a sparse and compact representation of the HD map.

8. The method according to claim 1, wherein, Trajectory-based topology estimation includes: establishing and updating the map topology based on the accurate vehicle trajectories and lane instance segmentation results obtained from the constructed geometry.

9. The method according to claim 1, wherein, Map fusion on the vehicle side includes merging the HD map received from the constructed geometry and the topology estimation with the vehicle-mounted HD map.

10. The method according to claim 9, wherein, Map fusion on the vehicle side includes: performing map element mapping, map alignment, and map element refitting.

11. A computer program product, comprising: A non-transitory computer-executable storage device having thereon computer-readable program instructions, which, when executed by a computer, cause the computer to perform a method for infrastructure-assisted real-time HD map construction, the computer-executable program instructions including: Construct geometry to project data onto the bird's-eye view BEV space; Topology estimation based on trajectory; and Map fusion is performed on the vehicle side.

12. The computer program product according to claim 11, wherein, Geometry construction includes performing BEV projection, feature extraction, BEV instance segmentation, and map vectorization.

13. The computer program product according to claim 12, wherein, Performing BEV projection includes: acquiring an accumulated point cloud and vehicle trajectory, projecting the point cloud and trajectory onto a ground plane in the BEV view, and rasterizing the points to generate a two-dimensional 2D mesh.

14. The computer program product according to claim 12, wherein, Performing feature extraction includes: extracting one or more features for each cell, and concatenating the extracted features to form a feature map, in which the trajectory direction vector occupies two channels.

15. The computer program product according to claim 12, wherein, Performing BEV instance segmentation involves configuring a 2D convolutional neural network (CNN) with a UNet-like structure to perform semantic instance segmentation using feature maps extracted from BEVs.

16. The computer program product according to claim 15, wherein, The 2D CNN is configured to take the feature map as input and generate a pixel-wise mask for each individual road element, thereby obtaining an instance mask as output.

17. The computer program product according to claim 12, wherein, Performing map vectorization includes: vectorizing the map to generate a sparse and compact representation of the HD map.

18. The computer program product according to claim 11, wherein, Trajectory-based topology estimation includes: establishing and updating the map topology based on the accurate vehicle trajectories and lane instance segmentation results obtained from the constructed geometry.

19. The computer program product according to claim 11, wherein, The on-vehicle map fusion includes merging the HD map received from the constructed geometry and the topology estimation with the on-vehicle HD map.

20. The computer program product according to claim 19, wherein, Map fusion on the vehicle side includes: performing map element mapping, map alignment, and map element refitting.

Images