Map generation method and equipment

Abstract

The invention discloses a map generation method and equipment. The method comprises the following steps: acquiring equipment information of a camera, a laser radar, a GPS (Global Positioning System) and the like, converting data into a high-precision map by applying an intelligent algorithm, constructing an immersive virtual environment by utilizing three-dimensional reconstruction, and automatically generating a high-precision two-dimensional map and a high-fidelity three-dimensional map. The two-dimensional map provides necessary road topological structures and semantic information, so that the vehicle can understand lane configuration and traffic laws and regulations, and becomes a key basis for safe navigation. The three-dimensional map comprises road environment high-fidelity three-dimensional rendering of terrains, buildings and other infrastructures, and the simulation environment is optimized by enhancing the depth sense and the reality sense, so that the vehicle perception capability and the interaction test with dynamic elements such as pedestrians and other vehicles are more comprehensive, and the driving environment is reproduced truly. The double-layer simulation capability significantly improves the testing and development efficiency of the automatic driving technology, and promotes the development of a safer and more reliable automatic driving system.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a method for generating high-precision, high-fidelity maps suitable for autonomous driving. Background Technology

[0002] Existing map generation methods often struggle to accurately capture complex road details and real-world environmental features, elements crucial for effective simulation and navigation. Technologies described in patents such as CN117635850B and CN113834492B primarily rely on single-sensor data (LiDAR or images), resulting in incomplete or unreliable map outputs due to a lack of data integration. These systems typically fail to integrate multimodal data, a key element for achieving the high accuracy and fidelity required for autonomous driving applications. This reliance on single-sensor data limits the system's ability to capture the complexity of real-world environments, leading to gaps in the mapping process and incomplete or unreliable map outputs. Furthermore, most existing methods only provide two-dimensional map representations, lacking real-time 3D scene reconstruction capabilities. This limitation significantly weakens their ability to construct immersive, realistic simulation environments, essential for the comprehensive testing and validation of autonomous driving systems.

[0003] Therefore, there is a need to provide a map generation method that can accurately capture complex road details and real-world environmental features. Summary of the Invention

[0004] The objective of this invention is to provide a map generation method that can simulate a detailed and accurate scene representation of the environment, generating an immersive and realistic simulation environment, thereby making it suitable for testing autonomous systems such as self-driving cars, drones, and robots in a virtual environment.

[0005] In a first aspect of the present invention, a map generation method is provided, the method comprising:

[0006] Acquire multi-view camera images, lidar point clouds, and inertial measurement unit data;

[0007] Feature extraction is performed on multi-view camera images to obtain multi-view camera image feature maps;

[0008] Convert multi-view camera image feature maps into bird's-eye view image features;

[0009] Divide the LiDAR point cloud into vertical cylinders and calculate the statistical features of each cylinder to obtain the LiDAR features in a bird's-eye view of each cylinder.

[0010] The bird's-eye view image features and the bird's-eye view lidar features are stitched together to obtain fused bird's-eye view features;

[0011] The fused bird's-eye view features are detected end-to-end using a DETR-like map element detector and labeled using binary matching to obtain learnable element query vectors and learnable key point query vectors. Each learnable element query vector corresponds to a map element, including road boundaries, lane dividers, and zebra crossings. The learnable key point query vectors correspond to the key point locations and geometric shapes of the map elements.

[0012] The learnable element query vector is used as input to the decoder to obtain an instance representation of each map element;

[0013] The learnable keypoint query vector is used as input to the decoder to obtain the keypoint representation of each map element;

[0014] Based on the instance representation and key point representation of each map element, the key points are sorted and connected to generate an ordered polyline.

[0015] Coordinate embedding, position embedding, and value embedding are performed on the ordered polyline to obtain a high-precision map polyline;

[0016] Calculate the geometric features of each point in the lidar point cloud, the geometric features including the distance from each point to its voxel plane and the angle between the point's normal vector and the voxel plane's normal vector;

[0017] Based on pixel blocks from different frames of multi-view camera images and geometric features from point clouds, pixel blocks are associated with LiDAR points to form textured point clouds.

[0018] Convert textured point clouds into COLMAP databases in COLMAP database format;

[0019] Based on the COLMAP database, neural rendering was performed using 3D Gaussian sputtering technology to obtain a 3D reconstructed model.

[0020] By aligning the positions in the polylines of the high-precision map with the positions in the 3D reconstruction model using inertial measurement unit data, an aligned high-precision map is obtained.

[0021] Post-processing is performed on the aligned high-precision map to obtain the final vectorized high-precision map and high-fidelity 3D reconstructed map.

[0022] In a second aspect of the invention, a computer device is provided, including a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0023] This invention integrates advanced technologies of 2D map generation and 3D reconstruction to create immersive and realistic simulation environments. Such environments are invaluable for training and validating autonomous systems such as self-driving vehicles, drones, and robots under diverse and complex conditions. By integrating cutting-edge technologies, this invention ensures that the generated maps are not only visually accurate but also contain rich data, thereby enabling precise navigation and interaction within the simulated space. It significantly improves safety by reducing high-risk real-world testing requirements while simultaneously reducing physical testing costs, and can be widely applied in fields such as urban planning and navigation.

[0024] This invention acquires information from devices such as cameras, LiDAR, and GPS, and uses intelligent algorithms to transform the data into high-precision maps. It then utilizes 3D reconstruction to construct an immersive virtual environment, automatically generating both 2D and 3D maps to provide a more comprehensive and accurate solution for autonomous driving platforms. The high-precision (HD) 2D maps generated by this invention include lane-level detail information, such as lane boundaries, medians, and road markings. These maps provide accurate road layouts and traffic rules, offering necessary road topology and semantic information, enabling vehicles to understand lane configurations and traffic regulations, thus becoming a crucial foundation for safe navigation in autonomous driving systems. In addition to the 2D maps, a high-fidelity 3D rendering model of the road environment, including terrain, buildings, and other infrastructure, is also generated. These 3D reconstruction models optimize the simulation environment by enhancing depth and realism, making vehicle perception capabilities and interactions with dynamic elements such as pedestrians and other vehicles more comprehensive and realistically reproducing the driving environment. This is crucial for simulating the physical and visual characteristics of real-world scenarios. This dual-layer simulation capability significantly improves the testing and development efficiency of autonomous driving technology, ultimately driving the development of safer and more reliable autonomous driving systems.

[0025] This invention allows users to create realistic test environments according to specific needs and ensures that the map is synchronized with the actual environment through real-time updates.

[0026] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0027] Figure 1 A flowchart of an embodiment of the method according to the present invention;

[0028] Figure 2 The actual street view image captured by the camera;

[0029] Figure 3 for Figure 2 Google 3D map of the location shown;

[0030] Figure 4 To obtain according to the method of the present invention Figure 2High-fidelity neural rendering of the 3D reconstructed map of the location shown.

[0031] For clarity, these figures are schematic and simplified, showing only the details necessary for understanding the invention, while omitting other details. Detailed Implementation

[0032] The embodiments and examples of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] The scope of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific examples illustrate preferred embodiments of the invention, they are given for illustrative purposes only.

[0034] Figure 1 A flowchart of an embodiment of the map generation method according to the present invention is shown.

[0035] In step S1, sensor data such as multi-view camera images, LiDAR point clouds, and inertial measurement unit (IMU) data are acquired. In other embodiments, Global Navigation Satellite System (GNSS) data may also be acquired. After acquiring this data, hardware time synchronization is used to align the data, thereby obtaining a synchronized multimodal data stream. Simultaneously, the multi-view camera images can be reprocessed, such as distortion correction, cropping, and scaling, and the LiDAR point clouds can be motion compensated to achieve distortion correction.

[0036] In step S2, feature extraction is performed on the multi-view camera images to obtain multi-view camera image feature maps. Pre-trained convolutional neural networks (CNNs), such as ResNet-18 or ResNet-50 models, can be used to extract features such as edges, corners, color blocks, and basic geometric patterns from the multi-view camera images to obtain multi-view camera image feature maps.

[0037] In step S3, the multi-view camera image feature map is converted into bird's-eye view image features. Through learnable priors or geometric priors, the multi-view camera image features are first projected into three-dimensional space, and then further projected onto the bird's-eye view (BEV) grid to obtain the bird's-eye view image features, which have dimensions of W×H×C1, where W represents width, H represents height, and C1 represents the number of feature channels.

[0038] In step S4, the LiDAR point cloud is divided into vertical cylinders, and the statistical features of each cylinder, such as mean and density, are calculated to obtain the cylinder-by-cylinder bird's-eye view LiDAR features, with dimensions of W×H×C2, where W represents width, H represents height, and C2 represents the number of feature channels. The calculation process of the statistical features of the cylinders involves cylinderization, feature enhancement, feature extraction and pooling, and mapping to the two-dimensional bird's-eye view (BEV) feature map.

[0039] In step S5, the bird's-eye view image features and the bird's-eye view LiDAR features are stitched together using a multilayer perceptron (MLP) or a convolutional layer to obtain fused bird's-eye view features. The dimensions of the fused bird's-eye view features are W×H×(C1+C2).

[0040] In step S6, a DETR-like map element detector is used to perform end-to-end detection on the fused bird's-eye view features and label assignment is performed using binary matching (such as the Hungarian algorithm). This allows the location and category of map elements to be represented by predicting key points and category labels from the bird's-eye view features, effectively modeling the shape of map elements and obtaining learnable element query vectors and learnable key point query vectors. Each learnable element query vector corresponds to a map element such as a road boundary, lane divider, or zebra crossing. The learnable key point query vector uses the positions of k key points to represent the outline of a map element, corresponding to the key point positions and geometric shape of the map element.

[0041] In step S7, the learnable element query vector is used as input to the decoder to obtain an instance representation of each map element.

[0042] In step S8, the learnable keypoint query vector is used as input to the decoder to obtain the keypoint representation of each map element.

[0043] In step S9, based on the instance representation and keypoint representation of each map element, the keypoints are sorted and connected to generate an ordered (vectorized) polyline. This process includes input and instance filtering, keypoint sorting, polyline connection and geometric simplification, and structured output generation. Simultaneously, geometric optimization methods such as Bezier curve fitting can be used to smooth the polyline.

[0044] In step S10, coordinate embedding, position embedding, and value embedding are performed on the ordered polyline to obtain a high-precision map polyline. Coordinate embedding is used to identify whether the vertex token represents the x-coordinate or the y-coordinate. Position embedding is used to indicate which vertex in the sequence the token belongs to. Value embedding is used to represent the coordinate value of the token after quantization.

[0045] In step S11, the LiDAR point cloud is voxelized and local plane fitted using a voxel map. The geometric features of each point in the LiDAR point cloud are calculated, such as the distance from each point to its voxel plane and the angle between the point's normal vector and the voxel plane's normal vector, capturing the local geometric structure. The distance from a point to its voxel plane can be calculated by constructing a vector from the plane point to the LiDAR point, calculating the dot product of this vector and the plane's normal vector, and taking the absolute value of the dot product (i.e., the distance from the point to its voxel plane). The angle between the point's normal vector and the voxel plane's normal vector can be calculated by calculating the dot product of the point's normal vector and the plane's normal vector, and taking the inverse cosine of the dot product to obtain the angle between the two normal vectors.

[0046] In step S12, pixel blocks (including image features such as frame global position and frame field of view) from different frames of multi-view camera images and geometric features from point clouds are associated with LiDAR points to form textured point clouds.

[0047] In step S13, the textured point cloud is converted into a COLMAP database in COLMAP database format, including cameras.bin, images.bin, and points3D.bin. The conversion includes: 1) converting the points to the COLMAP world coordinate system using the estimated camera pose, camera intrinsics, and textured point cloud; 2) generating the cameras.bin file by saving the intrinsics of all cameras; 3) generating the images.bin file by converting the camera pose to the COLMAP coordinate system and converting the rotations to quaternions; 4) generating the points3D.bin file by transforming and projecting the 3D points to find the image observations.

[0048] In step S14, based on the COLMAP database, neural rendering is performed using 3D Gaussian sputtering technology to obtain a 3D reconstructed model. To generate a high-fidelity 3D Gaussian sputtering (3DGS) model, dense textured point clouds and precise tightly coupled poses are required to achieve more detailed reconstruction results and obtain clearer textures and geometric structures.

[0049] In step S15, the positions in the high-precision map polylines are aligned with the positions in the 3D reconstruction model using inertial measurement unit (IMU) data, resulting in an aligned high-precision map. When simultaneously acquiring Global Navigation Satellite System (GNSS) data, the positions in the high-precision map polylines can be aligned with the positions in the 3D reconstruction model using both IMU and GNSS data. Specifically, initial pose estimation can be performed using IMU and / or GNSS data, followed by a Fast Iterative Closest Point (ICP) or graph optimization method to align the 3D reconstruction scene with the high-precision map, achieving accurate real-time alignment. Graph-based methods may run at a lower frequency (e.g., every few seconds) to correct any residual drift that occurs in the ICP-based processing flow.

[0050] In step S16, the aligned high-precision map is post-processed, such as denoising, smoothing, and semantic annotation, to obtain the final vectorized high-precision map and the high-fidelity 3D reconstructed Gaussian model.

[0051] Real-time identification of temporary obstacles, road construction, weather conditions, and other environmental changes, and the creation of corresponding models, are indispensable safety requirements for autonomous vehicles. The core advantage of the 3D reconstruction technology in this invention lies in its self-centered and highly geometrically accurate characteristics, a feature specifically designed for the real-time decision-making needs of autonomous vehicles. This invention employs the principle of 3D Gaussian sputtering (3DGS), enabling high-precision, real-time generation of photorealistic 3D scenes suitable for dynamic environments in autonomous driving simulations. The 3D scene reconstruction method of this invention can generate dense, colored 3D point clouds of the entire environment. This capability is crucial for tasks such as obstacle avoidance (e.g., identifying pedestrians behind bushes) and refined path planning that does not rely entirely on lane markings.

[0052] This invention integrates LiDAR point clouds, inertial measurement unit (IMU) data, and camera pixels to improve accuracy and robustness. The IMU provides high-frequency motion prior information, the LiDAR provides precise geometric constraints, and the camera provides dense texture constraints. This tight coupling generates robust and accurate pose estimation, ensuring system operation even if a single sensor fails. The method employs a unified voxel map to manage LiDAR and visual measurement data, aiding in the construction of geometric structures and achieving precise image alignment, thus ensuring perfect synchronization between geometric and appearance information. Each texture patch is intrinsically linked to its precise 3D position, which is crucial for generating coherent color point clouds and realistic reconstruction results.

[0053] This invention is specifically designed for autonomous driving systems, where high-precision maps and high-fidelity 3D reconstructed maps are crucial for real-time navigation, simulation, and environmental perception. Currently available consumer-oriented maps optimize their 2D and 3D views for human readability and geospatial navigation, rather than for machine perception or autonomous vehicle path planning. They provide a generic representation of roads, buildings, and terrain, failing to meet the centimeter-level accuracy required for autonomous driving. In contrast, this invention focuses on the semantic and geometric accuracy of map elements (such as lane lines, road boundaries, and traffic signs), while consumer-grade maps typically do not provide element-by-element vectorization or detailed spatial relationship information.

[0054] This invention employs multi-sensor fusion, bird's-eye view feature conversion, vectorized polyline generation, and 3D Gaussian sputtering neural rendering technology to generate vectorized high-precision maps, 3D textured meshes, and 3DGS models specifically designed for autonomous driving and simulation.

[0055] - Vectorized high-precision maps contain semantic elements (lane lines, zebra crossings, traffic signs, etc.), which are presented as polylines with attributes (such as type, curvature, and length).

[0056] - 3D textured meshes can restore the geometric and visual fidelity of the environment;

[0057] - The neural rendering capabilities of the 3DGS model enable high-fidelity real-time 3D visualization, ensuring consistent camera projection.

[0058] Figure 2 The image shows an actual street view captured by the camera. Figure 3 It shows Figure 2 Google 3D map of the location shown. Figure 4 The method obtained according to the present invention is shown Figure 2 A high-fidelity neural rendering of the 3D reconstructed map at the indicated location. It can be seen that... Figure 3 The existing 3D maps lack fine details and are unsuitable for or cannot be used for autonomous driving. However, the 3D reconstructed maps obtained by the method of this invention contain a large amount of fine details and have a high fidelity compared with actual street view images, making them suitable for autonomous driving.

[0059] In another embodiment, a computer device is provided, including a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0060] In another embodiment, a computer-readable storage medium is provided that stores a computer program / instructions thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described above.

[0061] In another embodiment, a computer program product is provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described above.

[0062] The various embodiments described herein, or their specific features, structures, or characteristics, may be suitably combined in one or more embodiments of the invention. Furthermore, in some cases, the order of steps described in the flowcharts and / or pipeline processes may be modified where appropriate, and they need not be performed in the exact order described. Additionally, various aspects of the invention may be implemented using software, hardware, firmware, or combinations thereof, and / or other computer-implemented modules or devices that perform the described functions. Software implementations of the invention may include executable code stored in a computer-readable medium and executed by one or more processors. Computer-readable media may include computer hard disk drives, ROM, RAM, flash memory, portable computer storage media such as CD-ROM, DVD-ROM, flash drives, and / or other devices having a Universal Serial Bus (USB) interface, and / or any other suitable tangible or non-transitory computer-readable medium or computer memory on which executable code can be stored and executed by a processor. The invention may be used in conjunction with any suitable operating system.

[0063] Unless explicitly stated otherwise, the singular forms “a” and “the” used herein include the plural meaning (i.e., meaning “at least one”). It should be further understood that the terms “having,” “comprising,” and / or “including” as used in the specification indicate the presence of the described features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The term “and / or” as used herein includes any and all combinations of one or more of the listed related items.

[0064] The foregoing has described some preferred embodiments of the present invention. However, it should be emphasized that the present invention is not limited to these embodiments, but can be implemented in other ways within the scope of the present invention. Those skilled in the art can make various modifications and variations to the present invention based on the inventive concept and without departing from the scope of the present invention, and such modifications or variations still fall within the protection scope of the present invention.

Claims

1. A map generation method, characterized in that, The method includes: Acquire multi-view camera images, lidar point clouds, and inertial measurement unit data; Feature extraction is performed on multi-view camera images to obtain multi-view camera image feature maps; Convert multi-view camera image feature maps into bird's-eye view image features; Divide the LiDAR point cloud into vertical cylinders and calculate the statistical features of each cylinder to obtain the LiDAR features in a bird's-eye view of each cylinder. The bird's-eye view image features and the bird's-eye view lidar features are stitched together to obtain fused bird's-eye view features; The fused bird's-eye view features are detected end-to-end using a DETR-like map element detector and labeled using binary matching to obtain learnable element query vectors and learnable key point query vectors. Each learnable element query vector corresponds to a map element, including road boundaries, lane dividers, and zebra crossings. The learnable key point query vectors correspond to the key point locations and geometric shapes of the map elements. The learnable element query vector is used as input to the decoder to obtain an instance representation of each map element; The learnable keypoint query vector is used as input to the decoder to obtain the keypoint representation of each map element; Based on the instance representation and key point representation of each map element, the key points are sorted and connected to generate an ordered polyline. Coordinate embedding, position embedding, and value embedding are performed on the ordered polyline to obtain a high-precision map polyline; Calculate the geometric features of each point in the lidar point cloud, the geometric features including the distance from each point to its voxel plane and the angle between the point's normal vector and the voxel plane's normal vector; Based on pixel blocks from different frames of multi-view camera images and geometric features from point clouds, pixel blocks are associated with LiDAR points to form textured point clouds. Convert textured point clouds into COLMAP databases in COLMAP database format; Based on the COLMAP database, neural rendering was performed using 3D Gaussian sputtering technology to obtain a 3D reconstructed model. By aligning the positions in the polylines of the high-precision map with the positions in the 3D reconstruction model using inertial measurement unit data, an aligned high-precision map is obtained. Post-processing is performed on the aligned high-precision map to obtain the final vectorized high-precision map and high-fidelity 3D reconstructed map.

2. The method according to claim 1, characterized in that, The alignment involves aligning the positions in the high-precision map polylines with the positions in the 3D reconstruction model using inertial measurement unit data and acquired global navigation satellite system data.

3. The method according to claim 1, characterized in that, The method further includes aligning multi-view camera images, lidar point clouds, inertial measurement unit data, and global navigation satellite system data using hardware time synchronization.

4. The method according to claim 1, characterized in that, The process of converting multi-view camera image feature maps into bird's-eye view image features includes: Multi-view camera image features are projected into three-dimensional space using learnable or geometric priors; The multi-view camera image features of the 3D spatial projection are further projected onto the bird's-eye view grid to obtain the bird's-eye view image features.

5. The method according to claim 1, characterized in that, The distance from the point to its voxel plane is calculated using the following steps: - Construct vectors from planar points to LiDAR points; - Calculate the dot product of the constructed vector and the plane normal vector; - The absolute value of the dot product is the distance from the point to its voxel plane.

6. The method according to claim 1, characterized in that, The angle between the point normal vector and the voxel plane normal vector is calculated using the following steps: - Calculate the dot product of the point normal vector and the plane normal vector; - Taking the inverse cosine of the calculated dot product yields the angle between the two normal vectors.

7. The method according to claim 1, characterized in that, The post-processing includes one or more of the following: denoising, smoothing, and semantic annotation.

8. A computer device, comprising a processor, a memory, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

Images