Methods and systems for generating 3D maps by combining 3D LiDAR with 2D mapping

Abstract

This invention discloses a method and system for generating 3D maps by combining 3D LiDAR with 2D mapping. Using this method, when the online 3D mapping on-vehicle platform is ineffective, 2D laser data compressed into a plane is used for offline 2D mapping. The 2D pose graph structure generated from the 2D mapping is then interpolated with the 3D pose graph structure from the online 3D mapping to generate a new 3D pose graph structure. This solves the problems of insufficient real-time performance and poor quality in online 3D mapping on the vehicle-mounted platform. Furthermore, since 2D laser data is much lighter than 3D laser data, downloading laser data from the vehicle-mounted platform to the local host computer is easier, making image editing on the host computer more convenient. Using derived point technology, it is even possible to abandon 3D mapping altogether and obtain high-quality 3D maps using only 2D mapping for mobile robot localization.

Description

Technical Field

[0001] This invention relates to the field of SLAM for mobile robots, specifically to a method and system for generating 3D maps by combining 3D LiDAR with 2D mapping. Background Technology

[0002] In the existing SLAM framework, mapping, localization, and navigation are several critical factors in mobile robots. Among them, the quality of mapping directly determines whether the mobile robot can operate stably, reliably, and safely in the map scene.

[0003] Typical SLAM processes utilize sensors including 2D lasers, 3D lasers, and depth cameras. In the SLAM industry, 3D lasers are used for mapping in large or complex scenarios such as AGVs, large supermarkets, hotels, and parking garages. However, complex scenes rarely allow for successful mapping on the first attempt. To address this issue, besides algorithm optimization, there are some project-specific solutions. These include breaking down large maps into smaller maps and then stitching them together, or manually modifying the pose of 3D laser keyframes to adjust the map structure. However, these methods have their drawbacks: stitching maps is time-consuming and cumbersome to implement. Map retouching methods require downloading 3D laser data from the vehicle to a host computer (download speeds are extremely slow). Typically, large scenes like AGVs contain several gigabytes of 3D data, making downloading very difficult. Furthermore, processing 3D data during 3D mapping requires significant time and CPU resources, making real-time 3D mapping on the vehicle difficult to guarantee, potentially resulting in poor mapping quality and making retouching extremely challenging. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for generating 3D maps by combining 3D LiDAR with 2D mapping, so as to solve the above-mentioned technical problems existing in related technologies.

[0005] According to a first aspect of the embodiments of this application, a method for generating a 3D map by combining 3D LiDAR with 2D mapping is provided, applied to a host computer, including:

[0006] The first 3D pose graph structure and 2D composition data are obtained. The first 3D pose graph structure and 2D composition data are obtained by acquiring scene observation data from 3D LiDAR, Odom odometry and IMU on the vehicle terminal, performing 3D real-time composition and converting 3D data into 2D data format. The first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp.

[0007] Based on the 2D mapping data, a 2D mapping is performed to generate a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp.

[0008] Interpolation is performed on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​of the timestamps corresponding to the 2D laser keyframes and 3D laser keyframes.

[0009] Based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining the 3D map.

[0010] Further, based on the Odom odometry data in the 2D mapping data, interpolation is performed to obtain the Odom odometry recursive values ​​for the corresponding timestamps of the 2D laser keyframes and 3D laser keyframes, including:

[0011] The Odom odometry data, 2D laser keyframes, and 3D laser keyframes in the 2D mapping data are arranged by timestamp size;

[0012] Interpolation is performed based on the recursive pose from the Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframes and 3D laser keyframes:

[0013]

[0014] Where: X 3D and X 2D The X-axis and Y-axis values ​​of the odometry recursive pose are interpolated from the timestamps of 3D and 2D laser keyframes, respectively. 3D and Y 2D These are the Y-direction values ​​for the interpolated pose, and θ is the value for the interpolated pose. 3D and θ 2D These represent the angular direction values ​​of the interpolated pose, t. 3D and t 2D These are the timestamps for the 3D laser keyframe and the 2D laser keyframe, respectively. X Odom Y Odom θ Odom and t Odom These are the X-direction value, Y-direction value, angle direction value, and corresponding timestamp of the recursive pose in the Odom odometry data.

[0015] Furthermore, if the change in adjacent Odom odometer data used for interpolation exceeds a predetermined threshold, the Odom odometer recursive value obtained by interpolating the adjacent Odom odometer data is deleted.

[0016] Furthermore, based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, including:

[0017] Based on the timestamps of the 3D laser keyframes, find the timestamp of the most recent 2D laser keyframe and calculate the odometry movement between these two timestamps:

[0018] X 2t3 =X 3D -1 *X 2D

[0019] Y 2t3 =Y 3D -1 *Y 2D

[0020] - 1

[0021] θ 2t3 =θ 3D *θ 2D

[0022] Where: X 3D and X 2D The X-direction values ​​of the odometry recursive pose are interpolated using the timestamps of 3D and 2D laser keyframes, respectively; the Y-direction and θ-direction values ​​are calculated similarly. 2t3 The interval between the 2D laser and 3D laser timestamps corresponds to the value of the Odom odometry pose change in the X direction, which is the recursive pose change value in the X direction obtained by interpolation. The same applies to the Y and θ directions.

[0023] The second pose in the world coordinate system corresponding to the 3D laser timestamp is calculated using the 3D LiDAR extrinsic parameters, 2D LiDAR extrinsic parameters, Odom odometry pose change, and the pose in the world coordinate system corresponding to the timestamp of the 2D laser keyframe in the 2D pose diagram.

[0024] X 3tW =X 2tW *X Ot2 *X 2t3 *X 3tO

[0025] Y 3tW =Y 2tW *Y Ot2 *Y 2t3 *Y 3tO

[0026] θ 3tW =θ 2tW *θ Ot2*θ 2t3 *θ 3tO

[0027] Where: X 3tW Let X be the X component of the second pose in the world coordinate system. 2tW X is the X component of the 2D laser pose in the world coordinate system corresponding to the timestamp of the 2D laser keyframe. Ot2 X is the inverse of the X component of the extrinsic parameters of a 2D lidar. 3tO The X component represents the extrinsic parameters of the 3D lidar; the Y and θ directions are similar.

[0028] According to a second aspect of the embodiments of this application, a method for generating a 3D map by combining 3D LiDAR with 2D mapping is provided, applied to an in-vehicle terminal, including:

[0029] Acquire scene observation data from 3D LiDAR, Odom odometry and IMU, perform 3D real-time mapping and convert 3D data into 2D data format to generate a first 3D pose graph structure and 2D mapping data, wherein the first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp.

[0030] The first 3D pose graph structure and 2D mapping data are uploaded to the host computer, so that the host computer can perform 2D mapping based on the 2D mapping data to generate a 2D pose graph. The 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp. Based on the Odom odometry data in the 2D mapping data, interpolation is performed to obtain the Odom odometry recursive values ​​of the timestamps of the 2D and 3D laser keyframes. Based on the calculated Odom odometry recursive values ​​and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining a 3D map.

[0031] According to a third aspect of the embodiments of this application, a system for generating 3D maps by combining 3D LiDAR with 2D mapping is provided, including vehicle-mounted sensors, vehicle-mounted terminal and host computer, wherein the vehicle-mounted sensors include 3D LiDAR, Odom odometer and IMU;

[0032] The vehicle-mounted terminal is used to acquire scene observation data from 3D LiDAR, Odom odometry and IMU, perform 3D real-time mapping and convert 3D data into 2D data format, generate a first 3D pose graph structure and 2D mapping data and upload them to the host computer. The first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp.

[0033] The host computer is used to perform 2D mapping based on the 2D mapping data, generating a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp; interpolation is performed based on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​of the timestamps of the 2D laser keyframe and the 3D laser keyframe; based on the calculated Odom odometry recursive values ​​and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining a 3D map.

[0034] According to a fourth aspect of the embodiments of this application, a method for generating a 3D map by combining 3D LiDAR with 2D mapping is provided, applied to a host computer, including:

[0035] A 2D pose graph structure is obtained, wherein the 2D pose graph structure includes the 2D laser keyframe pose in the world coordinate system and the corresponding timestamp. The observation data of the scene obtained by the vehicle-mounted terminal from the 3D LiDAR, 2D LiDAR, and Odom odometry are used to perform real-time 2D laser mapping.

[0036] Interpolation is performed based on the 2D laser keyframe pose, 3D lidar observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframe and the 3D laser observation frame.

[0037] Based on the calculated Odom odometry recursive value and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamps, thereby obtaining a 3D map.

[0038] According to a fifth aspect of the embodiments of this application, a method for generating a 3D map by combining 3D LiDAR with 2D mapping is provided, applied to an in-vehicle terminal, including:

[0039] Acquire scene observation data from 3D LiDAR, 2D LiDAR, and Odom odometry, perform real-time 2D laser mapping, and generate a 2D pose graph structure, wherein the 2D pose graph structure includes 2D laser keyframe poses in the world coordinate system and the corresponding timestamps.

[0040] The 2D pose graph structure is uploaded to the host computer, which then interpolates the 2D laser keyframe pose, 3D LiDAR observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​for the timestamps of the 2D laser keyframe and 3D laser observation frame. Based on the calculated Odom odometry recursive values ​​and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure using the corresponding timestamps, thereby obtaining a 3D map.

[0041] According to a sixth aspect of the embodiments of this application, a system for generating a 3D map by combining 3D LiDAR with 2D mapping is provided, including vehicle-mounted sensors, a vehicle-mounted terminal, and a host computer. The vehicle-mounted sensors include 3D LiDAR, 2D LiDAR, Odom odometer, and IMU. The process of generating the 3D map by this system is as follows:

[0042] (1) The vehicle-mounted terminal acquires the scene observation data of 3D LiDAR, 2D LiDAR and Odom odometry, performs 2D laser real-time mapping, generates a 2D pose graph structure and uploads it to the host computer. The 2D pose graph structure includes the 2D laser key frame pose in the world coordinate system and the corresponding timestamp.

[0043] (2) The host computer interpolates the 2D laser keyframe pose, 3D laser radar observation data, and Odom odometry data to obtain the Odom odometry recursive value of the timestamp corresponding to the 2D laser keyframe and 3D laser observation frame; based on the calculated Odom odometry recursive value and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamp to obtain a 3D map.

[0044] According to a seventh aspect of the embodiments of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the methods described in the first aspect, second aspect, fourth aspect, or fifth aspect.

[0045] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0046] This application provides a solution that uses 3D LiDAR instead of 2D LiDAR. When the 3D online vehicle-mounted mapping effect is unsatisfactory, 2D laser data compressed into a plane by 3D laser is used for 2D offline mapping. The 2D pose graph structure generated by the 2D mapping is then interpolated with the 3D pose graph structure of the online 3D mapping to generate a new 3D pose graph structure. This solves the problem that online 3D mapping on the vehicle side consumes a lot of time and CPU resources to process 3D data, resulting in insufficient real-time performance and poor 3D pose graph structure. This improves the quality of the mapping. Furthermore, since 2D laser data (hundreds of megabytes in AGV scenes) is much lighter than 3D laser data (tens of gigabytes in AGV scenes), downloading laser data from the vehicle to the local host computer (where the network is usually very poor) is much easier. Therefore, image editing operations on the host computer (manually changing the pose of laser nodes in the pose graph to improve the quality of the mapping) are more convenient.

[0047] This application also provides a solution that combines 3D LiDAR with 2D LiDAR, eliminating the need for 3D mapping using 3D laser data, thus reducing one mapping process. Since there is no 3D mapping, there is no need to port the 3D pose graph structure from the vehicle-mounted device to the PC, reducing workload and increasing efficiency. Similar to the previous solution, this solution also requires using pose interpolation to generate a new pose graph structure from the existing one. However, while the previous solution used a 2D pose graph structure plus a first 3D pose graph structure to generate a second 3D pose graph structure, this solution uses a 2D pose graph structure plus the original 3D laser information to generate the 3D pose graph structure.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 A schematic diagram of a system for generating 3D maps by combining 3D lidar with 2D mapping;

[0051] Figure 2 This is a schematic diagram of the sensor timestamp.

[0052] Figure 3 This is a schematic diagram of sensor filtering.

[0053] Figure 4 This is a schematic diagram of the pose change of the Odom odometer.

[0054] Figure 5 This is a sensor pose transformation diagram;

[0055] Figure 6 This is a schematic diagram of a system (including 2D lidar) that combines 3D lidar with 2D mapping to generate 3D maps;

[0056] Figure 7 This is a schematic diagram of an electronic device. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0058] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0060] This application provides as follows: Figure 1 The system shown combines 3D LiDAR with 2D mapping to generate a 3D map. The system includes vehicle-mounted sensors, a vehicle-mounted terminal, and a host computer. The vehicle-mounted sensors include a 3D LiDAR, an Odom odometer, and an IMU. The process of generating the 3D map using this system is as follows:

[0061] (1) The vehicle-mounted terminal acquires the scene observation data of 3D LiDAR, Odom odometry and IMU, performs 3D real-time mapping and converts the 3D data into 2D data format, generates the first 3D pose graph structure and 2D mapping data and uploads them to the host computer. The first 3D pose graph structure includes the first pose of the 3D laser key frame in the world coordinate system and the corresponding timestamp.

[0062] Specifically, a 3D LiDAR, Odom odometry, and IMU are used to observe the scene and perform real-time 3D online mapping on the vehicle to generate a first 3D pose graph structure (including the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp). Simultaneously with the online 3D mapping, the 3D laser point cloud data is converted into 2D mapping data by selecting appropriate line beams or heights. For example, a 16-line 3D laser uses a line beam of 0°-2° as the 2D laser point cloud, while the MId360 3D laser selects laser points within 5cm above and below the laser height as the 2D laser point cloud. This process yields the 2D mapping data converted from the 3D laser data.

[0063] The generation process of the first 3D pose graph structure is as follows:

[0064] The first pose (6 degrees of freedom) of the 3D laser keyframe in the world coordinate system is filled into the pose3d flag in the order of X, Y, Z, Yaw angle, Pitch angle, and Roll angle. The corresponding 3D laser point cloud data of the laser frame is saved in a specific bin file, which is filled into the laser3d_file_name flag.

[0065] Transmit the 3D pose graph structure (typically several hundred kilobytes for complex scenes), 2D mapping data (typically several hundred megabytes for complex scenes), and the extrinsic parameter configuration of the 3D LiDAR (several tens of kilobytes) to the host computer.

[0066] (2) The host computer performs 2D mapping based on the 2D mapping data to generate a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp; interpolation is performed based on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive value of the timestamp corresponding to the 2D laser keyframe and the 3D laser keyframe; based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining the 3D map;

[0067] Specifically, the four parameters of X, Y, Z, and Yaw angles in the 3D laser extrinsic parameters are used to simulate a virtual 2D laser extrinsic parameter (for convenience, the 2D laser mentioned below refers to this virtual 2D laser, since no real 2D laser is used here). Using this 2D laser extrinsic parameter and 2D mapping data, a 2D mapping is performed on the host computer to generate a 2D pose graph.

[0068] By utilizing the fact that the robot's motion in the Odom odometry coordinate system approximates its motion in the world coordinate system within a short local time, Odom odometry is used to interpolate laser data. The timestamps of the Odom odometry data, 2D laser keyframes, and 3D laser keyframes in the 2D mapping data are arranged by timestamp size, such as... Figure 2 As shown: Laser3D corresponds to the 3D laser timestamp, Laser2D corresponds to the 2D laser timestamp, and Odom corresponds to the timestamp and odom recursive pose of the Odom odom (the Odom odom frequency is generally 50Hz, the laser frequency is generally 10Hz, and the lasers here are all keyframes, so the distribution of the Odom odom is obviously more dense than that of the lasers).

[0069] Interpolate the timestamps of Laser3D and Laser2D to obtain the Odom odometry recursive values ​​corresponding to the Laser3D and Laser2D timestamps. The specific interpolation method is shown in the following formula:

[0070]

[0071]

[0072] Where: X 3D and X 2D The X-axis and Y-axis values ​​of the odometry recursive pose are interpolated from the timestamps of 3D and 2D laser keyframes, respectively. 3D and Y 2D These are the Y-direction values ​​for the interpolated pose, and θ is the value for the interpolated pose. 3D and θ 2D These represent the angle and direction values ​​of the interpolated pose. The interpolated poses here are all located in the Odom odometry coordinate system, i.e., the odometry recursive pose, and not the pose in PoseGraph (in world coordinates) mentioned earlier. This distinction is important to note. 3D and t 2D These are the timestamps for the 3D laser keyframe and the 2D laser keyframe, respectively. X Odom Y Odom θ Odom and t Odom These are the X-direction value, Y-direction value, and angle direction value of the recursive pose in the Odom odometry data, along with the corresponding timestamp. The subscripts Odom1 to Odom4 indicate adjacent odometry timestamps.

[0073] In practical implementation, when the Odom odometer moves at a low speed (i.e., the trolley moves slowly) and the odometer timestamp interval used for interpolation is short, the principle that the odometer's movement is uniform is satisfied. Therefore, the recursive pose data interpolated by the interpolation method can be considered the accurate data for that timestamp. It is then necessary to filter out those adjacent Odom odometer data that are faster and used for interpolation, as well as the laser sensor interpolated pose obtained from these two adjacent Odom odometer data, so that the pose obtained by the interpolation method can replace the true value. Specifically: when the data collected by adjacent Odom odometers has a high speed (especially angular velocity), the change in adjacent Odom odometer data is too large, and the laser sensor interpolated pose calculated by the interpolation method may have a large error. Therefore, it cannot be used, and the laser sensor interpolated pose for that timestamp should be deleted. Figure 3 As shown, the timestamp and interpolated pose of the Laser2D will be deleted.

[0074] This application uses interpolated poses instead of true values, employing the principle of uniform velocity. That is, within a short timeframe, the odometer's movement speed is relatively slow, allowing the odometer pose at any intermediate timestamp to be calculated from adjacent data (i.e., odometer poses). This is essentially linear interpolation. However, if the speed is too fast, the calculated odometer pose at any intermediate timestamp may have significant errors because, even at high speeds, the uniform velocity principle may not be satisfied within a short timeframe, thus viable as linear interpolation. The interpolated odometer pose may differ considerably from the true value. In one embodiment, a odometer linear velocity greater than 2 m / s and an angular velocity greater than 45° / s are set as thresholds.

[0075] After removing the Laser3D and Laser2D interpolated poses with large errors, the nearest Laser2D timestamp is found based on the Laser3D timestamp. This allows calculation of the odometry shift (i.e., the difference in recursive pose) between these two timestamps. Figure 4 As shown, the calculation method is as follows:

[0076] X 2t3 =X 3D -1 *X 2D

[0077] Y 2t3 =Y 3D -1 *Y 2D

[0078] - 1

[0079] θ 2t3 =θ 3D *θ 2D

[0080] Where: X 3D and X 2D These are the X-direction values ​​of the odometry recursive pose, interpolated from the previously calculated 3D and 2D laser timestamps; the Y-direction and θ-direction values ​​are calculated similarly. 2t3 The interval between the 2D and 3D laser timestamps corresponds to the Odom odometry pose change (movement) value in the X direction, which is the recursive pose change (delta value) value in the X direction obtained through interpolation. The Y and θ directions are treated similarly.

[0081] Then, using the 3D LiDAR extrinsic parameters (robot coordinate system, generally the Odom odometry coordinate system), the 2D LiDAR extrinsic parameters (robot coordinate system), the Odom odometry pose change, and the pose in the world coordinate system corresponding to the timestamps of the 2D LiDAR keyframes in the 2D pose diagram, the second pose in the world coordinate system corresponding to the 3D LiDAR timestamp can be calculated, such as... Figure 5 As shown, the calculation method is as follows:

[0082] X 3tW =X 2tW *X Ot2 *X 2t3 *X 3tO

[0083] Y 3tW =Y 2tW *Y Ot2 *Y 2t3 *Y 3tO

[0084] θ 3tW =θ 2tW *θ Ot2 *θ 2t3 *θ 3tO

[0085] Where: X 3tW For the X component of the 3D laser pose in the world coordinate system corresponding to the new 3D laser timestamp that needs to be calculated, X 2tW Let X be the X component of the 2D laser pose in the world coordinate system corresponding to the 2D laser timestamp in the 2D pose graph. Ot2 X is the inverse of the X component of the extrinsic parameters of a 2D lidar. 2t3 X is the X component of the odometry change between the previously calculated 2D and 3D timestamps. 3tO This represents the X component of the extrinsic parameters of the 3D LiDAR. The Y and θ directions are similar.

[0086] Finally, using a 3D timestamp, the first pose in the first 3D pose graph structure is replaced with the second pose to generate a second 3D pose graph structure. This second 3D pose graph structure is then combined with 3D laser point clouds used for 3D mapping (i.e., 3D LiDAR observation data processed according to mapping requirements) to generate a higher-quality 3D map using an occupancy grid map generation method.

[0087] Based on the above system, this application provides a method for generating 3D maps by combining 3D LiDAR with 2D mapping, applied to a host computer, including:

[0088] The first 3D pose graph structure and 2D composition data are obtained. The first 3D pose graph structure and 2D composition data are obtained by acquiring scene observation data from 3D LiDAR, Odom odometry and IMU on the vehicle terminal, performing 3D real-time composition and converting 3D data into 2D data format. The first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp.

[0089] Based on the 2D mapping data, a 2D mapping is performed to generate a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp.

[0090] Interpolation is performed on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​of the timestamps corresponding to the 2D laser keyframes and 3D laser keyframes.

[0091] Based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining the 3D map.

[0092] This application also provides a method for generating 3D maps by combining 3D LiDAR with 2D mapping, applicable to vehicle-mounted systems, including:

[0093] Acquire scene observation data from 3D LiDAR, Odom odometry and IMU, perform 3D real-time mapping and convert 3D data into 2D data format to generate a first 3D pose graph structure and 2D mapping data, wherein the first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp.

[0094] The first 3D pose graph structure and 2D mapping data are uploaded to the host computer, so that the host computer can perform 2D mapping based on the 2D mapping data to generate a 2D pose graph. The 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp. Based on the Odom odometry data in the 2D mapping data, interpolation is performed to obtain the Odom odometry recursive values ​​of the timestamps of the 2D and 3D laser keyframes. Based on the calculated Odom odometry recursive values ​​and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining a 3D map.

[0095] In one embodiment, the vehicle-mounted sensor may further include a 2D LiDAR, which is used for online real-time mapping, while a 3D LiDAR acquires 3D data (without using the 3D data for mapping). Then, using the 2D LiDAR-generated 2D pose map and the 3D LiDAR-acquired 3D data, a 3D pose map structure can be generated using the same method, and then a 3D map can be generated. In this case, the process of generating a 3D map by the system is as follows:

[0096] (1) The vehicle-mounted terminal acquires the scene observation data of 3D LiDAR, 2D LiDAR and Odom odometry, performs 2D laser real-time mapping, generates the 2D laser pose graph structure and uploads it to the host computer. The 2D laser pose graph structure includes the 2D laser key frame pose in the world coordinate system and the corresponding timestamp.

[0097] (2) The host computer interpolates the 2D laser keyframe pose, 3D laser radar observation data and Odom odometry data to obtain the Odom odometry recursive value of the timestamp corresponding to the 2D laser keyframe and 3D laser observation frame. Based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamp to obtain a 3D map.

[0098] Specifically, based on the aforementioned 2D laser keyframe pose, 3D lidar observation data, and Odom odometry data, interpolation is performed to obtain the Odom odometry recursive values ​​for the corresponding timestamps of the 2D laser keyframe and 3D laser observation frame, including:

[0099] 2D laser keyframes, Odom odometry data, and 3D laser observation frames are arranged by timestamp size;

[0100] Interpolation is performed based on the recursive pose in the Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframes and 3D laser observation frames. The specific interpolation process is the same as described above and will not be repeated here.

[0101] Based on the aforementioned system including 2D LiDAR, this application also provides a method for generating 3D maps by combining 3D LiDAR with 2D mapping, applied to a host computer, including:

[0102] A 2D pose graph structure is obtained, wherein the 2D pose graph structure includes the 2D laser keyframe pose in the world coordinate system and the corresponding timestamp. The observation data of the scene obtained by the vehicle-mounted terminal from the 3D LiDAR, 2D LiDAR, and Odom odometry are used to perform real-time 2D laser mapping.

[0103] Interpolation is performed based on the 2D laser keyframe pose, 3D lidar observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframe and the 3D laser observation frame.

[0104] Based on the calculated Odom odometry recursive value and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamps, thereby obtaining a 3D map.

[0105] This application also provides a method for generating 3D maps by combining 3D LiDAR with 2D mapping, applicable to vehicle-mounted systems, including:

[0106] Acquire scene observation data from 3D LiDAR, 2D LiDAR, and Odom odometry, perform real-time 2D laser mapping, and generate a 2D pose graph structure, wherein the 2D pose graph structure includes 2D laser keyframe poses in the world coordinate system and the corresponding timestamps.

[0107] The 2D pose graph structure is uploaded to the host computer, which then interpolates the 2D laser keyframe pose, 3D LiDAR observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​for the timestamps of the 2D laser keyframe and 3D laser observation frame. Based on the calculated Odom odometry recursive values ​​and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure using the corresponding timestamps, thereby obtaining a 3D map.

[0108] Accordingly, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method described above for generating a 3D map by combining 3D LiDAR with 2D mapping.

[0109] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described above for generating a 3D map by combining 3D LiDAR with 2D mapping. Figure 7 The diagram shown is a hardware structure diagram of any data processing device in which a system for generating 3D maps by combining 3D LiDAR with 2D mapping, according to an embodiment of the present invention, is located. (Except for...) Figure 7 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0110] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the method described above for generating a 3D map by combining 3D LiDAR with 2D mapping. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0111] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0112] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for generating 3D maps by combining 3D LiDAR with 2D mapping, characterized in that, Applications in host computers include: The first 3D pose graph structure and 2D composition data are obtained. The first 3D pose graph structure and 2D composition data are obtained by acquiring scene observation data from 3D LiDAR, Odom odometry and IMU on the vehicle terminal, performing 3D real-time composition and converting 3D data into 2D data format. The first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp. Based on the 2D mapping data, a 2D mapping is performed to generate a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp. Interpolation is performed on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​of the timestamps corresponding to the 2D laser keyframes and 3D laser keyframes. Based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining the 3D map.

2. The method according to claim 1, characterized in that, Interpolation is performed on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​for the corresponding timestamps of the 2D and 3D laser keyframes, including: The Odom odometry data, 2D laser keyframes, and 3D laser keyframes in the 2D mapping data are arranged by timestamp size; Interpolation is performed based on the recursive pose from the Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframes and 3D laser keyframes: Where: X 3D and X 2D The X-axis and Y-axis values ​​of the odometry recursive pose are interpolated from the timestamps of 3D and 2D laser keyframes, respectively. 3D and Y 2D These are the Y-direction values ​​for the interpolated pose, and θ is the value for the interpolated pose. 3D and θ 2D These represent the angular direction values ​​of the interpolated pose, t. 3D and t 2D These are the timestamps for the 3D laser keyframe and the 2D laser keyframe, respectively. X Odom Y Odom θ Odom and t Odom These are the X-direction value, Y-direction value, angle direction value, and corresponding timestamp of the recursive pose in the Odom odometry data.

3. The method according to claim 1, characterized in that, If the change in adjacent Odom odometer data used for interpolation exceeds a predetermined threshold, the Odom odometer recursive value obtained by interpolating the adjacent Odom odometer data is deleted.

4. The method according to claim 1, characterized in that, Based on the calculated Odom odometry recursive value and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, including: Based on the timestamps of the 3D laser keyframes, find the timestamp of the most recent 2D laser keyframe and calculate the odometry movement between these two timestamps: X 2t3 =X 3D -1 *X 2D AND 2t3 =Y 3D -1 *AND 2D -1 i 2t3 =θ 3D *i 2D Where: X 3D and X 2D The X-direction values ​​of the odometry recursive pose are interpolated using the timestamps of 3D and 2D laser keyframes, respectively; the Y-direction and θ-direction values ​​are calculated similarly. 2t3 The interval between the 2D laser and 3D laser timestamps corresponds to the value of the Odom odometry pose change in the X direction, which is the recursive pose change value in the X direction obtained by interpolation. The same applies to the Y and θ directions. The second pose in the world coordinate system corresponding to the 3D laser timestamp is calculated using the 3D LiDAR extrinsic parameters, 2D LiDAR extrinsic parameters, Odom odometry pose change, and the pose in the world coordinate system corresponding to the timestamp of the 2D laser keyframe in the 2D pose diagram. X 3tW =X 2tW *X Ot2 *X 2t3 *X 3tO AND 3tW =Y 2tW *AND Ot2 *AND 2t3 *AND 3tO i 3tW =θ 2tW *i Ot2 *i 2t3 *i 3tO Where: X etW Let X be the X component of the second pose in the world coordinate system. 2tW X is the X component of the 2D laser pose in the world coordinate system corresponding to the timestamp of the 2D laser keyframe. ot2 X is the inverse of the X component of the extrinsic parameters of a 2D lidar. 3tO The X component represents the extrinsic parameters of the 3D lidar; the Y and θ directions are similar.

5. A method for generating 3D maps by combining 3D lidar with 2D mapping, characterized in that, Applications in automotive applications include: Acquire scene observation data from 3D LiDAR, Odom odometry and IMU, perform 3D real-time mapping and convert 3D data into 2D data format to generate a first 3D pose graph structure and 2D mapping data, wherein the first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp. The first 3D pose graph structure and 2D mapping data are uploaded to the host computer, so that the host computer can perform 2D mapping based on the 2D mapping data to generate a 2D pose graph. The 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp. Based on the Odom odometry data in the 2D mapping data, interpolation is performed to obtain the Odom odometry recursive values ​​of the timestamps of the 2D and 3D laser keyframes. Based on the calculated Odom odometry recursive values ​​and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining a 3D map.

6. A system for generating 3D maps by combining 3D lidar with 2D mapping, characterized in that, It includes vehicle-mounted sensors, vehicle-mounted terminal and host computer, wherein the vehicle-mounted sensors include 3D LiDAR, Odom odometer and IMU; The vehicle-mounted terminal is used to acquire scene observation data from 3D LiDAR, Odom odometry and IMU, perform 3D real-time mapping and convert 3D data into 2D data format, generate a first 3D pose graph structure and 2D mapping data and upload them to the host computer. The first 3D pose graph structure includes the first pose of the 3D laser keyframe in the world coordinate system and the corresponding timestamp. The host computer is used to perform 2D mapping based on the 2D mapping data, generating a 2D pose graph, wherein the 2D pose graph includes the pose of the 2D laser keyframe in the world coordinate system and the corresponding timestamp; interpolation is performed based on the Odom odometry data in the 2D mapping data to obtain the Odom odometry recursive values ​​of the timestamps of the 2D laser keyframe and the 3D laser keyframe; based on the calculated Odom odometry recursive values ​​and the pose of the 2D laser keyframe, the second pose of the 3D laser keyframe is calculated, and the first pose in the first 3D pose graph structure is replaced by the corresponding timestamp to obtain the second 3D pose graph structure, thereby obtaining a 3D map.

7. A method for generating 3D maps by combining 3D LiDAR with 2D mapping, characterized in that, Applications in host computers include: A 2D pose graph structure is obtained, wherein the 2D pose graph structure includes the 2D laser keyframe pose in the world coordinate system and the corresponding timestamp. The observation data of the scene obtained by the vehicle-mounted terminal from the 3D LiDAR, 2D LiDAR, and Odom odometry are used to perform real-time 2D laser mapping. Interpolation is performed based on the 2D laser keyframe pose, 3D lidar observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​corresponding to the timestamps of the 2D laser keyframe and the 3D laser observation frame. Based on the calculated Odom odometry recursive value and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamps, thereby obtaining a 3D map.

8. A method for generating 3D maps by combining 3D LiDAR with 2D mapping, characterized in that, Applications in automotive applications include: Acquire scene observation data from 3D LiDAR, 2D LiDAR, and Odom odometry, perform real-time 2D laser mapping, and generate a 2D pose graph structure, wherein the 2D pose graph structure includes 2D laser keyframe poses in the world coordinate system and the corresponding timestamps. The 2D pose graph structure is uploaded to the host computer, which then interpolates the 2D laser keyframe pose, 3D LiDAR observation data, and Odom odometry data to obtain the Odom odometry recursive values ​​for the timestamps of the 2D laser keyframe and 3D laser observation frame. Based on the calculated Odom odometry recursive values ​​and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure using the corresponding timestamps, thereby obtaining a 3D map.

9. A system for generating 3D maps by combining 3D lidar with 2D mapping, characterized in that, The system includes vehicle-mounted sensors, a vehicle-mounted terminal, and a host computer. The vehicle-mounted sensors include 3D LiDAR, 2D LiDAR, an Odom odometer, and an IMU. The process of generating a 3D map using this system is as follows: (1) The vehicle-mounted terminal acquires the scene observation data of 3D LiDAR, 2D LiDAR and Odom odometry, performs 2D laser real-time mapping, generates a 2D pose graph structure and uploads it to the host computer. The 2D pose graph structure includes the 2D laser key frame pose in the world coordinate system and the corresponding timestamp. (2) The host computer interpolates the 2D laser keyframe pose, 3D laser radar observation data, and Odom odometry data to obtain the Odom odometry recursive value of the timestamp corresponding to the 2D laser keyframe and 3D laser observation frame; based on the calculated Odom odometry recursive value and the 2D laser keyframe pose, the 3D laser observation frame pose is calculated, and the 3D laser observation frame and 3D laser observation data are combined into a 3D laser pose graph structure through the corresponding timestamp to obtain a 3D map.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-5, 7-8.

Images