An Automatic Generation Method of Mining Area Map for Unstructured Roads

Abstract

The present invention provides a method for automatically generating a mining area map for unstructured roads, including two parts: equipment installation and data annotation, and map generation. A fast 3D point cloud semantic segmentation technology based on deep learning in the present invention projects the 3D point cloud onto a spherical surface to generate 2D image information with 5D features, so that a standard 2D convolutional neural network can be used to process the 3D point cloud, realizing real-time segmentation of the 3D point cloud. The map construction method of fusing GPS / IMU and lidar in the present invention uses GPS / IMU to calculate the accurate global coordinates of each frame of point cloud, avoiding the complex calculation and accuracy loss of inter-frame feature matching. A method for automatically generating a global map by fusing the edge and the cloud in the present invention stitches and updates the local maps established by multiple vehicles in the cloud. For map construction in large-scale scenarios such as mining areas, fast automatic map generation and update can be realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and particularly to a method for automatically generating a mining area map for unstructured roads. Background Art

[0002] In recent years, new generation of science and technology represented by artificial intelligence has developed rapidly. Intelligent driving is an important strategy for the development of new generation of artificial intelligence in China. Among them, mines have become important application scenarios for intelligent driving. On the one hand, the environment in mines is harsh, and the high-intensity working conditions have caused serious damage to the health of drivers. On the other hand, various mechanical operations in mine scenarios are relatively simple, and the uncontrollable factors such as vehicle conditions and pedestrians on the road are relatively lower than those in urban scenarios, making it more suitable for the implementation of driverless applications.

[0003] Environmental perception and positioning are the basis of intelligent driving systems. Unstructured roads in mining areas have higher requirements for environmental perception and positioning. Among them, high-precision maps can make up for the blind spot defects of autonomous vehicle perception and achieve accurate and reliable understanding and positioning of environmental information by combining semantic information.

[0004] CN202010408365.3 proposes a method, device, storage medium and electronic device for constructing a three-dimensional semantic map. In this invention, images are recognized and the recognition results of the images are matched to a point cloud map to obtain a three-dimensional semantic map.

[0005] CN201911340905.2 proposes a method for constructing a three-dimensional semantic map of a robot indoor environment based on deep learning. This invention proposes to obtain RGB images and depth images through a depth camera, use the ICP algorithm to match adjacent two-frame ORB features, estimate the pose between two frames, and perform semantic segmentation on the determined key frames to obtain a semantic map.

[0006] There are at least the following problems in the prior art:

[0007] Obtaining a semantic map by matching image recognition information with a point cloud map has high requirements for the calibration error between the camera and the lidar. The pose estimation algorithm based on RGB feature matching has a high computational complexity and is sensitive to light changes. And the above algorithms only aim at the construction of local maps and do not establish the association of different entity-constructed maps in the global map. Summary of the Invention

[0008] In order to solve the above problems existing in the current prior art, the present invention proposes a method for automatically generating a mining area map for unstructured roads. The technical solution is as follows:

[0009] A method for automatically generating a mining area map for unstructured roads includes equipment installation, data annotation and map generation. The specific steps are as follows:

[0010] (1) Equipment installation and data annotation:

[0011] 1.1): Install sensors on the auxiliary operation vehicle, including lidar, camera, GPS antenna and IMU module, calibrate the position of the lidar, and measure the XYZ direction deviation between the lidar position and the position of the vehicle body center; Park the vehicle on an open and flat road surface, vertically place a checkerboard calibration board 10 meters in front of the vehicle body center line, fit the plane where the checkerboard is located according to the extracted point features, obtain the yaw, roll and pitch angles between the vehicle body coordinate system and the lidar coordinate system, and determine the transformation matrix R and T between the vehicle body coordinate system and the lidar coordinate system according to the position deviations Δx, Δy, Δz;

[0012]

[0013] T = [Δx Δy Δz] T

[0014] 1.2): Collect lidar point cloud and image data of the mining area road, and use the image as a reference to label the point cloud, which is divided into categories such as cars, trucks, pedestrians, drivable areas and other categories;

[0015] 1.3): Perform spherical projection on the 3D point cloud coordinates (x, y, z), where w and h are the converted 2D image width and height, fup is the upper field of view angle, f is the sum of the upper field of view angle and the lower field of view angle, and obtain the pixel coordinates (u, v) of each point cloud on the image after projection;

[0016]

[0017]

[0018] Obtain 2D data including x, y, z, reflection intensity and distance, and use a convolutional neural network to train the 2D data to obtain a semantic segmentation model for mining area data;

[0019] (2) Map generation:

[0020] 2.1): Process the lidar, camera, GPS and IMU data collected synchronously by the processing terminal. The lidar provides dense 3D point cloud data, the GPS provides the WGS84 coordinates of the vehicle centroid, and the IMU provides the vehicle attitude information, that is, the pitch angle v , the heading angle yaw v , the roll angle roll v , and the image data provided by the camera is only for annotation reference;

[0021] 2.2): In the map construction initialization stage, convert the obtained first-frame WGS84 coordinates to the common UTM coordinate system (x o , y o , z o ), and save this coordinate as the map origin coordinate, and establish a reference coordinate system with this point as the origin;

[0022] 2.3): Judge the distance between two frames of data according to the WGS84 coordinates. If the distance is greater than the threshold, then determine that this frame is a key frame;

[0023] 2.4): Perform spherical projection on the key frame point cloud data, and then input it into the trained semantic segmentation model to obtain the semantic information of the point cloud data;

[0024] 2.5): Convert the point cloud from the lidar coordinate system to the map construction reference coordinate system;

[0025] 2.6): The point cloud with semantic information is saved as a key frame of the map after coordinate transformation. As the vehicle runs, the key frames are superimposed to generate the local point cloud of the vehicle driving area;

[0026] 2.7): After the vehicle stops collecting, it will store and upload the point cloud map and the WGS84 coordinates of the map origin. Among them, the point cloud map is stored and uploaded in the form of a pcd file, and the origin coordinates are stored and uploaded in the form of a txt file;

[0027] 2.8): When the cloud receives the point cloud map and the corresponding origin coordinates uploaded by the vehicle, convert the WGS84 coordinates of the origin to UTM coordinates, and convert the coordinates of each point in the point cloud map from the reference coordinate system to UTM coordinates;

[0028] 2.9): Sequentially splice the local map information uploaded by different vehicles according to the upload time sequence. For the information uploaded at the same time, update it according to the result with a higher confidence output by the semantic segmentation model.

[0029] Preferably, step 2.5) is specifically as follows:

[0030] ① Obtain the WGS84 coordinates of the current frame of the vehicle and convert them to UTM coordinates (x v , y v , z v ), and convert the vehicle attitude information obtained by the IMU: pitch angle pitch v , yaw angle yaw v , roll angle roll v into the rotation matrix P from the vehicle body coordinate system to the UTM coordinate system. According to the translation transformation matrix T between the vehicle body coordinate system and the lidar coordinate system, calculate the UTM coordinates (x l , yl , z l );

[0031]

[0032]

[0033] Calculate the coordinates (x l ', y l ', z l ') of the position where the lidar is located in the reference coordinate system;

[0034]

[0035] ② Calculate the attitude information α, β, γ of the vehicle according to the rotation matrix P from the vehicle coordinate system to the UTM coordinate system and the rotation matrix R from the lidar to the vehicle coordinate system

[0036]

[0037] ③ Calculate the selection matrix R' and translation matrix T' of the lidar point cloud to the reference coordinate system according to the position and attitude angles of the lidar in the reference coordinate system;

[0038]

[0039] T' = [x' l y' l z' l T

[0040] ④ Use the transformation matrix to calculate the position (X, Y, Z) of each point in the reference coordinate system;

[0041]

[0042] An automatic mining area map generation method for unstructured roads according to the present invention has the following advantages:

[0043] (1) A fast 3D point cloud semantic segmentation technology based on deep learning is proposed. The 3D point cloud is spherically projected to generate 2D image information with 5-dimensional features, so that the 3D point cloud can be processed by a standard 2D convolutional neural network, realizing real-time segmentation of the 3D point cloud.

[0044] (2) The map construction method of GPS / IMU and lidar fusion proposed by the present invention uses GPS / IMU to calculate the accurate global coordinates of each frame of point cloud, avoiding the complex calculation and accuracy loss of inter-frame feature matching.

[0045] ​(3) The present invention proposes a method for automatically generating a global map through end-cloud integration, which stitches and updates the local maps established by multiple vehicles in the cloud. For map construction in large-scale scenarios such as mining areas, rapid automatic map generation and update can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be construed as limiting the present invention in any way. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them:

[0047] Figure 1 is the flowchart of map building of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0049] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0050] A method for automatically generating a mining area map for unstructured roads includes two parts: equipment installation and data annotation, and map generation:

[0051] (1) Equipment installation and data annotation:

[0052] Step 1: Install sensors on the auxiliary operation vehicle, including lidar, camera, GPS antenna and IMU module, calibrate the position of the lidar, and obtain the conversion relationship between the vehicle body coordinate system and the lidar coordinate system;

[0053] The detailed steps are as follows: Install sensors on the auxiliary work vehicle, including lidar, cameras, GPS antennas, and IMU modules, and calibrate the position of the lidar. Measure the XYZ-direction deviations between the lidar position and the position of the vehicle body center; Park the vehicle on an open and flat road surface, vertically place a checkerboard calibration board 10 meters in front of the vehicle body center line, fit the plane where the checkerboard is located according to the extracted point features, obtain the yaw angle yaw, roll angle roll, and pitch angle pitch between the vehicle body coordinate system and the lidar coordinate system, and determine the transformation matrix R and T between the vehicle body coordinate system and the lidar coordinate system according to the position deviations Δx, Δy, and Δz;

[0054]

[0055] T = [Δx Δy Δz] T

[0056] Step 2: Collect lidar point clouds and image data of the mining area road, and, with the image as a reference, perform category annotation on the point clouds, classifying them into cars, trucks, pedestrians, drivable areas, and other categories;

[0057] Step 3: Perform spherical projection on the 3D point cloud coordinates (x, y, z), where w and h are the converted 2D image width and height, fup is the upper field of view angle, f is the sum of the upper field of view angle and the lower field of view angle, and after projection, obtain the pixel coordinates (u, v) of each point cloud on the image;

[0058]

[0059]

[0060] Obtain 2D data containing x, y, z, reflection intensity, and distance, and use a convolutional neural network to train the model on the 2D data to obtain a semantic segmentation model for mining area data;

[0061] (2) Map generation:

[0062] Step 1: Process the lidar, camera, GPS, and IMU data collected and synchronized by the terminal. The lidar provides dense 3D point cloud data, the GPS provides the WGS84 coordinates of the vehicle's centroid, the imu provides the vehicle's attitude information, namely the pitch angle, heading angle, and roll angle, and the image data provided by the camera is only for annotation reference;

[0063] Step 2: In the map building initialization stage, convert the obtained first-frame WGS84 coordinates into a common UTM coordinate system (x o , y o , z o ), and save this coordinate as the map building origin coordinate, and establish a reference coordinate system with this point as the origin;

[0064] Step 3: Determine the distance between two frames of data based on the WGS84 coordinates. If the distance is greater than the threshold, then determine that frame as a key frame;

[0065] Step 4: Perform spherical projection on the key frame point cloud data, and then input it into the trained semantic segmentation model to obtain the semantic information of the point cloud data;

[0066] Step 5: Convert the point cloud from the lidar coordinate system to the mapping reference coordinate system, specifically as follows:

[0067] ① Obtain the WGS84 coordinates of the current vehicle frame and convert them to UTM coordinates (x v , y v , z v ). Convert the vehicle attitude information pitch v , yaw v , roll v obtained by the imu into the rotation matrix P from the vehicle body coordinate system to the UTM coordinate system. According to the translation transformation matrix T between the vehicle body coordinate system and the lidar coordinate system, calculate the UTM coordinates (x l , y l , z l ) of the position where the lidar is located;

[0068]

[0069]

[0070] Calculate the coordinates (x′ l , y′ l , z′ l ) of the position where the lidar is located in the reference coordinate system;

[0071]

[0072] ② According to the rotation matrix P from the vehicle body coordinate system to the UTM coordinate system and the rotation matrix R from the lidar to the vehicle body coordinate system, calculate the vehicle attitude information α, β, γ

[0073]

[0074] ③ According to the position and attitude angles of the lidar in the reference coordinate system, calculate the selection matrix R’ and the translation matrix T’ of the lidar point cloud in the reference coordinate system;

[0075]

[0076] T‘ = [x′ l y′ l z′ l T ​

[0077] ④ Calculate the position (X, Y, Z) of each point in the reference coordinate system using the transformation matrix;

[0078]

[0079] Step 6: The point cloud with semantic information is saved as a key frame of the map after coordinate transformation. As the vehicle runs, the key frames are superimposed to generate a local point cloud of the vehicle's driving area;

[0080] Step 7: After the vehicle stops collecting, the point cloud map and the WGS84 coordinates of the mapping origin will be stored and uploaded to the cloud. The point cloud map is stored and uploaded in the form of a pcd file, and the origin coordinates are stored and uploaded in the form of a txt file;

[0081] Step 8: After the cloud receives the point cloud map and the corresponding origin coordinates uploaded by the vehicle, it converts the WGS84 coordinates of the origin to UTM coordinates, and converts the coordinates of each point in the point cloud map from the reference coordinate system to UTM coordinates;

[0082] Step 9: Sequentially splice the local map information uploaded by different vehicles according to the upload time order. For the information uploaded simultaneously, update it according to the result with a higher confidence output by the semantic segmentation model;

[0083] An automatic generation method of a mining area map for unstructured roads according to the present invention has the following advantages:

[0084] (1) A fast 3D point cloud semantic segmentation technology based on deep learning is proposed. The 3D point cloud is spherically projected to generate 2D image information with 5D features, so that a standard 2D convolutional neural network can be used to process the 3D point cloud, realizing real-time segmentation of the 3D point cloud.

[0085] (2) The map construction method of GPS / IMU and lidar fusion proposed by the present invention uses GPS / IMU to calculate the accurate global coordinates of each frame of point cloud, avoiding the complex calculation and accuracy loss of inter-frame feature matching.

[0086] (3) The present invention proposes a global map automatic generation method of edge-cloud fusion. The local maps established by multiple vehicles are spliced and updated in the cloud. For map construction in large-scale scenarios such as mining areas, fast automatic map generation and update can be realized.

[0087] The above are only the embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An automatic generation method of mining area maps for unstructured roads, characterized by including equipment installation, data annotation, and map generation. The specific steps are as follows: (1) Equipment installation and data annotation: 1.1): Install sensors on the auxiliary operation vehicle, including lidar, cameras, GPS antennas, and IMU modules, calibrate the position of the lidar, and measure the XYZ-direction deviations between the lidar position and the position of the vehicle body center. Park the vehicle on an open and flat road surface, vertically place a checkerboard calibration board 10 meters in front of the vehicle body center line, fit the plane where the checkerboard is located according to the extracted point features, obtain the yaw angle yaw, roll angle roll, and pitch angle pitch of the vehicle body coordinate system and the lidar coordinate system, and determine the transformation matrices R and T of the vehicle body coordinate system and the lidar coordinate system according to the position deviations Δx, Δy, and Δz. T = [ΔxΔyΔz] T 1.2): Collect lidar point cloud and image data of the mining area road, and use the image as a reference to label the point cloud into categories, including cars, trucks, pedestrians, drivable areas, and other categories. 1.3): Perform spherical projection on the 3D point cloud coordinates (x, y, z), where w and h are the width and height converted into a 2D image, and f up is the upper field of view angle, f is the sum of the upper field of view angle and the lower field of view angle. After projection, obtain the pixel coordinates (u, v) of each point cloud on the image; Obtain 2D data including x, y, z, reflection intensity, and distance, and use a convolutional neural network to train the 2D data to obtain a semantic segmentation model for mining area data. (2) Map generation: 2.1): The processing terminal collects synchronized lidar, camera, GPS, and IMU data. The lidar provides dense 3D point cloud data, the GPS provides the WGS84 coordinates of the vehicle's centroid, and the IMU provides the vehicle's attitude information, i.e., the pitch angle v , the yaw angle v , and the roll angle v . The image data provided by the camera is only for annotation reference; 2.2): In the map building initialization stage, convert the first frame of WGS84 coordinates obtained into the general UTM coordinate system (x o , y o , z o ), save this coordinate as the map building origin coordinate, and establish a reference coordinate system with this point as the origin; 2.3): Judge the distance between two frames of data according to the WGS84 coordinates. If the distance is greater than the threshold, determine that the latest frame is a key frame. 2.4): Perform spherical projection on the key frame point cloud data, and then input it into the trained semantic segmentation model to obtain the semantic information of the point cloud data. 2.5): Convert the point cloud from the lidar coordinate system to the mapping reference coordinate system. 2.6): The point cloud with semantic information is saved as a key frame of the map after coordinate transformation, and the key frames are superimposed as the vehicle runs to generate a local point cloud of the vehicle driving area. 2.7): After the vehicle stops collecting, it will store and upload the point cloud map and the WGS84 coordinates of the mapping origin to the cloud. The point cloud map is stored and uploaded in the form of a pcd file, and the origin coordinates are stored and uploaded in the form of a txt file. 2.8): The cloud receives the point cloud map and the corresponding origin coordinates uploaded by the vehicle, converts the WGS84 coordinates of the origin to UTM coordinates, and converts the coordinates of each point in the point cloud map from the reference coordinate system to UTM coordinates. 2.9): Sequentially splice the local map information uploaded by different vehicles according to the upload time sequence, and for the information uploaded at the same time, update it according to the result with a higher confidence output by the semantic segmentation model.

2. An automatic mining area map generation method for unstructured roads according to claim 1, characterized in that, step 2.5) is specifically as follows: ①Obtain the WGS84 coordinates of the current vehicle frame and convert them to UTM coordinates (x v , y v , z v ). Convert the vehicle attitude information obtained by the IMU: pitch angle v , yaw angle v , roll angle v into the rotation matrix P from the vehicle body coordinate system to the UTM coordinate system. According to the translation transformation matrix T between the vehicle body coordinate system and the lidar coordinate system, calculate the UTM coordinates (x l , y l , z l ) of the position where the lidar is located; Calculate the coordinates (x l ′, y l ′, z l ′) of the position where the lidar is located in the reference coordinate system; ② Calculate the attitude information α, β, γ of the vehicle according to the rotation matrix P from the vehicle body coordinate system to the UTM coordinate system and the rotation matrix R from the lidar to the vehicle body coordinate system. ③ Calculate the rotation matrix R' and translation matrix T' of the lidar point cloud to the reference coordinate system according to the position and attitude angle of the lidar in the reference coordinate system. T' = [x l 'y l 'z l '] T ④ Use the transformation matrix to calculate the position (X, Y, Z) of each point in the reference coordinate system.

Images