A method for constructing a terrain map containing height information for outdoor robots

Abstract

The present invention relates to a method for constructing a terrain map containing height information for an outdoor robot, including a terrain segmentation method, a map construction method, etc.; performing more detailed division and annotation of traversable areas according to the characteristics of different terrains; using the constructed terrain data set for model training; and constructing a mapping relationship between the terrain map and the point cloud map according to the geometric transformation relationship between the original image acquired by the sensor and the point cloud map, and then creating a terrain point cloud map. The point cloud map after height filtering is converted into an octree map, and the calculated height of the octree point cloud is stored as a height terrain map. The Bayesian probability update is integrated to reduce the noise of the semantic map. The present invention uses a semantic segmentation network to perform more detailed division of the terrain in the mobile environment of the outdoor robot, and creates a terrain height grid map containing rich information through pixel scanning, coordinate conversion and raster mapping, so as to improve the navigation efficiency of the outdoor robot.

Description

Technical Field

[0001] The present invention belongs to the field of robot perception and mapping technology, and relates to a method for constructing a terrain map containing height information for an outdoor robot. Background Art

[0002] Currently, more and more outdoor robots are working in unstructured field environments, which contain a variety of traversable terrains such as concrete, grass, brick and stone, and sand. When robots traverse these different traversable terrains, their safety, travel time, and energy consumption are all different. Mobile robots can simulate human travel strategies and plan routes based on different task requirements. For example, they can traverse high-cost terrain in a straight line to shorten travel time; bypass high-risk terrain such as stagnant water to ensure the robot's safety; and sacrifice travel time to choose easily traversable terrain to reduce energy consumption. The above-mentioned construction of a terrain map containing rich terrain information and travel costs for unstructured work terrain can help mobile robots maximize navigation efficiency while ensuring their own safety.

[0003] Currently, the technology for constructing two-dimensional environmental maps for mobile robots is highly mature. However, in unstructured outdoor scenes, environmental features at varying heights exist. Two-dimensional mapping methods cannot fully meet the robot's perception needs. Some areas that the robot could pass through, such as steps and grass between the road and the sidewalk, are detected as obstacles and cannot be passed. Three-dimensional mapping can provide robots with richer environmental information. However, the use of three-dimensional maps requires greater storage capacity. One solution to this problem is to create an octree-based voxel map. Some have proposed using octree-based voxel maps to compress the sample size of map data. However, while octree maps reduce the storage capacity of point cloud maps, the rich three-dimensional information they contain is redundant for field robot navigation.

[0004] Here, the present invention proposes to construct an environmental height map based on three-dimensional laser data, and rely on a monocular vision sensor to add auxiliary terrain information, ultimately avoiding the storage of a large amount of redundant three-dimensional data to further reduce the map storage volume, which can greatly improve the computing efficiency of the robot processor and reduce the performance requirements of the field robot chip. Summary of the Invention

[0005] The first object of the present invention is to address the deficiencies of the prior art and to propose a method for constructing a terrain map containing height information for an outdoor robot.

[0006] This invention primarily constructs environmental maps based on 3D laser data, relying on monocular vision sensors to add auxiliary topographic information. While map models for structured indoor environments cannot meet the requirements of outdoor map construction, researchers can develop effective 3D map models based on existing 2D map representation methods. Compared to 3D representation methods, using height maps to represent outdoor environments reduces storage complexity while discarding some environmental information.

[0007] To achieve the above object, the technical solution of the present invention is:

[0008] In a first aspect, the present invention provides a method for constructing a terrain map containing height information for an outdoor robot, comprising the following steps:

[0009] Step 1: Use a multi-line lidar to fuse a monocular camera to build a map. Before building the map, record the monocular camera's intrinsic parameters. Use the laser vision joint calibration tool Calibration_toolkit to jointly calibrate the lidar and camera extrinsic parameters to obtain the camera intrinsic parameter R and the camera radar extrinsic parameter RT.

[0010] Step 2: Use the LOAM outdoor SLAM mapping algorithm to fuse data from the mobile robot's lidar, IMU, and other sensors to construct a 3D point cloud map of the mobile robot's current outdoor environment. During the LOAM mapping process, point clouds with heights exceeding the robot's are filtered out, that is, point clouds above zero degrees of radar horizontal are discarded, resulting in a point cloud map suitable for efficient navigation of the mobile robot.

[0011] Step 3: Build a high-precision and lightweight landform segmentation network model;

[0012] A mobile robot's onboard camera is used to collect a variety of landform images in different scenarios. After manual data annotation, a landform dataset is constructed. The landform information in the dataset is divided into multiple landform types. The complex landform dataset is input into the constructed landform segmentation network to generate a landform segmentation network model.

[0013] The landform segmentation network includes a first 3x3 convolutional layer, three serially connected skip residual convolutional layers, three parallel pooling layers, a second 3x3 convolutional layer, two serially connected convolutional upsampling layers, and an upsampling layer;

[0014] The first 3x3 convolutional layer adjusts the number of channels of the input image through a 3x3 convolution and reduces the image size;

[0015] The three serially connected skip residual convolution layers include a first skip residual convolution layer, a second skip residual convolution layer, and a third skip residual convolution layer in sequence; the first skip residual convolution layer receives the output of the first 3x3 convolution layer, adjusts the number of channels through a 1x1 convolution, and uses the skip residual to extract image features; the second skip residual convolution layer and the third skip residual convolution layer both receive the image features output by the previous skip residual convolution layer, adjust the number of channels through a 1x1 convolution, and use the skip residual to further extract image features;

[0016] The skip residual consists of two pathways: the left backbone pathway first undergoes a 1x1 convolutional reconstructed pass, followed by three residual network calculations to extract features and output the final output. The right backbone pathway undergoes only a single 1x1 convolutional reconstructed pass. The skip residual module prevents parameter overfitting and improves network speed without sacrificing accuracy.

[0017] The three parallel pooling layers of different sizes receive the image features output by the third skip residual convolution layer, and process the image features through convolution pooling modules of different sizes to achieve enhanced feature extraction of features of different sizes of the image;

[0018] The second 3x3 convolutional layer receives the features output by the three parallel pooling layers and adjusts the number of channels;

[0019] The two convolutional upsampling layers connected in series include a first convolutional upsampling layer and a second convolutional upsampling layer;

[0020] The first convolution upsampling layer receives the output of the second 3x3 convolution layer and the third skip residual convolution layer, adjusts the channel and upsamples the image to the original size output through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image;

[0021] The second convolutional upsampling layer receives the output of the first convolutional upsampling layer and the second skip residual convolution layer, adjusts the channels and upsamples the image to its original size through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image;

[0022] The upsampling layer receives the output of the second convolutional upsampling layer and restores the image to its original size for output;

[0023] Step 4: Use the monocular camera on the mobile robot to capture raw images of the environment and input them into the terrain segmentation network model to generate a terrain segmentation semantic map of the local environment. Scan pixels and transform coordinates of each frame of the terrain image. Based on the terrain segmentation network model, achieve terrain-level semantic segmentation of single-frame images, and use the terrain segmentation semantic map combined with point cloud information to build a single-frame semantic annotation server.

[0024] Step 5: Based on the geomorphic semantic information contained in the geomorphic pixel point pt, the point cloud specified in the occupied point cloud map is incrementally mapped based on the Bayesian probability update to improve the point cloud semantic noise;

[0025] Step 6: Use the octree map format to reduce the point cloud map to a 3D raster map. The octree uses the connection between the root node and the leaf node to compress the data, and indicates whether the grid is occupied based on the information stored in the child node point cloud.

[0026] Step 7: Record the point cloud pz coordinates of each ground grid in the 3D grid map, and cluster the z coordinates according to the k-median algorithm;

[0027] Step 8: Save the z coordinates of the clustered point cloud as heights in the height raster map data format to obtain a height relief map.

[0028] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0029] In a third aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention uses a jump residual network to reduce the number of network parameters and improve the network's terrain segmentation speed, making it more suitable for mobile robots with limited on-board computing resources. A more detailed terrain segmentation dataset for the mobile robot's working environment is used to provide rich terrain semantic information for the mobile robot's mapping. The terrain information is mapped to the occupancy grid map, and the incremental update based on Bayesian judgment improves the mathematical completeness of the terrain semantic information mapping. A terrain grid map layer is constructed, and the height information of the terrain is mapped to a new height layer layer. A height grid map layer is constructed. The constructed terrain height map provides more detailed environmental information for the mobile robot's navigation, improving the accuracy and efficiency of the robot's navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 This is the process of constructing a landform map of the present invention.

[0034] Figure 2 Schematic diagram of the skip residual network of the present invention.

[0035] Figure 3 Schematic diagram of the semantic segmentation network of the present invention.

[0036] Figure 4 The present invention provides a structure of a height topographic map that facilitates outdoor navigation of an outdoor mobile robot. DETAILED DESCRIPTION

[0037] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] This embodiment uses a wheeled robot, a computer with a GPU, a multi-line laser radar and a depth camera to build an experimental platform. A method for constructing a high-altitude terrain map that facilitates efficient navigation of outdoor mobile robots, such as Figure 1 The specific implementation steps include the following:

[0039] Step 1: Use a multi-line lidar to fuse a monocular camera to build a map. Before building the map, record the monocular camera's intrinsic parameters. Use the laser vision joint calibration tool Calibration_toolkit to jointly calibrate the lidar and camera extrinsics to obtain the camera intrinsic parameter R and the camera-radar extrinsic parameter RT to obtain accurate coordinate transformation information for subsequent sensor information fusion.

[0040] Step 2: The LOAM (Lidar Odometry and Mapping in Real-time) outdoor SLAM mapping algorithm, a lidar-based SLAM system built on the ROS platform, integrates sensor data from the mobile robot's lidar, inertial measurement unit (IMU), and other sensors to construct a 3D point cloud map of the mobile robot's current outdoor environment. During the LOAM mapping process, point clouds above the robot's height are filtered out, discarding points above zero degrees of radar horizontal. This results in a point cloud map suitable for efficient navigation of the mobile robot.

[0041] The specific operation for filtering out the point cloud whose height exceeds the robot is as follows:

[0042] Calculate the elevation angle of the radar point (point.x, point.y, point.z), calculate the angle line number scanID of the lidar velodyne based on the elevation angle, and filter out the point cloud with line numbers above zero degrees horizontally;

[0043]

[0044]

[0045] Step 3: Build a high-precision, lightweight landform segmentation network model. Use a mobile robot's onboard camera to capture twelve types of landforms in ten different scenarios. After manual data annotation, a landform dataset is constructed. The landform information in the dataset is divided into various types, including concrete, grass, mud, woodland, gravel, water, and ditches. The complex landform dataset is input into the constructed landform segmentation network to generate a landform segmentation model.

[0046] Figure 3 The landform segmentation network includes a first 3x3 convolutional layer, three serially connected skip residual convolutional layers, three parallel pooling layers, a second 3x3 convolutional layer, two serially connected convolutional upsampling layers, and an upsampling layer;

[0047] The first 3x3 convolutional layer adjusts the number of channels of the input image through a 3x3 convolution and reduces the image size;

[0048] The three serially connected skip residual convolution layers include a first skip residual convolution layer, a second skip residual convolution layer, and a third skip residual convolution layer in sequence; the first skip residual convolution layer receives the output of the first 3x3 convolution layer, adjusts the number of channels through a 1x1 convolution, and uses the skip residual to extract image features; the second skip residual convolution layer and the third skip residual convolution layer both receive the image features output by the previous skip residual convolution layer, adjust the number of channels through a 1x1 convolution, and use the skip residual to further extract image features;

[0049] Figure 2 The skip residual consists of two pathways: the left backbone pathway first undergoes a 1x1 convolutional reconstructed pass, followed by three residual network calculations to extract features and output the final output. The right backbone pathway undergoes only a single 1x1 convolutional reconstructed pass. The skip residual module prevents parameter overfitting and improves network speed without sacrificing accuracy.

[0050] The three parallel pooling layers of different sizes receive the image features output by the third skip residual convolution layer, and process the image features through convolution pooling modules of different sizes to achieve enhanced feature extraction of features of different sizes of the image;

[0051] The second 3x3 convolutional layer receives the features output by the three parallel pooling layers and adjusts the number of channels;

[0052] The two convolutional upsampling layers connected in series include a first convolutional upsampling layer and a second convolutional upsampling layer;

[0053] The first convolution upsampling layer receives the output of the second 3x3 convolution layer and the third skip residual convolution layer, adjusts the channel and upsamples the image to the original size output through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image;

[0054] The second convolutional upsampling layer receives the output of the first convolutional upsampling layer and the second skip residual convolution layer, adjusts the channels and upsamples the image to its original size through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image;

[0055] The upsampling layer receives the output of the second convolutional upsampling layer and restores the image to its original size for output;

[0056] Specifically:

[0057] 1. Pass the input image through a 3x3 convolutional layer to adjust the number of channels and reduce the image size.

[0058] 2. Adjust the number of channels through three 1x1 convolutions and use the skip residual module to extract image features.

[0059] 3. Process image features through convolutional pooling modules of different sizes to enhance the extraction of features of different sizes in the image.

[0060] 4. The inputs of the last two layers are connected to the output layers of the first two layers through channels, and a pyramid connection is established to integrate information of different dimensions and enhance the network's ability to understand different receptive fields of the image.

[0061] 5. Adjust the channel and upsample the image to its original size through 3x3 convolution.

[0062] Step 4: Use the mobile robot's onboard monocular camera to capture raw environmental images and input them into the terrain segmentation model to generate a semantic map of the local environment. Each frame of the terrain image is subjected to pixel scanning and coordinate conversion. A deep learning model is used to achieve terrain-level semantic segmentation for each frame, and a single-frame semantic annotation server is built by combining point cloud information.

[0063] 1. Convert the current lidar point to homogeneous coordinates and store them in the 4D variable X.

[0064] 2. Apply the extrinsic projection matrix to map the lidar point variable X to the camera's image plane and store the result in Y.

[0065] 3. After completing the coordinate conversion, convert Y back to Euclidean coordinates and store the image coordinates in the variable pt.

[0066] 4. Transfer the current frame image img pixel pt color to the point cloud pt color

[0067] pt color .r=img.(pt.y,py.x)[2] (3)

[0068] p tcolor .g=img.(pt.y,py.x)[1] (4)

[0069] p tcolor .b=img.(pt.y,py.x)[0] (5)

[0070] Step 5: Based on the geomorphic semantic information contained in the geomorphic pixels, the point cloud specified in the occupied point cloud map is incrementally mapped based on the Bayesian probability update to improve the semantic noise of the point cloud;

[0071] Use the instance semantic probability provided by the semantic annotation server for the current frame to update the terrain category probability p. Use the Bayesian estimation method to fuse the category probability at the current moment with the category probability predicted at the previous moment:

[0072]

[0073] Among them, c i Represents category i, x t Represents the current prediction result, p(c i |x 1:t-1 ) represents the category probability fusion result at time t-1, and the category probability is recorded as:

[0074]

[0075] The formula can be simplified to:

[0076] L(c i |x 1:t )=L(c i |x 1:t-1 )+L(c i |x t ) (8)

[0077] In this way, probability fusion can be simplified to simple addition, subtraction and log operations, and the category c to which the instance belongs can be updated according to the result of semantic fusion.

[0078] Step 6: Use the octree map format to reduce the point cloud map to a three-dimensional raster map. The octree uses the connection between the root node and the leaf node to compress the data, and indicates whether the grid is occupied based on the information stored in the child node point cloud.

[0079] Step 7: Record the point cloud pz coordinates of each ground grid in the 3D grid map, and cluster the z coordinates according to the k-median algorithm

[0080] Randomly select k cluster center points, calculate the distance between each point and each center point, and assign the point to the cluster to which the center point closest to it belongs

[0081]

[0082] Where x i Represents the i-th attribute of point x, med k Represents the median value of the data in the kth cluster. The ultimate goal of the algorithm is to minimize this objective function and calculate the median value of each cluster to update the center point of the cluster.

[0083] Step 8: Save the z coordinate of the clustered point cloud as height and store it in the height grid map data format to obtain the height relief map ( Figure 4 ).

Claims

1. A method for constructing a terrain map containing height information for an outdoor robot, characterized in that The following steps are involved: Step 1: Use a multi-line lidar to fuse a monocular camera to build a map. Before building the map, record the monocular camera's intrinsic parameters. Use the laser vision joint calibration tool Calibration_toolkit to jointly calibrate the lidar and camera extrinsic parameters to obtain the camera intrinsic parameter R and the camera radar extrinsic parameter RT. Step 2: Use the LOAM outdoor SLAM mapping algorithm to fuse data from the mobile robot's lidar, IMU, and other sensors to construct a 3D point cloud map of the mobile robot's current outdoor environment. During the LOAM mapping process, point clouds with heights exceeding the robot's are filtered out, that is, point clouds above zero degrees of radar horizontal are discarded, resulting in a point cloud map suitable for efficient navigation of the mobile robot. Step 3: Build a high-precision and lightweight landform segmentation network model; A mobile robot's onboard camera is used to collect a variety of landform images in different scenarios. After manual data annotation, a landform dataset is constructed. The landform information in the dataset is divided into multiple landform types. The complex landform dataset is input into the constructed landform segmentation network to generate a landform segmentation network model. The landform segmentation network includes a first 3x3 convolutional layer, three serially connected skip residual convolutional layers, three parallel pooling layers, a second 3x3 convolutional layer, two serially connected convolutional upsampling layers, and an upsampling layer; The first 3x3 convolutional layer adjusts the number of channels of the input image through a 3x3 convolution and reduces the image size; The three serially connected skip residual convolution layers include a first skip residual convolution layer, a second skip residual convolution layer, and a third skip residual convolution layer in sequence; the first skip residual convolution layer receives the output of the first 3x3 convolution layer, adjusts the number of channels through a 1x1 convolution, and uses the skip residual to extract image features; the second skip residual convolution layer and the third skip residual convolution layer both receive the image features output by the previous skip residual convolution layer, adjust the number of channels through a 1x1 convolution, and use the skip residual to further extract image features; The skip residual consists of two paths, the left main path first passes through a 1x1 convolution adjustment channel, and then performs three residual network calculations to extract features and output the final output, while the right path only undergoes one 1x1 convolution adjustment channel; The three parallel pooling layers have different sizes, receive the image features output by the third skip residual convolution layer, and process the image features through convolution pooling of different sizes to achieve enhanced feature extraction of features of different sizes of the image; The second 3x3 convolutional layer receives the features output by the three parallel pooling layers and adjusts the number of channels; The two convolutional upsampling layers connected in series include a first convolutional upsampling layer and a second convolutional upsampling layer; The first convolution upsampling layer receives the output of the second 3x3 convolution layer and the third skip residual convolution layer, adjusts the channel and upsamples the image to the original size output through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image; The second convolutional upsampling layer receives the output of the first convolutional upsampling layer and the second skip residual convolution layer, adjusts the channels and upsamples the image to its original size through 3x3 convolution, and integrates information of different dimensions to enhance the network's ability to understand different receptive fields of the image; The upsampling layer receives the output of the second convolutional upsampling layer and restores the image to its original size for output; Step 4: Use the monocular camera on the mobile robot to collect the original image of the environment and input it into the landform segmentation network model to generate a landform segmentation semantic map of the local environment; perform pixel scanning and coordinate conversion on each frame of the landform image in the landform segmentation semantic map; and use the landform segmentation semantic map combined with point cloud information to build a single-frame semantic annotation server; Step 5: Based on the geomorphic semantic information contained in the geomorphic pixels, the point cloud specified in the occupied point cloud map is incrementally mapped based on the Bayesian probability update to improve the semantic noise of the point cloud; Step 6: Use the octree map format to reduce the point cloud map to a 3D raster map. The octree uses the connection between the root node and the leaf node to compress the data, and indicates whether the grid is occupied based on the information stored in the child node point cloud. Step 7: Record the point cloud coordinates of each ground grid in the 3D grid map and cluster the z coordinates using the k-median algorithm; Step 8: Save the z coordinates of the clustered point cloud as heights and store them in the height raster map data format to obtain a height relief map.

2. The method according to claim 1, characterized in that The specific operation for filtering out the point cloud whose height exceeds the robot is as follows: Calculate the elevation angle of the radar point, calculate the angle line number of the lidar velodyne based on the elevation angle, and filter out the point cloud with line numbers above zero degrees horizontally; 3. The method according to claim 1, characterized in that Step 4 is as follows: 4-1 Convert the current lidar point to homogeneous coordinates and store them in a 4D variable; 4-2 Apply the internal and external parameter projection matrix to map the lidar point variables to the camera's image plane and store the results in; 4-3 After the coordinate conversion is completed, it will be converted back to Euclidean coordinates and the image coordinates will be stored in the current frame image topographic pixel points; 4-4Transfer the current frame image terrain pixel color to the point cloud pt color .r=img.(pt.y,py.x)[2] (3) pt color .g=img.(pt.y,py.x)[1] (4) pt color .b=img.(pt.y,py.x)[0] (5)。 4. The method according to claim 1, characterized in that Step 5 is specifically: Use the instance semantic probability provided by the semantic annotation server for the current frame to update the landform category probability. Use the Bayesian estimation method to fuse the category probability at the current moment with the category probability predicted at the previous moment: Among them, represents the category, represents the current prediction result, represents the category probability fusion result at the moment, and the category probability is recorded as: Then simplify formula (7) to: L(c i |x 1:t )=L(c i |x 1:t-1 )+L(c i |x t ) (8) According to the results of semantic fusion, the category to which the instance belongs is updated.

5. The method according to claim 1, characterized in that Step seven is as follows: Randomly select k cluster center points, calculate the distance between each point and each center point, and assign the point to the cluster to which the center point closest to it belongs; Where, represents the kth attribute of the point, represents the median value of the data in the kth cluster; Minimize the objective function (9) and calculate the median of each cluster to update the center point of the cluster.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

7. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 5 is implemented.

Images