Terrain semantic segmentation method for humanoid robot in field environment
By constructing a passability data set and aggregating feature maps using sparse convolutional layers and convolutional gated recursive units, the problem that humanoid robots are difficult to judge the traversability of terrain under complex natural terrain is solved, and efficient and secure unmanned system navigation is achieved.
Patent Information
- Application Number
- CN202510153336.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-12
- Publication Date
- 2025-06-06
AI Technical Summary
Under complex natural terrain, it is difficult for humanoid robots to judge whether the terrain can pass through through sparse lidar data, especially in environments such as rapid changes in ground planes, dense vegetation and hanging branches.
A terrain semantic segmentation method in the wild environment of humanoid robots is adopted. By constructing a passability data set, lidar scan data is aggregated, semantic labels are mapped to passability levels, and feature maps are aggregated using sparse convolutional layers and convolutional gate recursive units, and finally a dense traversability map is generated through the repair network.
It realizes end-to-end identification of environmental terrain passability, improves efficiency, facilitates the unmanned system to efficiently and safely patrol and navigation in the wild environment, and effectively filters out irrelevant obstacles that do not affect traversability.
Smart Images
Figure CN120107580A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of humanoid robot patrol navigation, and in particular relates to a terrain semantic segmentation method for a humanoid robot in a field environment. Background Art
[0002] There has been a great deal of interest in developing humanoid robots in recent years, but the majority of this work has focused on roads and cities. However, autonomous field humanoid robots that navigate complex natural terrains can benefit a wide range of applications including defense, agriculture, conservation, and search and rescue. In such environments, understanding the traversability of the terrain surrounding the humanoid robot is critical for successful planning and control. Since field terrains are often characterized by rapidly changing ground planes, dense vegetation, overhanging branches, and negative obstacles, determining whether the terrain is traversable from sparse LiDAR data is a challenging problem. In other words, a successful field unmanned system must study the geometric semantic content of its surroundings to determine which terrain is traversable and which is not. Summary of the invention
[0003] The purpose of the present invention is to provide a terrain semantic segmentation method for a humanoid robot in a field environment, so as to realize end-to-end identification of the passability of the environmental terrain, with high efficiency, and facilitate the completion of unmanned system inspection and navigation.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a terrain semantic segmentation method for a humanoid robot in a field environment, comprising the following steps:
[0005] Step S1: Construct a passability dataset,
[0006] Given a dataset of semantically labeled lidar scans, we convert it into a passability dataset by following the following steps.
[0007] Step S1.1: Scanning aggregation,
[0008] For each scan, aggregate it with the past St-1 scan and the future St scan, where St-1 and St are lidar data, and build a larger point set using step size s. Set St to a sufficiently large number to obtain dense traversability information for a large area around the unmanned system. These parameters can be adjusted according to the speed of the humanoid robot and the density of lidar points.
[0009] Step S1.2: Passability Mapping,
[0010] Map semantic class labels to a 4-level passability scale.
[0011] Step S1.3: Point cloud stacking and ground height estimation,
[0012] For each point in the aggregate scan, a downward projection is performed to find its position x, y on the traversability map. Therefore, each x, y position of the map contains a point column. The ground height map is estimated by running an average filter kernel on the lowest z coordinate of the points marked as free and low cost at each x, y position in the map. This height map serves as a reference for the final traversability projection.
[0013] Step S1.4: Passability projection,
[0014] For each point column, a certain threshold is set for the points above the local ground to filter out overhanging obstacles, as they will not collide with the unmanned system.
[0015] Step S2: First, the input LiDAR scan is discretized into a 512×512×31 grid with a resolution of 0.2m. Sparse discretization is performed so that only occupied voxels are retained, each containing a 4D feature It includes the coordinates and intensity averages of the points within the voxel. This sparse voxel grid is fed into a series of sparse convolutional layers that train the z value by the convolution stride, keeping the x and y coordinate sizes unchanged. The output of the sparse convolutional layer is a sparse feature tensor S of size 512×512×C, where C is the feature dimension.
[0016] Step S3: Let the network learn to aggregate sparse feature maps from past LiDAR scans through a convolutional gated recurrent unit (ConvGRU). ConvGRU uses a 2D latent feature map M that shares the same coordinate system and size as the final traversability map. The latent feature map M is updated as:
[0017] M t+1 =ConvGRU(WarpAffine(M t , Δτ t+1 ), S t+1 );
[0018] where Δτ t+1 is the relative transformation of the unmanned system odometer coordinates from t to t+1, and the affine transformation is converted from the previous odometer frame to the current odometer frame so that the potential feature map is transformed from M t and S t+1 Perform spatial alignment where the affine transformation is differentiable to allow gradients to backpropagate through time.
[0019] Step S4: Let the inpainting network utilize local and global context clues to fill in the blank space and obtain the bird's-eye view network. The inpainting network is a fully convolutional network inspired by FCHardNet, which was originally designed for fast image segmentation; it consists of a series of downsampling and upsampling layers with skip connections, which enables it to effectively capture local and global context information to predict the lost content.
[0020] Step S5: Construct a traversable dataset from SemanticKITTI and RELLIS-3D to evaluate the bird’s-eye view network in highway and field scenarios.
[0021] Furthermore, in step S2, the sizes of the x and y channels are kept unchanged.
[0022] Furthermore, the output of the sparse convolutional layer in step S2 is a sparse feature tensor of size 512×512×C, where C is the feature dimension.
[0023] Furthermore, the convolutional gated recursive unit in step S3 uses a 2D latent feature map M, which shares the same coordinate system and size as the traversability map.
[0024] Furthermore, the repair network in step S4 is composed of a series of downsampling and upsampling layers with skip connections.
[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0026] The present invention proposes a new framework to construct the BEV cost map, which has the following advantages: ① aggregating observations that change over time; ② predicting invisible areas in the map; ③ filtering out irrelevant obstacles that do not affect traversability, such as overhanging branches.
[0027] To train the model, the present invention uses past and future labeled LiDAR scans to build a complete 3D semantic point cloud and constructs a ground truth 2D traversability map. The prior art uses a set of foldable cube structures with ground / overhang classification to remove irrelevant overhangs based on their gaps, but this rule-based filtering lacks generalization when it is difficult to estimate accurate ground elevations from sparse LiDAR scans. In contrast, the model of this patent is trained with a ground truth BEV map built from a fully observed and labeled environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.
[0029] Figure 1 An example diagram of a target scene provided by the present invention;
[0030] Figure 2 The network structure diagram of BEVNet provided by the present invention;
[0031] Figure 3 A schematic diagram of the process of generating a traversable data set on SemanticKITTI according to the present invention;
[0032] Figure 4 This is a schematic diagram of the prediction results of the model provided by the present invention for SemanticKITTI;
[0033] Figure 5 A schematic diagram of the prediction results of RELLIS-3D by the model provided by the present invention;
[0034] Figure 6 This is a schematic diagram of actual scene detection by BEVNet provided by the present invention. DETAILED DESCRIPTION
[0035] The technical solution of the present invention will be described clearly and completely in conjunction with the embodiments below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The embodiments of the present invention and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] In the study of semantic map generation for autonomous driving, traversability estimation is formulated as a semantic terrain classification problem. The motivation is to unify semantics and geometry to transform terrain into a single cost ontology. From a semantic perspective, objects such as large rocks and tree trunks are not traversable, while gravel, grass, and bushes are traversable by off-road vehicles with increasing difficulty. From a geometric perspective, overhanging obstacles can be ignored. The traversability of objects of the same semantic category may vary depending on their height (e.g., tall bushes vs. short bushes). To this end, this patent uses a set of discrete traversability levels to facilitate grouping semantic classes according to their traversability, while allowing the traversability level of specific instances to be adjusted based on their geometric structure.
[0037] This patent designs a Bird's Eye View Network (BEVNet), which is a recurrent neural network that directly predicts the terrain category around the unmanned system in the form of a 2D grid through lidar scans. The model has three main parts:
[0038] 1) 3D sparse convolutional sub-network for processing voxelized point clouds;
[0039] 2) Convolutional Gated Recurrent Unit (ConvGRU), which uses convolutional layers in gated recurrent units to aggregate 3D information;
[0040] 3) An efficient 2D convolutional encoder-decoder based on that simultaneously repairs gaps and projects 3D data into a 2D bird's-eye view (BEV) map. To train the model, past and future labeled LiDAR scans are used to construct a complete 3D semantic point cloud, and a ground-truth 2D traversability map is constructed. Some scholars have studied the use of a set of foldable cube structures with ground / overhang classification to remove irrelevant overhangs based on their gaps, but this rule-based filtering lacks generalization when it is difficult to estimate accurate ground elevations from sparse LiDAR scans. In contrast, the model in this patent is trained with a ground-truth BEV map constructed from a fully observed and labeled environment, which allows accurate ground level estimates for learning. The learning of this network can detect and remove overhanging obstacles in sparse LiDAR scans without the need for an explicit filtering mechanism.
[0041] Example 1
[0042] Specifically, the present invention provides a terrain semantic segmentation method for a humanoid robot in a field environment, comprising the following steps:
[0043] Step S1: Constructing a passability dataset
[0044] Similar research focuses on road driving, where reasoning over a large number of fine-grained semantic classes is required. This patent considers a more general driving paradigm where only the traversability of the surrounding terrain is of concern, which makes the model applicable to both road and off-road driving. Given a dataset of semantically labeled LiDAR scans, it is converted into a traversability dataset (e.g. Figure 3 shown).
[0045] Step S1.1: Scanning aggregation,
[0046] For each scan, aggregate it with the past St-1 scan and the future St scan, where St-1 and St are lidar data, and build a larger point set using step size s. Set St to a sufficiently large number to obtain dense traversability information for a large area around the unmanned system. These parameters can be adjusted according to the speed of the humanoid robot and the density of lidar points.
[0047] Step S1.2: Passability mapping,
[0048] Mapping semantic classes into a 4-level traversability hierarchy, the general principle is to map semantic classes with similar costs to the same traversability label, for example, cars and buildings are mapped as dangerous, while dirt and grass are mapped as low cost.
[0049] Step S1.3: Point cloud stacking and ground height estimation,
[0050] For each point in the aggregate scan, a downward projection is performed to find its position x,y on the traversability map, so each x,y position of the map contains a point column, and the ground height map is estimated by running an average filter kernel on the lowest z coordinate of the points marked as free and low cost at each x,y position in the map. This height map serves as a reference for the final traversability projection.
[0051] Step S1.4: Passability projection,
[0052] For each point column, a certain threshold is set for points above the local ground to filter out overhanging obstacles, as they will not collide with the unmanned system. In addition, the passability level of certain points is adjusted according to their height from the ground and the mobility of the unmanned system. For large field humanoid robots, points marked as medium cost but very close to the local ground can be considered negligible, so they are remapped to low-cost points, just like other nearby points. Finally, the minimum traversable point (i.e., the most difficult point) at each x, y position is taken as the final traversability label.
[0053] like Figure 2 As shown in the figure, the lidar scan data is first discretized into a sparse voxel grid, and then the sparse voxel grid is fed into a series of sparse convolutional layers to compress the z value. The compressed sparse feature tensor is aggregated over time through the ConvGRU unit, and a differentiable affine transformation is used to align the latent feature map with the current odometry frame. Finally, the repair network outputs a dense traversability map through the latent map.
[0054] Step 2: The architecture of BEVNet is as follows Figure 2 As shown in Figure 1, the input LiDAR scan is first discretized into a 512×512×31 grid with a resolution of 0.2m, and sparse discretization is performed so that only occupied voxels are retained, each containing a 4D feature It includes the coordinates and the average of the intensity of the points within the voxel. This sparse voxel grid is fed into a series of sparse convolutional layers that train the z value by the convolution stride. Keeping the x and y coordinate dimensions unchanged, the output of the sparse convolutional layer is a sparse feature tensor S of size 512×512×C, where C is the feature dimension.
[0055] Step 3: As the distance increases, individual LiDAR scans become increasingly sparse, making it difficult to classify the traversability level of areas far from the unmanned system. Unlike traditional SLAM, which aggregates LiDAR measurements over time via a manually designed Bayesian update rule, the present invention lets the network learn to aggregate sparse feature maps from past LiDAR scans via a convolutional gated recurrent unit (ConvGRU). ConvGRU uses a 2D latent feature map M that shares the same coordinate system and size as the final traversability map. The latent feature map M is updated as:
[0056] M t+1 =ConvGRU(WarpAffine(M t , Δτ t+1 ), S t+1 );
[0057] where Δτ t+1 is the relative transformation of the unmanned system odometer coordinates from t to t+1, and the affine transformation is converted from the previous odometer frame to the current odometer frame so that the potential feature map is transformed from M t and S t+1 Perform spatial alignment where the affine transformation is differentiable to allow gradients to backpropagate through time.
[0058] Step 4: Since ConvGRU only aggregates sparse feature tensors, the areas without LiDAR points in M contain little information. Instead of treating the unscanned areas as unknown, the inpainting network uses local and global context clues to fill in the empty space. The inpainting network is a fully convolutional network inspired by FCHardNet, which was originally designed for fast image segmentation; it consists of a series of downsampling and upsampling layers with skip connections, which enables it to effectively capture local and global context information to predict the missing content.
[0059] Step 5: Construct traversable datasets from SemanticKITTI and RELLIS-3D to evaluate BEVNet in road and wild scenes. For SemanticKITTI, 71 frames are aggregated with 2 steps to generate a passability map. For RELLIS-3D, 141 frames are aggregated with 5 steps. Both datasets provide per-frame mileage measurements, which are used for the differential affine layer in ConvGRU. The traversability map size is 102.4m×102.4m with a resolution of 0.2m. Among them, the traversability map contains additional "unknown" class labeled areas that have never been observed.
[0060] We use mIoU, a widely used metric in image segmentation, as a quantitative measure of prediction accuracy. Our model predicts an additional “unknown” class to improve visual consistency, but the “unknown” class is excluded from the evaluation. To better understand the model’s ability to predict the future, we report mIoU in three modes: visible, invisible, and all. In the “visible” mode, we do not include the ground truth labels obtained from future frames, effectively excluding any future predictions. For the “unseen” model, we only include predictions for the future. In the “all” scenario, we evaluate both.
[0061] exist Figure 4 and Figure 5 In the figure, BEVNet can better preserve small dynamic objects, such as cyclists, which are easily ignored by hand-designed temporal aggregations as noise. In contrast, BEVNet can learn to preserve small dynamic objects while maintaining smoothness in static areas. The right half shows the impact of noisy odometry, with BEVNet producing significantly cleaner output. Overall, BEVNet shows strong performance in predicting future traversability. It can predict entire vehicles, alley entrances, and trails from extremely sparse lidar points.
[0062] exist Figure 6 In this paper, BEVNet trained on SemanticKITTI and RELLIS-3D can generalize to new environments on a humanoid robot equipped with a 64-line lidar, which is fed into BEVNet for terrain classification. The first environment is a country road, a dirt road with sparse vegetation. The second environment is a jagged road with scattered grass and branches. In both cases, BEVNet is able to predict the complete traversable graph using sparse lidar input and judge the traversability of the surrounding environment using semantic features and geometric features.
[0063] The specific working principle of the method is as follows: in order to enable the unmanned system to navigate efficiently and safely in the wild environment, the unmanned system constructs an online traversable map around it. The traversability map is similar to the traditional occupancy map and semantic map, in which each unit stores the probability distribution of the traversability label; a supervised learning method is used to predict this traversability map; first, a traversability dataset is constructed from the laser radar segmentation dataset through a traversability-aware projection process; then, a BEVNet recursive neural network is designed, the current laser radar scan point cloud data is input, and its accumulation is used to construct a dense traversability map. Four levels of traversability are used (the number of traversability levels can be simply expanded); the traversability map is located in the laser odometer framework of the unmanned system; the traversability map maps each traversability level to the corresponding cost value through a lookup table, thereby converting it into a cost map. The converted cost map of the present invention can be easily connected with a local planning or global planning algorithm (such as A*) to find the lowest cost path to the target. The end-to-end identification of the environmental terrain traversability can be achieved, with high efficiency, and it is convenient to complete the inspection navigation of the unmanned system.
[0064] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A terrain semantic segmentation method for a humanoid robot in a field environment, characterized in that: The following steps are involved: Step S1: Construct a passability dataset, Given a dataset of semantically labeled lidar scans, we convert it into a passability dataset by following the following steps. Step S1.1: Scanning aggregation, For each scan, aggregate it with the past St-1 scan and the future St scan, where St-1 and St are lidar data, and build a larger point set using step size s. Set St to a sufficiently large number to obtain dense traversability information for a large area around the unmanned system. These parameters can be adjusted according to the speed of the humanoid robot and the density of lidar points. Step S1.2: Passability Mapping, Map semantic class labels to a 4-level passability scale. Step S1.3: Point cloud stacking and ground height estimation, For each point in the aggregate scan, a downward projection is performed to find its position x, y on the traversability map. Therefore, each x, y position of the map contains a point column. The ground height map is estimated by running an average filter kernel on the lowest z coordinate of the points marked as free and low cost at each x, y position in the map. This height map serves as a reference for the final traversability projection. Step S1.4: Passability projection, For each point column, a certain threshold is set for the points above the local ground to filter out overhanging obstacles, as they will not collide with the unmanned system. Step S2: First, the input LiDAR scan is discretized into a 512×512×31 grid with a resolution of 0.2m. Sparse discretization is performed so that only occupied voxels are retained, each containing a 4D feature It includes the coordinates and intensity averages of the points within the voxel. This sparse voxel grid is fed into a series of sparse convolutional layers that train the z value by the convolution stride, keeping the x and y coordinate sizes unchanged. The output of the sparse convolutional layer is a sparse feature tensor S of size 512×512×C, where C is the feature dimension. Step S3: Let the network learn to aggregate sparse feature maps from past LiDAR scans through a convolutional gated recurrent unit (ConvGRU). ConvGRU uses a 2D latent feature map M that shares the same coordinate system and size as the final traversability map. The latent feature map M is updated as: M t+1 =ConvGRU(WarpAffine(M t ,Dt t+1 ),S t+1 ); where Δτ t+1 is the relative transformation of the unmanned system odometer coordinates from t to t+1, and the affine transformation is converted from the previous odometer frame to the current odometer frame so that the potential feature map is transformed from M t and S t+1 Perform spatial alignment where the affine transformation is differentiable to allow gradients to backpropagate through time. Step S4: Let the inpainting network utilize local and global context clues to fill in the blank space and obtain the bird's-eye view network. The inpainting network is a fully convolutional network inspired by FCHardNet, which was originally designed for fast image segmentation; it consists of a series of downsampling and upsampling layers with skip connections, which enables it to effectively capture local and global context information to predict the lost content. Step S5: Construct a traversable dataset from SemanticKITTI and RELLIS-3D to evaluate the bird’s-eye view network in highway and field scenarios.
2. The method for terrain semantic segmentation of a humanoid robot in a field environment according to claim 1, characterized in that: In step S2, the sizes of the x and y channels are kept unchanged.
3. The terrain semantic segmentation method for a humanoid robot in a field environment according to claim 1, characterized in that: The output of the sparse convolutional layer in step S2 is a sparse feature tensor of size 512×512×C, where C is the feature dimension.
4. The method for terrain semantic segmentation of a humanoid robot in a field environment according to claim 1, characterized in that: The convolutional gated recurrent unit in step S3 uses a 2D latent feature map M, which shares the same coordinate system and size as the traversability map.
5. The method for terrain semantic segmentation of a humanoid robot in a field environment according to claim 1, characterized in that: The repair network in step S4 is composed of a series of downsampling and upsampling layers with skip connections.