A terrain semantic segmentation method in off-road environment

Abstract

The application discloses a terrain semantic segmentation method in off-road environment, comprising the following steps: S1, converting a given laser radar scanning data set with semantic labels into a passability data set; S2, discretizing the input laser radar scanning point cloud into voxels, each voxel containing a 4-dimensional feature, and sending the sparse voxel grid into a sparse convolution layer to compress the z channel value through pooling convolution; S3, enabling the network to learn to aggregate sparse feature maps from past laser radar scans through convolution gating recurrent units; S4, enabling the repair network to fill in the blank space by using local and global context clues to obtain a bird's eye view network; and S5, evaluating the bird's eye view network in highway and off-road scenes through the passability data set.

Description

Technical Field

[0001] This invention relates to the field of unmanned vehicle patrol and navigation, specifically to a terrain semantic segmentation method for off-road environments. Background Technology

[0002] In recent years, there has been considerable interest in developing autonomous vehicles, but this work has primarily focused on on-road and urban driving. However, autonomous off-road vehicles navigating complex natural terrain can benefit a wide range of applications, including defense, agriculture, conservation, and search and rescue. In such environments, understanding the traversability of the terrain surrounding the vehicle is crucial for successful planning and control. Determining traversability from sparse LiDAR data is challenging due to the typically rapid changes in terrain level, dense vegetation, overhanging branches, and negative obstacles. In other words, a successful off-road unmanned system must study the geometric semantics of its surroundings to determine which terrain features are traversable and which are not. Summary of the Invention

[0003] The purpose of this invention is to provide a terrain semantic segmentation method for off-road environments, so as to achieve end-to-end identification of environmental terrain accessibility, which has high efficiency and facilitates unmanned system inspection and navigation.

[0004] To achieve the above objectives, the present invention provides a terrain semantic segmentation method for off-road environments, comprising:

[0005] Step S1: Convert the given dataset of LiDAR scans with semantic tags into a passability dataset;

[0006] Step S2: Discretize the input LiDAR scan point cloud into voxels. Each voxel contains a 4-dimensional feature. The sparse voxel grid is fed into a sparse convolutional layer to compress the z-channel value through pooling convolution.

[0007] Step S3: Enable the network to learn to aggregate sparse feature maps from past LiDAR scans through convolutional gated recursive units;

[0008] Step S4: Make the repair network fill the blank space using local and global contextual clues to obtain the bird's-eye view network;

[0009] Step S5: Evaluate the bird's-eye view network in highway and off-road scenarios using the accessibility dataset.

[0010] Optionally, in step S1, for each LiDAR scan, it is aggregated with past t-scan data and future t-scan data, and a larger point set is constructed using a step size s, and semantic tags are mapped to a 4-level accessibility level.

[0011] Optionally, step S1 includes:

[0012] Step S11: For each lidar scan, aggregate it with past t-scan data and future t-scan data, construct a larger point set using step size s, and set t to a sufficiently large number to obtain dense traversability information over a large area around the unmanned system.

[0013] Step S12: Map the semantic tags to the 4-level accessibility levels;

[0014] Step S13: For each point in the aggregated scan, project downwards to find its position x and y on the traversability map, and estimate the ground height map by running an average filter kernel on the lowest z coordinate of the points marked as free and low cost at each x and y position on the map.

[0015] Step S14: For each point column, filter out overhanging obstacles by setting a certain threshold for points higher than the local ground level.

[0016] Optionally, in step S2, the x and y channel sizes are kept unchanged.

[0017] Optionally, in step S2, the output of the sparse convolutional layer is a sparse feature tensor of size 512×512×C, where C is the feature dimension.

[0018] Optionally, in step S3, the convolutional gated recursive unit uses a 2D latent feature map M, which shares the same coordinate system and size as the traversability map.

[0019] Optionally, in step S4, the repair network consists of a series of downsampling and upsampling layers with skip connections.

[0020] The beneficial effects of this invention are as follows:

[0021] (1) This patent proposes a new framework for constructing BEV cost maps by: ① aggregating observations that change over time; ② predicting areas that are not visible in the map; ③ filtering out irrelevant obstacles that do not affect traversability, such as overhanging tree branches.

[0022] (2) To train the model, this invention uses past and future labeled LiDAR scans to construct a complete 3D semantic point cloud and a ground-based 2D traversable map. Existing technologies utilize a set of foldable cube structures with ground / cantilever classifications to remove irrelevant cantilevers based on their gaps. However, this rule-based filtering lacks generalization when it is difficult to estimate accurate ground elevations from sparse LiDAR scans. In contrast, the model in this patent is trained using a ground-based BEV map constructed from a fully observed and labeled environment. Attached Figure Description

[0023] Figure 1 Example diagram of the target scenario provided by the present invention.

[0024] Figure 2 The network structure diagram of BEVNet provided by this invention.

[0025] Figure 3 This is a schematic diagram illustrating the process of generating an iterable dataset on SemanticKITTI according to the present invention.

[0026] Figure 4 This is a schematic diagram illustrating the prediction results of the model provided by this invention for SemanticKITTI.

[0027] Figure 5 This is a schematic diagram illustrating the prediction results of the model provided by this invention for RELLIS-3D.

[0028] Figure 6 This is a schematic diagram of BEVNet used for real-world scene detection, as provided by the present invention. Detailed Implementation

[0029] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In research on semantic map generation for autonomous driving, traversability estimation is formulated as a semantic terrain classification problem. The motivation is to unify semantics (what objects exist?) and geometry (where are the objects located?) to transform terrain into a single-cost ontology. Semantically, objects such as large rocks and tree trunks are impassable, while gravel, grass, and shrubs are traversable by off-road vehicles, but the difficulty increases progressively. Geometrically, overhanging obstacles can be ignored, and the traversability of objects of the same semantic category may vary due to their height (e.g., tall shrubs versus short shrubs). To address this, this patent uses a set of discrete traversability levels to facilitate grouping semantic classes based on their traversability, while allowing the traversability level of specific instances to be adjusted according to their geometry.

[0031] This invention discloses a terrain semantic segmentation method for off-road environments. The method includes: to enable an unmanned system (UHV) to navigate efficiently and safely in off-road environments, the UHV constructs an online traversability map around itself. This traversability map is similar to a traditional occupancy map and semantic map, where each cell stores the probability distribution of traversability labels; a supervised learning method is used to predict this traversability map; firstly, a traversability dataset is constructed from a LiDAR segmentation dataset through a traversability-aware projection process; then, a BEVNet recurrent neural network is designed, taking the current LiDAR scan point cloud data as input and utilizing its cumulative values ​​to construct a dense traversability map. Four levels of traversability are used (the number of traversability levels can be easily expanded); the traversability map is located within the LiDAR odometry framework of the UHV; the traversability mapping maps each traversability level to a corresponding cost value through a lookup table, thereby converting it into a cost map. The converted cost map of this invention can be easily connected to local or global programming algorithms (such as A*) to find the lowest-cost path to the target. It can achieve end-to-end identification of environmental terrain accessibility, has high efficiency, and facilitates unmanned system inspection and navigation.

[0032] The Bird's-Eye Network (BEVNet) designed in this patent is a recurrent neural network that directly predicts the terrain category around an unmanned system in the form of a 2D grid through LiDAR scanning. The model has three main parts:

[0033] 1) 3D sparse convolutional subnetwork for processing voxelized point clouds;

[0034] 2) Convolutional Gated Recursive Unit (ConvGRU), which uses convolutional layers in the gated recursive unit to aggregate 3D information;

[0035] 3) A highly efficient 2D convolutional encoder-decoder is used to simultaneously fill in gaps and project 3D data onto a 2D bird's-eye view (BEV) map. To train the model, past and future labeled LiDAR scans are used to construct a complete 3D semantic point cloud and build a ground-based 2D traversability map. Some researchers have used a set of foldable cube structures with ground / cantilever classifications to remove irrelevant cantilevers based on their gaps, but this rule-based filtering lacks generalization when it is difficult to estimate accurate ground elevation from sparse LiDAR scans. In contrast, this patented model is trained using a ground-based BEV map constructed from a fully observed and labeled environment, allowing accurate ground level estimation to be used for learning. The network can learn to detect and remove overhanging obstacles in sparse LiDAR scans without explicit filtering mechanisms.

[0036] Specifically, this invention provides a terrain semantic segmentation method for off-road environments, comprising the following steps:

[0037] Step 1: Construct the accessibility dataset

[0038] Similar research focuses on road driving, which requires reasoning over a large number of fine-grained semantic classes. This patent considers a more general driving paradigm, concerned only with the traversability of the surrounding terrain, making the model applicable to both highway and off-road driving. Given a dataset of LiDAR scans with semantic labels, it is transformed into a traversability dataset (e.g., ...) through the following process. Figure 3 (As shown).

[0039] 1) Scan Aggregation

[0040] For each scan, it is aggregated with past t scans and future t scans, and a larger set of points is constructed using a step size s. t is set to a sufficiently large number to obtain dense traversability information over a large area around the unmanned system. These parameters can be adjusted according to vehicle speed and the density of LiDAR points.

[0041] 2) Accessibility mapping

[0042] Semantic classes are mapped to four accessibility levels. The general principle is to map semantic classes with similar costs to the same traversability label. For example, cars and buildings are mapped to dangerous, while dirt and grass are mapped to low cost.

[0043] 3) Point cloud stacking and ground height estimation

[0044] For each point in the aggregated scan, a downward projection is performed to find its x, y position on the traversability map. Therefore, each x, y position on the map contains a point column. The ground elevation map is estimated by running an average filter kernel on the lowest z-coordinate of the points marked as free and low-cost at each x, y position on the map. This elevation map serves as a reference for the final traversability projection.

[0045] 4) Permissible projection

[0046] For each point pillar, a threshold is set for points above the local ground level to filter out overhanging obstacles, as they will not collide with the unmanned system. Furthermore, the accessibility level of these points is adjusted based on their height above the ground and the mobility of the unmanned system. For large off-road vehicles, points marked as medium-cost but very close to the local ground level can be considered negligible and are therefore remapped as low-cost points, like other nearby points. Finally, the minimum traversable point (i.e., the most difficult point) at each x, y position is used as the final traversability label.

[0047] The passability dataset constructed in step S1 includes a test dataset and a training dataset. The training dataset is used in steps S2-S4 to train the model; the test dataset is used in step S5 to evaluate the model.

[0048] like Figure 2 As shown, the LiDAR scan data is first discretized into a sparse voxel grid, and then the sparse voxel grid is fed into a series of sparse convolutional layers to compress the z-value. The compressed sparse feature tensors are aggregated over time using ConvGRU units, and a differentiable affine transformation is used to align the latent feature map with the current odometry frame. Finally, the repair network outputs a dense ergodicity map through latent mapping.

[0049] Step 2: The architecture of BEVNet is as follows Figure 2 As shown. First, the input LiDAR scan is discretized into a 512×512×31 grid with a resolution of 0.2m. Sparse discretization is then performed to retain only the occupied voxels. Each voxel contains a 4-dimensional feature. It includes the coordinates of points within the voxels and the average intensity. This sparse voxel grid is fed into a series of sparse convolutional layers, which are trained with z-values ​​using convolution strides. The x and y coordinate dimensions are kept constant. The output of the sparse convolutional layers is a sparse feature tensor S of size 512 × 512 × C, where C is the feature dimension.

[0050] Step 3: As distance increases, individual LiDAR scans become increasingly sparse, making it difficult to classify the traversability level of areas far from the unmanned system. Unlike traditional SLAM, which aggregates LiDAR measurements over time using a manually designed Bayesian update rule, this invention allows the network to learn to aggregate sparse feature maps from past LiDAR scans via a Convolutional Gated Recursive Unit (ConvGRU). The ConvGRU uses a 2D latent feature map M, which shares the same coordinate system and size as the final traversability map. The latent feature map M is updated to...

[0051] M t+1 =ConvGRU(WarpAffine(M t ,Δτ t+1 ), S t+1 (1)

[0052] Where Δτ t+1 This is a relative transformation of the odometry coordinates of the unmanned system from t to t+1. The affine transformation converts the previous odometry frame to the current odometry frame so that the latent feature map changes from M... t and S t+1 Spatial alignment is performed, where the affine transformation is differentiable to allow gradients to propagate backward over time.

[0053] Step 4: Since ConvGRU only aggregates sparse feature tensors, areas in M ​​without LiDAR points contain very little regional information. Instead of treating unscanned areas as unknown, the inpainting network utilizes local and global contextual cues to fill in the blank spaces. The inpainting network is a fully convolutional network inspired by FCHardNet, originally designed for fast image segmentation; it consists of a series of downsampling and upsampling layers with skip connections, enabling it to effectively capture local and global contextual information to predict missing content.

[0054] Step 5: Construct traversable datasets from SemanticKITTI and RELLIS-3D to evaluate BEVNet in on-road and off-road scenarios. For SemanticKITTI, aggregate 71 frames with a 2-step stride to generate a traversability map. For RELLIS-3D, aggregate 141 frames with a 5-step stride. Both datasets provide per-frame mileage measurements, which are used in differential affine layers in ConvGRU. The traversability map is 102.4m × 102.4m in size with a resolution of 0.2m. The traversability map includes additional "unknown" class-labeled regions that have never been observed before.

[0055] The model uses mIoU, a metric widely used in image segmentation, as a quantitative measure of prediction accuracy. This patent's model predicts an additional "unknown" class to improve visual consistency, but excludes the "unknown" class in the evaluation. To better understand the model's ability to predict the future, mIoU is reported in three modes: visible, invisible, and all. In the "visible" mode, ground truth labels obtained from future frames are not included, effectively excluding any future predictions. For the "invisible" model, only predictions for the future are included. In the "all" scenario, both are evaluated.

[0056] exist Figure 4 and Figure 5 In the comparison, BEVNet better preserves small dynamic objects, such as cyclists. Manually designed temporal aggregations tend to treat these small dynamic objects as noise and ignore them. In contrast, BEVNet can learn to preserve small dynamic objects while maintaining the smoothness of static regions. The right half shows the impact of noisy odometers; BEVNet's output is significantly cleaner. Overall, BEVNet demonstrates strong performance in predicting future traversability. It can predict entire vehicles, alleyway entrances, and walkways using extremely sparse LiDAR points.

[0057] exist Figure 6In this study, BEVNet, trained on SemanticKITTI and RELLIS-3D, can be generalized to new environments on a robot equipped with a 64-line LiDAR. The LiDAR data is fed into BEVNet for terrain classification. The first environment is a rural road, a sparsely vegetated dirt track. The second environment is a jagged road with scattered grass and branches. In both cases, BEVNet is able to predict a complete traversable map using the sparse LiDAR input and determine the traversability of the surrounding environment using semantic and geometric features.

[0058] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A terrain semantic segmentation method for off-road environments, characterized in that, include: Step S1: Convert the given dataset of LiDAR scans with semantic tags into a passability dataset; Step S2: Discretize the input LiDAR scan point cloud into voxels, with each voxel containing a 4-dimensional feature. It includes the coordinates of points within the voxels and the average intensity. The sparse voxel mesh is fed into a sparse convolutional layer to compress the z-channel values ​​through pooling convolution. Step S3: Enable the network to learn to aggregate sparse feature maps from past LiDAR scans through convolutional gated recursive units; Step S4: Make the repair network fill the blank space using local and global contextual clues to obtain the bird's-eye view network; Step S5: Evaluate the bird's-eye view network in highway and off-road scenarios using the accessibility dataset; S1 includes: Step S11: For each lidar scan, aggregate it with past t-scan data and future t-scan data, and use step size s to construct a larger point set to obtain dense traversability information over a large area around the unmanned system. Step S12: Map the semantic tags to the 4-level accessibility levels; Step S13: For each point in the aggregated scan, project downwards to find its position x and y on the traversability map, and estimate the ground height map by running an average filter kernel on the lowest z coordinate of the points marked as free and low cost at each x and y position on the map. Step S14: For each point column, filter out overhanging obstacles by setting a threshold for points above the local ground level.

2. The terrain semantic segmentation method in an off-road environment as described in claim 1, characterized in that, In step S2, the x and y channel sizes remain unchanged.

3. The terrain semantic segmentation method in an off-road environment as described in claim 1, characterized in that, In step S2, the output of the sparse convolutional layer is a sparse feature tensor of size 512×512×C, where C is the feature dimension.

4. The terrain semantic segmentation method in an off-road environment as described in claim 1, characterized in that, In step S3, the convolutional gated recursive unit uses a 2D latent feature map M, which shares the same coordinate system and size as the traversability map.

5. The terrain semantic segmentation method in an off-road environment as described in claim 1, characterized in that, In step S4, the repair network consists of a series of downsampling and upsampling layers with skip connections.

Images