Terrain trafficability evaluation method and system suitable for ground unmanned vehicle in unstructured environment

Abstract

The invention discloses a terrain trafficability evaluation method and system suitable for a ground unmanned vehicle in an unstructured environment, and belongs to the technical field of terrain trafficability analysis. The method comprises the following steps: acquiring image data and laser radar point cloud data based on data acquisition equipment, and performing feature extraction on the image data and the laser radar point cloud data by adopting a pre-training model to obtain an image feature map and a point cloud feature map; the data acquisition equipment comprises a camera and a laser radar; performing feature fusion on the image feature map and the point cloud feature map to obtain a multi-modal feature map; performing spatial conversion on the multi-modal feature map to obtain a converted multi-modal feature map; and performing classification calculation on the converted multi-modal feature map to obtain an elevation map and a trafficability map, and taking the trafficability map as an output result of final terrain trafficability evaluation. The method is suitable for unstructured scenes with complex terrain types and frequent environment dynamic changes.

Description

Technical Field

[0001] The present invention belongs to the technical field of terrain traversability analysis, and in particular relates to a terrain traversability assessment method and system suitable for ground unmanned vehicles in unstructured environments. Background Art

[0002] Performing tasks in unstructured environments, such as post-disaster rescue, agricultural and forestry operations, mining transportation, and military reconnaissance, has garnered increasing attention in recent years. These tasks operate in complex and varied environments, often lacking clear road markings or road structures, and with complex terrain conditions, including gravel, slopes, woodlands, wetlands, and other types. In such scenarios, accurate assessment of terrain traversability is a key component in enabling autonomous vehicle navigation and path planning. Accurately identifying areas where a vehicle can safely navigate is a prerequisite for ensuring smooth and safe mission execution. Therefore, research on terrain traversability assessment technology applicable to unstructured environments is crucial for improving the environmental adaptability, mission execution efficiency, and operational safety of autonomous vehicles.

[0003] Currently, research on terrain accessibility assessment mainly relies on modeling environmental perception data obtained by on-board sensors (such as lidar, RGB-D cameras, inertial measurement units, etc.), and then performing geometric or semantic analysis on the modeling results. Environmental models represented in the form of digital elevation maps, point clouds, or semantically segmented images have been widely used in the industry. Related methods usually extract indicators such as terrain roughness and slope from a geometric perspective, or identify different surface types such as ground, grass, rocks, and water bodies from a semantic perspective, and then evaluate the difficulty of a certain area for operating vehicles to pass. However, the effectiveness of these methods depends to a large extent on the selected evaluation indicators and the corresponding threshold settings, and their adaptability to the complex, dynamic, and diverse terrains in actual environments is relatively limited.

[0004] Therefore, there is an urgent need for a terrain accessibility assessment method that can more comprehensively integrate multi-source information and adapt to environmental changes, so as to improve the accessibility judgment ability of autonomous operating vehicles in complex mission scenarios. Summary of the Invention

[0005] This invention addresses the technical problem of how to evaluate the cost and risk of a ground unmanned vehicle passing through the surrounding terrain by fusing multi-sensor information, neural network reasoning and other perception methods when a ground unmanned vehicle is traveling in an unstructured environment, and ultimately generate a traversability map. This invention proposes a terrain traversability assessment method and system suitable for ground unmanned vehicles in unstructured environments.

[0006] To achieve the above object, the present invention provides the following solution: a method for evaluating the terrain passability of a ground unmanned vehicle in an unstructured environment, comprising the following steps:

[0007] S1. Collect image data and lidar point cloud data based on a data acquisition device, and use a pre-trained model to extract features from the image data and the lidar point cloud data to obtain an image feature map and a point cloud feature map; the data acquisition device includes: a camera and a lidar;

[0008] S2. performing feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map;

[0009] S3. performing spatial transformation on the multimodal feature map to obtain a transformed multimodal feature map;

[0010] S4. Classify and calculate the converted multimodal feature map to obtain an elevation map and a traversability map, and use the traversability map as the output result of the final terrain traversability assessment.

[0011] Further preferably, S2 includes the following steps:

[0012] S21, performing a convolution operation on the image feature map and the point cloud feature map to obtain a first three-dimensional feature tensor and a second three-dimensional feature tensor;

[0013] S22. Concatenate the first three-dimensional feature tensor and the second three-dimensional feature tensor to obtain a fused feature map;

[0014] S23. Perform deep processing on the fused feature map to obtain the multimodal feature map.

[0015] Further preferably, the method of performing deep processing on the fused feature map to obtain the multimodal feature map includes:

[0016] Q=F fused W Q ,K=F fused W K ,V=F fused W V ,

[0017]

[0018] Where, Indicates that the feature map F will be fused fused The query, key, and value vectors obtained after flattening, W Q ,W K ,W V is a learnable linear projection matrix; d represents the attention dimension; H and W represent the fusion feature map F fused height and width; F SA Represents a multimodal feature map.

[0019] Further preferably, S3 includes the following steps:

[0020] S31. Construct a voxel grid in a three-dimensional world coordinate system, and project the three-dimensional center coordinates of each voxel onto the image plane based on the intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the lidar to obtain pixel coordinates;

[0021] S32, sampling the multimodal feature map based on the pixel coordinates to obtain voxel features;

[0022] S33: Perform dimensionality reduction processing on the voxel features to obtain the converted multimodal feature map.

[0023] Further preferably, the method for obtaining the elevation map and the accessibility map in S4 includes: inputting the converted multimodal feature map into a U-shaped network for processing to obtain a U-shaped feature map; inputting the U-shaped feature map into a dual-branch network to obtain the elevation map and the accessibility map.

[0024] Further preferably, the U-shaped network includes a downsampling layer, a residual enhancement module and an upsampling layer;

[0025] The downsampling layer reduces the spatial resolution of the converted multimodal feature map through continuous convolution and pooling operations;

[0026] The residual enhancement module is connected through stacked convolution and short-circuited pathways;

[0027] The upsampling layer restores the spatial resolution of the converted multimodal feature map through bilinear interpolation or deconvolution operation to obtain the U-shaped feature map.

[0028] Further preferably, the dual-branch network includes a first independent branch and a second independent branch; the first independent branch is used to generate the elevation map and the elevation confidence map; the elevation confidence map is used to measure the confidence level of the prediction result in a specific area;

[0029] The second independent branch is used to generate the passability map and the passability confidence map; the passability confidence map is used to assist in improving decision reliability.

[0030] The present invention also provides a terrain accessibility assessment system suitable for ground unmanned vehicles in unstructured environments, comprising: a data processing system, a fusion processing system, a space conversion system, and an assessment system;

[0031] The data processing system is used to extract features from the image data and the laser radar point cloud data collected by the data acquisition device using a pre-trained model to obtain an image feature map and a point cloud feature map; the data acquisition device includes: a camera and a laser radar;

[0032] The fusion processing system is used to perform feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map;

[0033] The spatial conversion system is used to perform spatial conversion on the multimodal feature map to obtain a converted multimodal feature map;

[0034] The evaluation system is used to perform classification calculations on the converted multimodal feature map to obtain an elevation map and a traversability map, and use the traversability map as the output result of the final terrain traversability evaluation.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This paper proposes a multimodal map construction framework that integrates vision and lidar point clouds. Based on the input image and point cloud, a neural network is used to extract and fuse features, and simultaneously predicts and generates elevation maps and accessibility maps around the vehicle. It is suitable for unstructured scenarios with complex terrain types, uncertain obstacle distribution, and frequent dynamic environmental changes.

[0037] The present invention adopts a multi-task learning method that introduces uncertainty modeling, and simultaneously trains two sub-tasks: terrain accessibility assessment and confidence estimation. This enables the model to adaptively allocate the importance of different tasks while sharing feature expressions, which helps to improve the model's adaptability to blurred, occluded or interfered areas, and enhances the system's reliability in accessibility analysis under complex environments and with high noise in perception data.

[0038] This paper introduces a large-scale visual pre-training model as an image feature extractor, and utilizes its strong representation ability acquired through training in diverse semantic scenarios, so that the system can still stably extract discriminative deep features in unfamiliar or unseen environments, improve the stability and accuracy of the accessibility assessment method in cross-scenario applications, and enhance the generalization ability of overall perception.

[0039] The present invention introduces a self-attention mechanism to achieve precise alignment and fusion of key features between images and lidar point clouds, fully tapping into the complementary information in the two perception modalities and improving the representation capability of fused features, thereby improving the accuracy of terrain accessibility judgment and environmental adaptability in key areas such as obstacle edges and complex height differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a method for terrain accessibility assessment for a ground unmanned vehicle in an unstructured environment according to an embodiment of the present invention;

[0042] Figure 2 This is an overall framework diagram of a terrain accessibility assessment method for a ground unmanned vehicle in an unstructured environment according to an embodiment of the present invention;

[0043] Figure 3 This is a framework diagram of the feature fusion stage of an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of a framework for obtaining a traversability map through a multimodal feature map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1:

[0048] like Figure 1 、 Figure 2 As shown, this embodiment provides a terrain accessibility assessment method applicable to a ground unmanned vehicle in an unstructured environment, comprising the following steps:

[0049] S1. Collect image data and lidar point cloud data based on data acquisition equipment, and use a pre-trained model to extract features from the image data and lidar point cloud data to obtain image feature maps and point cloud feature maps; the data acquisition equipment includes: cameras and lidars.

[0050] The input data of this embodiment relies on the collection of cameras and lidars installed on the vehicle, which form the basis for subsequent multimodal feature construction and map generation. The camera should be installed in a position to ensure that its field of view covers the direction of the vehicle's movement, and the imaging range should include the surface of the road ahead and the space above the ground where obstacles may exist, so as to fully obtain environmental information. The lidar adopts a rotating structure and is installed horizontally at the highest point of the vehicle body to ensure 360-degree field of view coverage and enhance the overall and accuracy of obstacle detection. In order to meet the needs of three-dimensional structure perception, the lidar harness used should be no less than 32 lines, thereby improving the integrity and resolution of the point cloud data.

[0051] The collected image data and lidar point cloud data are then input into a pre-trained model that has undergone large-scale semantic learning to extract semantic and spatial features. Both the image feature map and the lidar point cloud feature map are in the form of three-dimensional tensors (i.e., feature maps), containing spatial dimension and channel dimension information. The selected pre-trained model is based on a self-supervised learning strategy and is trained on a large-scale, highly heterogeneous dataset. It has good cross-domain generalization capabilities and rich semantic expression capabilities. For example, the DINOv2 model can be used to extract image features. This model obtains robust and structured visual semantic information through knowledge distillation and self-attention mechanisms; for lidar point cloud feature extraction, open source models such as OpenShape can be used. It achieves effective encoding of high-dimensional spatial structural information through joint alignment learning of large-scale point cloud-text pairing data. This type of pre-trained model can provide high-quality abstract semantic representation for subsequent multimodal fusion and downstream tasks.

[0052] S2. Perform feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map.

[0053] Considering the complementarity of different modal feature maps from images and point clouds, the self-attention mechanism is used in the feature fusion stage to enhance the fusion effect. Specifically, S2 includes the following steps:

[0054] S21, image feature map F img And point cloud feature map F pc A convolution operation is performed to unify the spatial resolution and obtain a first three-dimensional feature tensor and a second three-dimensional feature tensor with the same width and height.

[0055]

[0056] Where, Represents the convolution operation; H, W, C represent F' img and F′ pc The length, width, and number of channels of the feature map.

[0057] S22. Concatenate the first three-dimensional feature tensor and the second three-dimensional feature tensor to obtain a fused feature map containing multimodal information.

[0058]

[0059] Where Concat represents the concatenation process along the channel dimension.

[0060] S23. Introduce a feature fusion network based on the self-attention mechanism to perform deep processing on the fusion feature map to capture the long-distance dependency and contextual association between different modal features, thereby improving the information complementarity effect and obtaining a multimodal feature map with stronger semantic expression ability. Figure 3 shown.

[0061] Q=F fused W Q , K=F fused W K , V=F fused W V , (3)

[0062]

[0063] Where, Indicates that the feature map F will be fused fused The query, key, and value vectors obtained after flattening, W Q ,W K ,W V is a learnable linear projection matrix; d represents the attention dimension; H and W represent the fusion feature map F fused height and width; F SA Represents a multimodal feature map.

[0064] S3. Perform spatial transformation on the multimodal feature map to obtain a transformed multimodal feature map.

[0065] In a further implementation, S3 includes the following steps:

[0066] S31. Construct a voxel grid in a three-dimensional world coordinate system, and project the three-dimensional center coordinates of each voxel onto the image plane based on the intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the lidar to obtain pixel coordinates.

[0067] The fused feature map processed by the self-attention mechanism is converted from pixel space to BEV (Bird's Eye View) space to improve the efficiency of terrain understanding and environmental analysis. The specific process includes the following steps: First, a regular voxel grid is constructed in the 3D world coordinate system. The 3D center coordinates of each voxel are projected onto the image plane by combining the intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the lidar to obtain the corresponding pixel coordinates. The relevant projection relationship is as follows:

[0068]

[0069] Where K is the camera intrinsic parameter matrix; T is the extrinsic parameter matrix from the camera to the lidar; (u, v, depth) is the pixel coordinate and depth information; (x, y, z) is the three-dimensional coordinate of the voxel.

[0070] S32. Sampling the multimodal feature map based on pixel coordinates to obtain voxel features.

[0071] Based on the above mapping relationship, sampling is performed in the multimodal feature map, and the two-dimensional pixel features are "lifted" to the three-dimensional voxel space, thereby realizing the conversion from image frames to three-dimensional representation:

[0072] T voxel (x,y,z)=S(F EEV ,(u(x,y,z),v(x,y,z))), (7)

[0073] Where S represents the bilinear interpolation sampling function; (u(x,y,z),v(x,y,z)) represents the voxel T with coordinates (x,y,z) voxel The pixel coordinates obtained after projection, F BEV Represents the converted multimodal feature map.

[0074] S33. Perform dimensionality reduction processing on the voxel features to obtain a converted multimodal feature map.

[0075] Finally, a 3D convolution operation is used to structurally model the voxel features and explore their local spatial relationships. Dimensionality reduction is then used to compress the 3D features into a 2D plane, completing the feature conversion to BEV space. This step lays the spatial prior foundation for subsequent ground-based environmental understanding tasks. The conversion method is as follows:

[0076]

[0077] Where, Represents a 3D convolution operation, Agg z Represents the maximum pooling operation.

[0078] S4. Classify and calculate the converted multimodal feature map to obtain an elevation map and a traversability map, and use the traversability map as the output result of the final terrain traversability assessment.

[0079] A further implementation is that Figure 4 As shown, the method for obtaining an elevation map and a accessibility map includes: inputting the converted multimodal feature map into a U-shaped network for processing to obtain a U-shaped feature map; inputting the U-shaped feature map into a dual-branch network to obtain an elevation map and a accessibility map.

[0080] Specifically, the two-dimensional feature map constructed in the BEV space (the converted multimodal feature map) will be further processed through the U-Net structure to obtain a U-shaped feature map; in order to achieve comprehensive perception and modeling of multi-scale semantic information and spatial details in the terrain environment. The U-Net architecture consists of three parts: the downsampling layer, the residual enhancement module, and the upsampling layer: the downsampling layer gradually reduces the spatial resolution of the feature map through continuous convolution and pooling operations, and extracts more advanced abstract features at the semantic level; the residual module strengthens the gradient flow through stacked convolutions and skip connections, and improves the feature propagation and generalization capabilities; the upsampling layer gradually restores the spatial resolution through operations such as bilinear interpolation or deconvolution, and introduces low-order detail features from the downsampling symmetric layer, thereby achieving layer-by-layer restoration and multi-scale fusion of spatial information. This network structure can effectively model terrain features of different scales in the environment, providing stable feature support for elevation estimation and accessibility analysis.

[0081] This embodiment ultimately outputs two types of environmental understanding results: an elevation map and a traversability map, which correspond to the modeling and evaluation of terrain geometry and traversability conditions, respectively. It also generates a related confidence map to indicate the reliability of the prediction.

[0082] A further implementation is to input the U-shaped feature map into a two-branch network, which includes a first independent branch and a second independent branch; the first independent branch is used to generate an elevation map and an elevation confidence map; the elevation map can provide detailed elevation information for each ground grid unit; the elevation confidence map is used to measure the confidence level and uncertainty of the prediction results in a specific area; the second independent branch is oriented to passability judgment, and is used to generate a passability map and a passability confidence map; the passability map is used to evaluate whether the corresponding area has the ability to pass (such as driving in, walking, etc.), and simultaneously generate the corresponding passability confidence map, to help improve the robustness of the system and the reliability of task-driven decision-making. This output structure is convenient for direct integration into autonomous driving systems, high-precision map construction or outdoor robot path planning modules, providing rich data support for environmental cognition and navigation. The passability confidence map is used to help improve the robustness of the system and the reliability of task-driven decision-making.

[0083] Example 2:

[0084] This embodiment provides a terrain accessibility assessment system suitable for ground unmanned vehicles in unstructured environments, including: a data processing system, a fusion processing system, a spatial conversion system and an assessment system; the data processing system is used to use a pre-trained model to extract features from image data and lidar point cloud data collected by a data acquisition device to obtain an image feature map and a point cloud feature map; the data acquisition device includes: a camera and a lidar; the fusion processing system is used to perform feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map; the spatial conversion system is used to perform spatial conversion on the multimodal feature map to obtain a converted multimodal feature map; the assessment system is used to perform classification calculations on the converted multimodal feature map to obtain an elevation map and a accessibility map, and the accessibility map is used as the output result of the final terrain accessibility assessment.

[0085] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for evaluating the terrain passability of a ground unmanned vehicle in an unstructured environment, characterized in that: The following steps are involved: S1. Collect image data and lidar point cloud data based on a data acquisition device, and use a pre-trained model to extract features from the image data and the lidar point cloud data to obtain an image feature map and a point cloud feature map; The data acquisition equipment includes: a camera and a laser radar; S2. performing feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map; S3. performing spatial transformation on the multimodal feature map to obtain a transformed multimodal feature map; S4. Classify and calculate the converted multimodal feature map to obtain an elevation map and a traversability map, and use the traversability map as the output result of the final terrain traversability assessment.

2. The method for terrain accessibility assessment for unmanned ground vehicles in unstructured environments according to claim 1, characterized in that: S2 includes the following steps: S21, performing a convolution operation on the image feature map and the point cloud feature map to obtain a first three-dimensional feature tensor and a second three-dimensional feature tensor; S22. Concatenate the first three-dimensional feature tensor and the second three-dimensional feature tensor to obtain a fused feature map; S23. Perform deep processing on the fused feature map to obtain the multimodal feature map.

3. The method for terrain accessibility assessment for unmanned ground vehicles in unstructured environments according to claim 2, wherein: The method of performing deep processing on the fused feature map to obtain the multimodal feature map includes: Q=F fused W Q ,K=F fused W K ,V=F fused W V , Where, Indicates that the feature map F will be fused fused The query, key, and value vectors obtained after flattening, W Q ,W K ,W V is a learnable linear projection matrix; d represents the attention dimension; H and W represent the fusion feature map F fused height and width; F SA Represents a multimodal feature map.

4. The method for terrain accessibility assessment for unmanned ground vehicles in unstructured environments according to claim 1, wherein: S3 includes the following steps: S31. Construct a voxel grid in a three-dimensional world coordinate system, and project the three-dimensional center coordinates of each voxel onto the image plane based on the intrinsic parameter matrix of the camera and the extrinsic parameter matrix of the lidar to obtain pixel coordinates; S32, sampling the multimodal feature map based on the pixel coordinates to obtain voxel features; S33: Perform dimensionality reduction processing on the voxel features to obtain the converted multimodal feature map.

5. The method for terrain accessibility assessment for unmanned ground vehicles in unstructured environments according to claim 1, wherein: The method for obtaining the elevation map and the accessibility map in S4 includes: inputting the converted multimodal feature map into a U-shaped network for processing to obtain a U-shaped feature map; inputting the U-shaped feature map into a dual-branch network to obtain the elevation map and the accessibility map.

6. The method for evaluating terrain accessibility for unmanned ground vehicles in unstructured environments according to claim 5, wherein: The U-shaped network includes a downsampling layer, a residual enhancement module and an upsampling layer; The downsampling layer reduces the spatial resolution of the converted multimodal feature map through continuous convolution and pooling operations; The residual enhancement module is connected through stacked convolution and short-circuited pathways; The upsampling layer restores the spatial resolution of the converted multimodal feature map through bilinear interpolation or deconvolution operation to obtain the U-shaped feature map.

7. The method for evaluating terrain accessibility for unmanned ground vehicles in unstructured environments according to claim 5, wherein: The dual-branch network includes a first independent branch and a second independent branch; the first independent branch is used to generate the elevation map and the elevation confidence map; the elevation confidence map is used to measure the confidence level of the prediction result in a specific area; The second independent branch is used to generate the passability map and the passability confidence map; the passability confidence map is used to assist in improving decision reliability.

8. A terrain accessibility assessment system for unmanned ground vehicles in unstructured environments, the system being used to implement the method according to any one of claims 1 to 7, characterized in that: include: Data processing system, fusion processing system, space conversion system and evaluation system; The data processing system is used to extract features from the image data and the laser radar point cloud data collected by the data acquisition device using a pre-trained model to obtain an image feature map and a point cloud feature map; The data acquisition equipment includes: a camera and a laser radar; The fusion processing system is used to perform feature fusion on the image feature map and the point cloud feature map to obtain a multimodal feature map; The spatial conversion system is used to perform spatial conversion on the multimodal feature map to obtain a converted multimodal feature map; The evaluation system is used to perform classification calculations on the converted multimodal feature map to obtain an elevation map and a traversability map, and use the traversability map as the output result of the final terrain traversability evaluation.

Images