Device and method for estimating the traverseability of a terrain using a mobile system
The terrain mapping device on mobile systems uses depth and height maps with semantic segmentation to create traversability maps, leveraging deep learning, addressing the challenge of identifying traversable areas in unstructured environments.
Patent Information
- Application Number
- FR2022008427
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-08-22
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-08-22
AI Technical Summary
Existing navigation technologies fail to effectively identify traversable areas in unstructured environments, particularly off-road paths and terrain, beyond traditional road segmentation, and lack comprehensive traversability estimation.
A terrain mapping device embedded on a mobile system uses a traversability estimation unit to determine depth and height maps from camera data, combined with semantic segmentation, to create a traversability map by merging geometric and semantic masks, leveraging a deep learning model and neural networks for efficient terrain analysis.
The solution provides accurate and efficient traversability estimation, refining the identification of traversable areas by integrating geometric and semantic information, effectively addressing the challenges of unstructured environments.
Smart Images

Figure 00000013_0000 
Figure 00000013_0001 
Figure 00000014_0000
Abstract
Description
Title of the invention: Device and method for estimating the traversability of a terrain by a mobile system Technical field
[0001] The field of the invention is that of navigation assistance for a mobile system of the robot or autonomous vehicle type. The invention relates more particularly to the estimation of the traversability of a terrain by the mobile system, that is to say the identification of areas of the terrain on which the mobile system is able to move. Prior art
[0002] In the field of mobile system navigation, methods are known which aim to find the presence of a road in images acquired by a camera on board the mobile system. These methods use visual clues such as vanishing points, textures or even relief to delineate the contours of a road in an image, or pose the problem directly as a problem of segmentation of the road in the image. However, these methods are not interested in paths in the broadest sense of the term, which may in particular be off-road paths not necessarily paved or correctly delineated, and even less so in the more general theme of traversability corresponding to the identification in the acquired images of areas of the terrain on which the mobile system is able to move.
[0003] There are also methods that identify the type of ground on which the vehicle is moving. For example, document US 2019 / 0129435 A1 aims to predict the traversability of a terrain for a vehicle by means of sensors embedded in the vehicle such as an inertial unit, a GPS as well as a lidar to identify the topography of the ground surface and generate a 3D map of the terrain. The position of the vehicle relative to this map is determined by means of a simultaneous localization and mapping algorithm (known in English as SLAM for Si-multaneous Localization And Mapping). Then the path to be followed by the vehicle is determined according to characteristics of the terrain extracted from the sensor data, such as for example data from an accelerometer or a gyroscope. Statement of the invention
[0004] The invention aims to propose a technique for estimating the traversability of a terrain by a mobile system which is both efficient and simple to implement, whether or not there is a road on the terrain (for example in the case of off-piste navigation or in an unstructured environment).
[0005] To this end, the invention proposes a terrain mapping device intended to be embedded in a mobile system, comprising a traversability estimation unit configured to: - determine a depth map of the terrain; - determine a height map of the terrain from the depth map and of terrain image data acquired by a camera on board the mobile system; - determine a first mask of traversability of the terrain by the mobile system by means of thresholding the height map; - determining a second terrain traversability mask for the mobile system by means of semantic segmentation of the terrain image data acquired by the camera on board the mobile system; - determine a traversability map of the terrain by the mobile system by merging the first and second traversability masks.
[0006] Some preferred but non-limiting aspects of this device are as follows: - the traversability estimation unit is configured to determine the terrain depth map from the terrain image data acquired by the camera; - to determine the depth map the traversability estimation unit uses a deep learning model; - the traversability estimation unit includes a pre-trained neural network to implement the deep learning model; - the neural network is also pre-trained to perform semantic segmentation of terrain image data; - the neural network comprises an encoder, a first decoder dedicated to depth inference and a second decoder dedicated to semantic segmentation; - the thresholding of the height map is carried out in such a way that the first traversability mask preserves, when applied to the terrain image data, points of the terrain having a height between a first threshold and a second threshold greater than the first threshold; - the traversability estimation unit is further configured to determine a third traversability mask which retains, when applied to the terrain image data, terrain points having a height less than the first threshold and wherein determining the traversability map further comprises adding the third traversability mask to the merging of the first and second traversability masks. Brief description of the drawings
[0007] Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings in which:
[0008] - [Fig.l] is a diagram illustrating different steps implemented during a method according to the invention for estimating the traversability of a terrain by a mobile system;
[0009] - [Fig.2] is an image acquired by a camera;
[0010] - [Fig.3] is a depth map of the image of [Fig.2];
[0011] - [Fig.4] is a height map determined from the depth map of the [Fig.3] and the image of [Fig.2];
[0012] - [Fig.5] illustrates a first traversability mask;
[0013] - [Fig.6] illustrates a semantic segmentation of the image of [Fig.2];
[0014] - [Fig.7] is another representation of the first traversability mask;
[0015] - [Fig.8] illustrates the result of merging the first and second tra masks versatility.
[0016] DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS
[0017] The invention relates to a terrain mapping device intended to be embedded on a mobile system, for example an all-terrain land mobile system such as a robot, a drone or an autonomous vehicle. As will be detailed later, this device comprises a traversability estimation unit configured to carry out the fusion of a geometric navigation solution taking into account the height of the obstacles encountered on the terrain and a semantic navigation solution identifying the areas traversable by the mobile system in terrain image data acquired by a camera embedded on the mobile system, typically a monocular camera.The traversability estimation unit delivers a traversability map, for example a binary map in which each point of the terrain imaged by the camera is identified as being traversable or not by the mobile system or a map in which a probability of traversability is associated with each point of the terrain.
[0018] The images acquired by the camera are typically RGB images of the terrain. The device is then functional in visible light. In an alternative embodiment, the functionality of the device is extended to nighttime operation by exploiting another wavelength range (typically infrared).
[0019] With reference to [Fig.l], the traversability estimation unit is configured to determine a depth map of the terrain during a step EL
[0020] In a preferred embodiment, the determination of the depth map of the terrain during this step E1 is carried out directly from the image data. of the terrain It acquired by the camera. More specifically, this determination can be obtained by means of a supervised learning model, such as a model based on a deep neural network.
[0021] Figures 2 to 8 are given as illustrative examples of the invention. [Fig.2] thus represents an image from the public NYU Depth v2 dataset while [Fig.3] represents the corresponding depth map.
[0022] In an alternative embodiment, the depth map can be determined by the traversability estimation unit by means of a direct measurement by a dedicated emissive sensor (such as a LiDAR or a structured light 3D scanner). In another alternative, this determination is still carried out passively but using not one but several cameras, for example cameras of a stereovision system. It will be noted that the use of a single monocular camera has the advantages of low consumption for an embedded system, reduced cost compared to a stereovision system or a LiDAR type system and of allowing the development of a non-emissive solution.
[0023] The depth map thus determined makes it possible to supplement the information on the terrain obtained by the camera to thus provide knowledge on the 3D structure of the terrain. Assuming that the mobile system moves on the ground, the traversable areas of the terrain then correspond to the ground plane unlike the obstacles encountered.
[0024] One of the common principles in mobile robotics to address the navigability problem of a system is to define an occupancy grid of the robot's environment. The basic idea of the occupancy grid is to represent a map of the environment as a uniformly spaced field of random variables each representing the probability of the presence of an obstacle at that location in the environment. Based on this principle, the invention considers the image acquired by the camera as a map of the environment where each pixel corresponds to a point on the grid. By knowing the 3D structure of the terrain, the invention associates with each point of the grid the height of the element present at that point to decide whether that element is an obstacle or not.
[0025] To do this, the traversability estimation unit is configured to determine during a step E3 a height map of the terrain from the depth map and the image data of the terrain. With the image data of the terrain, the depth map makes it possible to calculate the 3D coordinates of each point of the terrain, coordinates which can be reprojected along the axis corresponding to the height of these points in order to calculate the height map. The determination of the height map may in particular comprise an inverse projective transformation based on the projective model of the camera, for example in the form of a linear matrix operation. [Fig.4] illustrates the height map determined from the depth map in [Fig.3] and the image in [Fig.2].
[0026] This determination of the height map can be carried out as follows. We consider [ the coordinates of a pixel in the original images presenting geometric distortions. In order to determine the elevation (height) of this pixel, it is necessary to know the intrinsic parameters of the camera, namely the projection matrix P (focal length and optical center) of the camera as well as the distortion parameters K — [ / q, k-,, ... ]• These parameters are conventionally determined by calibrating the camera. A first step aims to correct the distortion of the pixel by applying the distortion model determined during the calibration. For example, in the case of a radial distortion model, the distortion correction is carried out by the following equation:
[0027] [WK1 . / x2" [ vJ = (1 + ^ûr2 + kLH + , r= xc) + ( vd - ye)
[0028] with yj representing the center of the distortion.
[0029] A second step is to determine the 3D position of each pixel in 3D space, by applying the inverse projection and using the estimated depth, as follows:
[0030] (u u -c x ).d (V,rCy)z / y= ~T~ z = d
[0031] With:
[0032] [xy, z] the position of the 3D point in the camera frame,
[0033] fyf Cy] the camera parameters (focal length and optical center), and
[0034] d the estimated depth.
[0035] The point [xy, z] being determined in the camera's frame of reference, it is possible to determine its height relative to the ground, because the position of the camera in the vehicle is known a priori (distance between the camera and the vehicle's wheels).
[0036] In a possible embodiment in which the mobile system has an inertial unit, the traversability estimation unit can, in a step E2 prior to step E3, adapt the projective model to the pose of the camera as estimated by the inertial unit.
[0037] As previously indicated, the height that we wish to estimate is relative to the ground and not in the camera's frame of reference. Thus, it is necessary to take into account the position of the camera. To do this, it is sufficient to take a flat ground as a reference and to estimate the camera position at each image capture relative to an initial position using inertial data. This amounts to estimating the relative poses of the camera relative to an initial position using inertial data.
[0038] In particular, this adaptation makes it possible to standardize the view of the terrain in order to eliminate biases linked to the orientation of the vehicle relative to the ground. A change of reference, from the reference of the camera to the reference of the imaged terrain, can in particular be carried out by means of the different angles provided by the inertial unit.
[0039] In a step E4, the traversability estimation unit is configured to determine a first traversability mask of the terrain by the mobile system by means of a thresholding of the height map. This thresholding makes it possible to determine for each pixel of the height map whether the point of the terrain associated with this pixel is traversable or not by the mobile system, that is to say whether it is part of the ground or whether, on the contrary, its height indicates that it is an obstacle. This thresholding is thus carried out as a function of the characteristics of the mobile system according to the dimensions of the obstacles to be avoided by it.
[0040] In one possible embodiment, the first traversability mask preserves, when applied to the terrain image data, terrain points having a height less than a threshold height. [Fig. 5] illustrates in this regard the thresholding of the height map of [Fig. 4] to determine the first traversability mask, which preserves, when applied to the terrain image data, the terrain points associated with the white areas in [Fig. 5].
[0041] In another possible embodiment, the first traversability mask is determined so that it preserves, when applied to the terrain image data, terrain points having a height between a first threshold and a second threshold greater than the first threshold. In this other embodiment, another traversability mask, referred to as a third traversability mask in the following, can be determined which preserves, when applied to the terrain image data, terrain points having a height less than the first threshold.
[0042] The traversability estimation unit is further configured to, during a step E5, determine a second mask of traversability of the terrain by the mobile system by means of a semantic segmentation of the terrain image data acquired by the camera on board the mobile system. [Fig. 6] illustrates in this regard the result of a semantic segmentation of the image of [Fig. 2] in which the areas of the terrain identified by the semantic segmentation to the category “traversable by the mobile system” bear the reference Tsm. This semantic segmentation can be obtained by means of a supervised learning model, such as a model based on a deep neural network. In one possible embodiment, this segmentation is binary, the pixels of the image being segmented into two classes distinct: traversable by the mobile system and non-traversable by the mobile system.
[0043] The geometric information represented by the first traversability mask and the semantic information represented by the second mask prove to be complementary.
[0044] Indeed, on the one hand, objects of small size or of which a part has a height included in the height range authorized for the determination of the first traversability mask are not taken into account as obstacles when only the geometric information is used. On the other hand, by coupling the geometric information with the semantic information indicating that these objects or parts of objects are not traversable, it proves possible to correctly identify them as obstacles.
[0045] Furthermore, on the other hand, semantic segmentation alone does not allow the traversability estimation problem to be solved since it is solely based on the appearance of the image. However, the traversable zones represent a potentially complex class, in particular due to varied textures (earth, grass or even bitumen are three types of potentially traversable ground but give very different renderings and textures). Furthermore, decorrelated from geometric notions, semantic segmentation can provide aberrant results such as "traversable" pixels in improbable places whose homogeneous texture is locally similar to that of the ground (for example the section of wall between the bed and the desk identified in the zone bearing the reference ZA in [Fig.6]).On the other hand, by coupling the semantic information with the geometric information indicating that these places are not traversable, it turns out to be possible to correctly identify them as obstacles.
[0046] According to the invention, the traversability estimation unit is advantageously configured to, during a step E6, determine a traversability map CT by merging the first and second traversability masks. In such a way, only the pixels of the environment grid which are conserved by both the first and second traversability masks are ultimately considered to be traversable by the mobile system. This merging typically comprises the application of a logical AND between the first and second traversability masks and thus makes it possible to conserve only the geometrically viable pixels whose semantics are proven.
[0047] This solves both the problems of small obstacles present in the range of values used for the determination of the first traversability mask, while removing the aberrant pixels whose predicted semantic class is "traversable" at improbable locations in the image. [Fig.7] illustrates in this regard the first traversability mask M1 while [Fig.8] illustrates the fusion of the first mask of [Fig.7] with the second mask to provide the CT traversability map. We can see that the traversability estimate is refined. In particular, the legs of the chair are cropped and are not considered as traversable pixels whereas they were in [Fig.7]. Furthermore, no pixel whose height is too high in the image is proposed in the final estimate (cf. the section of wall in the ZA zone of [Fig.6]).
[0048] In the embodiment previously mentioned where the first traversability mask is determined so that it preserves, when applied to the terrain image data, the terrain points having a height between the first and second thresholds and where a third traversability mask is determined so that it preserves, when applied to the terrain image data, the terrain points having a height lower than the first threshold, only the first mask is merged with the second mask and the third mask is then added to the result of this merging. Such an embodiment makes it possible to consider that below the first height threshold, all the pixels (of the third mask) are considered traversable while on the rest of the image the pixels considered to be traversable are those resulting from the merging of the first mask (geometric approach) and the second mask (semantic approach).
[0049] It has been seen in the above that the traversability estimation unit can exploit a deep learning model to determine the depth map.
[0050] This deep learning model can exploit a neural network that has an encoder-decoder architecture. The encoder is responsible for extracting features that are subsequently provided to the decoder to generate the depth maps. The feature extractor includes successive convolution layers followed by nonlinearities such as data normalization functions, dimension reduction functions or nonlinear reprojection functions such as, among others, the sigmoid or the rectified linear unit (convolutional network). The reduced-dimensional features extracted from the image are then delivered to the decoder (also typically a convolutional network) which is responsible for recovering the spatial dimension of the image while calculating the features necessary for decoding the features, thus producing a depth map.
[0051] Training of the deep learning model can be carried out as follows: - Acquisition of a training database that includes scene images and their corresponding depth maps. These ground truth depth maps are obtained by a third-party method, for example a depth map acquired by LIDAR, by means of structured light, by stereovision, etc. - Calculation of a representative value of the performance of the learning model by comparison with the depth map of the provided scene output of the neural network and the ground truth depth map. The weights of the neural network connections are then adjusted to reduce the depth map prediction error. For example, the gradient of the error can be calculated to determine a direction of variation and a shift in a direction opposite to the gradient is then performed.
[0052] Thus, in one possible embodiment, the traversability estimation unit may comprise a pre-trained neural network to implement the deep learning model. In one possible variant, this neural network may also be pre-trained to perform the semantic segmentation of the terrain image data. In particular, the neural network may comprise an encoder and, at the output of the encoder, a first decoder dedicated to determining the depth map and a second decoder dedicated to the semantic segmentation.
[0053] The invention is not limited to the device previously described but also extends to a method during which the steps previously described are implemented by computer as well as to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of such a method.
Claims
Claims
1. Terrain mapping device intended to be embedded on a mobile system, comprising a traversability estimation unit configured to: - determine (E1) a depth map of the terrain; - determine (E3) a height map of the terrain from the depth map and image data of the terrain acquired by a camera embedded on the mobile system; - determine (E4) a first traversability mask of the terrain by the mobile system by means of a thresholding of the height map, the thresholding of the height map being carried out in such a way that the first traversability mask preserves, when it is applied to the image data of the terrain, points of the terrain having a height comprised between a first threshold and a second threshold greater than the first threshold;- determining (E5) a second traversability mask of the terrain by the mobile system by means of a semantic segmentation of the terrain image data acquired by the camera on board the mobile system; - determining a third traversability mask which preserves, when applied to the terrain image data, points of the terrain having a height lower than the first threshold; - determining (E6) a traversability map of the terrain by the mobile system (CT) by merging the first and second traversability masks and adding the third traversability mask to the merging of the first and second traversability masks.;
2. The device of claim 1, wherein the traversability estimation unit is configured to determine the depth map of the terrain from the image data of the terrain acquired by the camera.
3. Device according to claim 2, in which to determine the depth map the traversability estimation unit exploits a deep learning model.
4. A device according to claim 3, wherein the estimation unit traversability includes a pre-trained neural network to implement the deep learning model.
5. The device of claim 4, wherein the neural network is also pre-trained to perform semantic segmentation of the terrain image data.
6. Device according to claim 5, wherein the neural network comprises an encoder, a first decoder dedicated to depth inference and a second decoder dedicated to semantic segmentation.
7. A computer-implemented method for estimating the traversability of terrain by a mobile system, comprising the steps of: - determining a depth map of the terrain; - determining a height map of the terrain from the depth map and image data of the terrain acquired by a camera on board the mobile system;- determining a first mask of traversability of the terrain by the mobile system by means of a thresholding of the height map, the thresholding of the height map being carried out in such a way that the first traversability mask preserves, when it is applied to the terrain image data, points of the terrain having a height between a first threshold and a second threshold higher than the first threshold; - determining a second mask of traversability of the terrain by the mobile system by means of a semantic segmentation of terrain image data acquired by the camera on board the mobile system; - determining a third traversability mask which preserves, when it is applied to the terrain image data, points of the terrain having a height lower than the first threshold;- determining a traversability map of the terrain by the mobile system by merging the first and second traversability masks and adding the third traversability mask to the merging of the first and second traversability masks.;
8. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to claim 7.
