System and method for assisting navigation of a mobile system
The method enhances mobile system navigation by using convolutional neural networks to process optical and depth data, generating traversable paths with improved reliability and accuracy through semantic and confidence-based terrain mapping.
Patent Information
- Application Number
- FR2022011564
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-07
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2042-11-07
Smart Images
Figure 00000014_0000 
Figure 00000015_0000 
Figure 00000016_0000
Abstract
Description
Title of the invention: System and method for assisting navigation of a mobile system Technical field
[0001] The field of the invention is that of assisting the navigation of a mobile system of the robot or autonomous vehicle type moving on a terrain, and more particularly that of generating a trajectory navigable by the mobile system on the terrain. Prior art
[0002] In the field of mobile system navigation, methods are known which aim to find the presence of a road in images acquired by a camera on board a mobile system. These methods use visual clues such as vanishing points, textures or even relief to delineate the contours of a road in an image, or pose the problem directly as a problem of segmentation of the road in the image. However, these methods are not interested in paths in the broadest sense of the term, which may in particular be off-road paths not necessarily paved or correctly delineated, and even less in the more general theme of traversability corresponding to the identification in the acquired images of areas of the terrain on which the mobile system would be able to move.
[0003] On the other hand, there are methods based on neural network models that identify the type of ground on which the vehicle is moving. For example, document WO 2019 / 241022 A1 describes a solution using a pre-trained deep neural network to detect a waterway that is not necessarily delimited by road markings. Statement of the invention
[0004] The invention aims to propose a solution for generating a traversable trajectory for a mobile system moving on terrain which is both reliable and efficient.
[0005] To this end, the invention proposes a computer-implemented method for assisting navigation of a mobile system, comprising: - obtaining an optical image of a scene acquired by a camera on board the mobile system; - obtaining a 3D point cloud of the scene acquired by a rangefinder on board the mobile system; - the projection, in 2D in the camera frame, of the 3D points of the cloud and of an uncertainty relating to each of the 3D points of the cloud to provide res respectively a depth image and an uncertainty mask of the depth image; - determining a semantic map of the scene, a depth map of the scene and a confidence map of the depth map from the optical image, the depth image and the uncertainty mask of the depth image; - the determination of a traversability map of the scene by the mobile system by merging the semantic map, the depth map and the confidence map of the depth map.
[0006] Some preferred but non-limiting aspects of this method are as follows: - determining the semantic map of the scene, the depth map of the scene and the confidence map of the depth map comprises processing the optical image, the depth image and the uncertainty mask of the depth image by a first convolutional neural network which comprises a succession of convolutional layers, each convolutional layer comprising a first convolution block capable of estimating a semantic attribute map, a second convolution block capable of estimating a depth attribute map and a third convolution block capable of estimating a confidence attribute map; - the second convolution block of a convolutional layer of rank N+l in the succession of convolutional layers is configured to: • calculating the product of the confidence attribute map estimated by the third convolution block of the convolutional layer of rank N in the succession of convolutional layers with the depth attribute map estimated by the second convolution block of the convolutional layer of rank N in the succession of convolutional layers; • calculate a first convolution result by applying a convolution kernel to said product; • calculate a second convolution result by applying the convolution kernel to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers; • calculate the ratio of the first and second correlation results; - the second convolution block of a convolutional layer of rank N+l in the succession of convolutional layers is configured to: • calculate the product of the confidence attribute map estimated by the third convolution block of the convolutional layer of rank N
[0007] in the succession of convolutional layers with a concatenation map resulting from the concatenation of the semantic attribute map estimated by the first convolution block of the convolutional layer of rank N in the succession of convolutional layers and the depth attribute map estimated by the second convolution block of the convolutional layer of rank N in the succession of convolutional layers; • calculate a first convolution result by applying a convolution kernel to said product; • calculate a second convolution result by applying the convolution kernel to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers; • calculate the ratio of the first and second correlation results; - the second convolution block of the convolutional layer of rank N+l in the succession of convolutional layers is further configured to add a bias to the ratio of the first and second correlation results; - the first convolution block of a convolutional layer of rank N+1 in the succession of convolutional layers takes as input a concatenation map resulting from the concatenation of the semantic attribute map estimated by the first convolution block of the convolutional layer of rank N in the succession of convolutional layers with the depth attribute map estimated by the second convolution block of the convolutional layer of rank N in the succession of convolutional layers; - the fusion of the semantic map, the depth map and the confidence map of the depth map includes determining a concatenation map by concatenating the semantic map, the depth map and the confidence map of the depth map and processing the concatenation map by a second convolutional neural network. The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to the invention. The invention also extends to a terrain mapping device intended to be embedded on a mobile system, comprising a processor configured to implement the steps of the method according to the invention. Brief description of the drawings
[0008] Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings in which:
[0009] - [Fig.l] is a diagram illustrating a possible embodiment of a method according to the invention;
[0010] - [Fig.2] represents the operations carried out by a convolutional layer of a first convolutional neural network that can be used by the invention;
[0011] - [Fig.3] represents more particularly the operations carried out by the second and the third convolution block of a convolutional layer of a first convolutional neural network usable by the invention.
[0012] DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS
[0013] The invention relates in particular to a terrain mapping device intended to be embedded in a mobile system, for example an all-terrain land mobile system such as a robot, a drone or an autonomous vehicle.
[0014] This device comprises a traversability estimation unit configured to generate a trajectory traversable by the mobile system from a stream of images coming from a camera as well as depth measurements from a rangefinder.
[0015] Since the generation of a reliable trajectory requires a perception of the geometry and semantics of the terrain, the traversability estimation unit advantageously performs the fusion of a geometric solution for estimating the 3D of the terrain (its depth in this case) with a semantic segmentation solution of the terrain. The traversability estimation unit also uses a confidence map associated with the reliability of the prediction of the geometric solution, which makes it possible to greatly improve performance.
[0016] The traversability estimation unit delivers a traversability map, for example a binary map in which each point of the terrain imaged by the camera is identified as being traversable or not by the mobile system or a map in which a probability of traversability is associated with each point of the terrain.
[0017] The traversability estimation unit is configured to implement the method which will be described below with reference to [Fig.l].
[0018] This method comprises obtaining RGB of an optical image of a scene (in this case a terrain on which the mobile system is moving), acquired by a camera on board the mobile system. The camera is for example a monocular camera. The images successively acquired by the camera are typically RGB images of the terrain, ensuring functionality in visible light. In an alternative embodiment, night operation is ensured by using another wavelength range (e.g. infrared).
[0019] The method further comprises a LiDAR step of obtaining a 3D point cloud of the scene acquired by a rangefinder on board the mobile system. The rangefinder is for example a laser rangefinder, such as a LiDAR. The method then comprises a step of projecting, in 2D in the frame of reference of the camera, the 3D points of the cloud and an uncertainty relating to each of the 3D points of the cloud to respectively provide a depth image and an uncertainty mask of the depth image (i.e., a map in which an uncertainty relating to the determination of the depth is associated with each point of the terrain).
[0020] The rangefinder provides sparse depth measurements which are generally artificially densified by encoding the unobserved pixels. Furthermore, by using the power (amplitude) of the signal received by the rangefinder, which corresponds for example to the quantity of light which returns to the sensor after a shot, it is possible to deduce an uncertainty on the depth measurements. Indeed, the quantity of light received in return by the sensor is directly correlated to the material on which it is projected and gives information on the reliability of the distance calculated at this point.
[0021] The method then comprises a step of determining a semantic map MS of the scene, a depth map MD of the scene and a confidence map MT of the depth map from the optical image, the depth image and the uncertainty mask of the depth image. This step is for example implemented by a first convolutional neural network CNN 1 suitably pre-trained for this purpose.
[0022] This step performs the simultaneous inference of the 3D (the depth map) and the semantics of the image (the semantic map). This results in a better prediction of these two modalities, with minimized calculation time. Furthermore, this step exploits an uncertainty determined a priori from the telemetry data to estimate a reliability (the confidence map) on the predictions.
[0023] The method continues with a step consisting of determining a CT traversability map of the scene by the mobile system by merging the semantic map, the depth map and the confidence map of the depth map. This step is for example implemented by a second CNN2 convolutional neural network suitably pre-trained for this purpose. This step takes advantage of the two modalities (3D and semantic) and merges them using confidence as weighting.
[0024] With reference to [Fig.2], the first convolutional neural network CNN1 comprises a succession of convolutional layers CN, CN+i and each convolutional layer may comprise a first convolution block B1N, B1n+i capable of estimating a semantic attribute map FMSN, FMSn+i, a second convolution block B2N, B2N+i capable of estimating a depth attribute map FMDN, FMDN+i and a third block of convolution B3N, B3N+i capable of estimating a confidence attribute map FMTN, fmtn+1.
[0025] In a possible embodiment, the first convolution block B1n+i of the convolutional layer of rank N+l in the succession of convolutional layers takes as input the FMSN semantic attribute map estimated by the first convolution block B1N of the convolutional layer of rank N in the succession of convolutional layers. This is true for N integer greater than or equal to 1, while the first convolution block of the first convolutional layer in the succession of convolutional layers takes as input the optical image.
[0026] In an alternative embodiment shown in [Fig. 2], the first convolution block B1n+i of the convolutional layer of rank N+l in the succession of convolutional layers takes as input a concatenation map resulting from the concatenation, identified by the reference and in FIG. 2, of the semantic attribute map FMSN estimated by the first convolution block B1N of the convolutional layer of rank N in the succession of convolutional layers and the depth attribute map FMDn estimated by the second convolution block B2N of the convolutional layer of rank N. This is true for N integer greater than or equal to 1, while the first convolution block of the first convolutional layer in the succession of convolutional layers takes as input the concatenation of the optical image and the depth image.
[0027] In this alternative embodiment, the first convolutional neural network thus comprises a first branch (the succession of the first convolution blocks) which works on the estimation of the semantics of the scene by taking advantage of the optical information from the camera but also the depth information from the rangefinder. The semantic segmentation is thus improved.
[0028] In a possible embodiment, the second convolution block B2N+i of the convolutional layer of rank N+l in the succession of convolutional layers takes as input the depth attribute map FMDN estimated by the second convolution block B2N of the convolutional layer of rank N in the succession of convolutional layers and the confidence attribute map FMSN estimated by the third convolution block B3N of the convolutional layer of rank N in the succession of convolutional layers. This is true for N integer greater than or equal to 1, while the second convolution block of the first convolutional layer in the succession of convolutional layers takes as input the depth image and the uncertainty mask of the depth map.
[0029] In an alternative embodiment shown in [Fig.2], the second convolution block B2N+i of the convolutional layer of rank N+l in the succession of convolutional layers takes as input, on the one hand the concatenation map resulting from the concatenation, identified by the reference and, of the semantic attribute map FMSN estimated by the first convolution block B1N of the convolutional layer of rank N in the succession of convolutional layers and the depth attribute map FMDn estimated by the second convolution block B2N of the convolutional layer of rank N and, on the other hand, the confidence attribute map FMSN estimated by the third convolution block B3N of the convolutional layer of rank N in the succession of convolutional layers. This is true for N integer greater than or equal to 1, while the second convolution block of the first convolutional layer in the succession of convolutional layers takes as input, on the one hand, the concatenation of the optical image and the depth image and, on the other hand, the uncertainty mask of the depth map.
[0030] In this alternative embodiment, the first convolutional neural network thus comprises a second branch (the succession of second convolution blocks) which works on estimating the depth of the scene by taking advantage of the depth information from the rangefinder but also the optical information from the camera. The semantic depth estimation is thus improved.
[0031] Furthermore, in the two previously mentioned embodiments, an a priori uncertainty on the measurements of the rangefinder is propagated throughout the succession of convolutional layers, which makes it possible to obtain confidence in the quality and reliability of the output predictions.
[0032] Figure 3 represents a possible realization of operations implemented by the second and third convolution blocks of a convolutional layer of the first convolutional neural network. In this figure 3, • corresponds to a point-by-point multiplication, * to a convolution, / to a division and + to an addition. F(W) represents the kernel of the convolution.
[0033] Consider X as a tensor representing an input signal, C as a positive scalar function representing the confidence (or certainty) for each value of X, B as a tensor representing the basis of a filtering operator and B* as its conjugate, and A as a positive scalar function representing the applicability for each value of B. The normalized convolution can be written as follows: Cl)
[0034] where I) = AB * C'A (2? A = (ABBB^C (3)
[0035] In equation (1), N is the normalization factor. For example, considering the case where confidence C is constant and B=1, equation (1) becomes:
[0036] where the convolution parameters A' are the normalized version of A.
[0037] In the context of the invention, the training of the first neural network is carried out so as to determine the parameters corresponding to the product AB for a task of generating the depth map from sparse input data associated with an a priori confidence. More particularly, the basis B is fixed to be equal to a tensor of 1 and the applicability function A is learned during the network training phase.
[0038] Referring to Figure 3, the applicability function A corresponds to the convolution parameters. Because the applicability must remain a positive function, the positivity of the convolution weights must be guaranteed. Thus, a softplus function F (. ) can be applied to the convolution weights W. Based on equation (1), the depth propagation becomes:
[0039] Thus, the second convolution block B2N+i of a convolutional layer of rank N+l in the succession of convolutional layers can be configured to: - calculating the product (by means of point-by-point multiplication •) of the FMTN confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers with a concatenation map resulting from the concatenation of the FMSN semantic attribute map estimated by the first convolution block of the convolution layer of rank N in the succession of convolutional layers and the FMDN depth attribute map estimated by the second convolution block of the convolution layer of rank N in the succession of convolutional layers; - calculate a first convolution result by applying the convolution kernel F ( W ) to said product; - calculating a second convolution result by applying the convolution kernel F(W) to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers; - calculate the ratio, by means of division / , of the first and second correlation results.
[0040] As seen previously, in another possible embodiment, the second Convolution blocks take as input only the FMDNet depth attributes not the result of their concatenation with the FMSN semantic attributes. This other possible realization is illustrated in [Fig.3] and according to it the second convolution block B2N+i of a convolutional layer of rank N+l in the succession of convolutional layers is configured to: - calculating the product (by means of point-by-point multiplication •) of the FMTN confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers with the FMDN depth attribute map estimated by the second convolution block of the convolution layer of rank N in the succession of convolutional layers; - calculate a first convolution result by applying the convolution kernel r(W) to said product; - calculating a second convolution result by applying the convolution kernel F(W) to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers;
[0041] calculate the ratio, by means of division / , of the first and second correlation results.
[0042] Furthermore, in either of the embodiments mentioned above, and as also shown in [Fig. 3], each second convolution block may further be configured to add a bias term BS to the result of the ratio of the first and second correlation results. This bias term makes it possible to increase the capacity of the first neural network.
[0043] [Fig.3] further illustrates a third B3N+i convolution block. This block performs a conventional convolution for confidence propagation. This block may include a ReLU (Rectifier Linear Unit) activation function to ensure positivity and maintain dimensionality between the confidence attribute maps and the depth attribute maps.
[0044] Similarly, the first convolution blocks that determine the semantic attribute maps can take the form of conventional convolution blocks.
[0045] A possible realization of the training of the first convolutional neural network exploits the following cost function to learn to regress depth and to model the inverse of uncertainty (i.e., confidence). Let S be a set of coordinates where the depth value is given in the ground truth, log(C(HV)) the predicted l°g_ confidence, the depth ground truth and the predicted depth. The cost function can be defined as follows:
[0046] In equation (8), 2 is a hyperparameter, Lp is the regression error defined by equation (6) and Pen is a penalty term defined by equation (7) which prevents the case where the output confidences are equal to 0. In this equation (8), the left term is the product of the regression error by the confidence. The P ' norm is to be replaced by the desired regression error.
[0047] Through this multiplication, confidence acts as a weighting on the regression error and therefore impacts the learning speed, both globally and relatively. First globally, because when 2 decreases, the value of the average confidence also decreases, so the learning speed decreases globally. And relatively because the greater the entropy of the confidence distribution, the more the impact on the learning speed will be varied depending on the spatial locations. The choice of 2 therefore controls the average confidence and the entropy of the distribution, thus impacting learning.
[0048] In practice, a log confidence prediction can be performed to improve the stability of the learning. Also, in order to keep the confidence outputs in the interval [0, 1] to facilitate the interpretation of the results, a (-1) x ReLU activation can be performed on the last layer to obtain a negative log confidence, which makes it possible to produce a final confidence output in the interval [0, 1].
[0049] The first convolutional neural network outputs a semantic map MS, a depth map MD and a confidence map MT of the depth map. The determination of the traversability map CT of the scene by the mobile system may comprise determining a concatenation map by concatenating the semantic map, the depth map and the confidence map of the depth map and processing the concatenation map by the second convolutional neural network CNN2. This second network may be a convolutional network of conventional architecture.
Claims
Claims
1. A computer-implemented method of assisting in the navigation of a mobile system, comprising: - obtaining (RGB) an optical image of a scene acquired by a camera on board the mobile system; - obtaining (LiDAR) a 3D point cloud of the scene acquired by a rangefinder on board the mobile system; - the projection, in 2D in the camera frame, of the 3D points of the cloud and of an uncertainty relating to the measurement of each of the 3D points of the cloud to respectively provide a depth image and an uncertainty mask of the depth image; - determining a semantic map of the scene (MS), a depth map of the scene (MD) and a confidence map (MT) of the depth map from the optical image, the depth image and the uncertainty mask of the depth image, this determination comprising the processing of the optical image, the depth image and the uncertainty mask of the depth image by a first convolutional neural network (CNN1) which comprises a succession of convolutional layers, each convolutional layer comprising a first convolution block (B 1N, BlN+i) capable of estimating a semantic attribute map (FMSN, FMSn+i), a second convolution block (B2N, B2N+i) capable of estimating a depth attribute map (FMDN, FMDn+i) and a third convolution block (B3N, B3N+i) capable of estimating a confidence attribute map (FMTN, FMTn+i); - the determination of a traversability map (CT) of the scene by the mobile system by merging the semantic map (MS), the depth map (MD) and the confidence map (MT) of the depth map.
2. The method of claim 1, wherein the second convolution block (B2N+i) of a convolutional layer of rank N+l (CN+i) in the succession of convolutional layers is configured to: - calculate the product of the trust attribute map (FMTN) estimated by the third convolution block (B3N) of the convolutional layer of rank N (CN) in the succession of convolutional layers with the depth attribute map (FMDn) estimated by the second convolution block (B2N) of the convolutional layer of rank N (CN) in the succession of convolutional layers; - calculate a first convolution result by applying a convolution kernel to said product; - calculating a second convolution result by applying the convolution kernel to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers; - calculate the ratio of the first and second correlation results.
3. Method according to claim 1, in which the second convolution block (B2N+i) of a convolutional layer of rank N+l (CN+i) in the succession of convolutional layers is configured to: - calculating the product of the confidence attribute map (FMTN) estimated by the third convolution block (B3N) of the convolutional layer of rank N (CN) in the succession of convolutional layers with a concatenation map resulting from the concatenation of the semantic attribute map (FMSN) estimated by the first convolution block (B1N) of the convolutional layer of rank N (CN) in the succession of convolutional layers and the depth attribute map (FMDN) estimated by the second convolution block (B2N) of the convolutional layer of rank N (CN) in the succession of convolutional layers; - calculate a first convolution result by applying a convolution kernel to said product; - calculating a second convolution result by applying the convolution kernel to the confidence attribute map estimated by the third convolution block of the convolution layer of rank N in the succession of convolutional layers; - calculate the ratio of the first and second correlation results.
4. Method according to one of claims 2 and 3, in which the second convolution block (B2N+i) of the convolutional layer of rank N+l (CN+i) in the succession of convolutional layers is further configured to add a bias (BS) to the ratio of the first and second correlation results.
5. Method according to claim 1, in which the first convolution block (BlN+[) of a convolutional layer of rank N+l (CN+i) in the succession of convolutional layers takes as input a concatenation map resulting from the concatenation of the semantic attribute map (FMSn) estimated by the first convolution block (B1N) of the convolutional layer of rank N in the succession of convolutional layers with the depth attribute map (FMDN) estimated by the second convolution block (B2N) of the convolutional layer of rank N in the succession of convolutional layers.
6. The method of one of claims 1 to 5, wherein merging the semantic map, the depth map, and the confidence map of the depth map comprises determining a concatenation map by concatenating the semantic map, the depth map, and the confidence map of the depth map and processing the concatenation map by a second convolutional neural network (CNN2).
7. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to one of claims 1
8. a o. Terrain mapping device intended to be embedded on a mobile system, comprising a processor configured to implement the steps of the method according to one of claims 1 to 6.