System and method for aiding the navigation of a mobile system by means of a terrain traversability prediction model for the mobile system
Patent Information
- Application Number
- FR2022008426
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-08-22
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2042-08-22
Smart Images
Figure 00000013_0000 
Figure 00000014_0000 
Figure 00000015_0000
Abstract
Description
Title of the invention: System and method for assisting the navigation of a mobile system by means of a model for predicting the traversability of a terrain by the mobile system Technical field
[0001] The field of the invention is that of navigation assistance for a mobile system of the robot or autonomous vehicle type. The invention relates more particularly to the machine learning of a model and the use of the model having been the subject of machine learning to predict, from images of a terrain, the traversability of the terrain by the mobile system. Prior art
[0002] In the field of mobile system navigation, methods are known which aim to find the presence of a road in images acquired by a camera on board a mobile system. These methods use visual clues such as vanishing points, textures or even relief to delineate the contours of a road in an image, or pose the problem directly as a problem of segmentation of the road in the image. However, these methods are not interested in paths in the broadest sense of the term, which may in particular be off-road paths not necessarily paved or correctly delineated, and even less in the more general theme of traversability corresponding to the identification in the acquired images of areas of the terrain on which the mobile system would be able to move.
[0003] On the other hand, there are methods based on neural network models that identify the type of ground on which the vehicle is moving. For example, document WO 2019 / 241022 A1 describes a solution using a suitably pre-trained deep neural network to perform the detection of a waterway that is not necessarily delimited by road markings.
[0004] The training of such neural network models is based on the acquisition of images during driving of the mobile system in real situations, followed by a step of manual annotation of the images by human operators. This phase of manual annotation of the training data is long and costly. Statement of the invention
[0005] The invention aims to propose a navigation assistance technique in which the annotation of data dedicated to machine learning of a model for predicting the traversability of a terrain by a mobile system could be automated.
[0006] To this end, the invention proposes a computer-implemented method for assisting with the navigation of a mobile system, comprising: - the generation of training data for machine learning of a model for predicting the traversability of a terrain by the mobile system, said generation comprising: • obtaining a sequence of images of an environment acquired by a camera on board a vehicle moving in the environment; • determining a footprint of the vehicle when acquiring an image from the sequence forming a first reference image; • the annotation, as an area traversable by the mobile system, of an image region corresponding to the projection of said imprint into another image of the sequence forming a first projection image; - machine learning of the model using training data; - obtaining images of a scene acquired by a camera on board the mobile system; and - prediction of scene traversability by providing images of the scene to the machine-learned model.
[0007] The invention thus makes it possible to automate the annotation of images. This results in a less expensive annotation phase that can be carried out more quickly. It also makes it possible to build a learning base comprising a greater number of annotated images. Automatic annotation of images is also less subject to errors than manual annotation. This results in the learning of a better quality prediction model and hence more precise navigation assistance.
[0008] Some preferred but non-limiting aspects of this method are as follows: - the generation of training data further includes: • obtaining inertial data acquired by an inertial unit on board the vehicle moving in the environment; • the determination, by simultaneous localization and mapping using the images of the sequence and the inertial data, of the position of the camera in a reference frame of the environment during the acquisition of the images of the sequence;
[0009] and the determination of the footprint of the vehicle during the acquisition of the first reference image is carried out from the position of the camera in the reference frame of the environment during the acquisition of the first reference image; - the generation of the training data further comprises the calculation of a similarity score between the image region corresponding to the projection of said fingerprint in the first projection image and an image region corresponding to the projection of said fingerprint in another image of the sequence forming a second projection image and, when the similarity score is less than a threshold, the image region corresponding to the projection of said fingerprint in the second projection image is not the subject of an annotation; - when generating the training data, the determination of a vehicle footprint is repeated to determine a vehicle footprint when acquiring another image of the sequence forming a second reference image and further comprising the annotation as a traversable area of an image region corresponding to the projection in the first projection image of the vehicle footprint when acquiring the second reference image; - generating the training data further comprises annotating as a traversable area an intermediate region in the first projection image between the image region corresponding to the projection of the vehicle footprint during the acquisition of the first reference image and the image region corresponding to the projection of the vehicle footprint during the acquisition of the second reference image. Brief description of the drawings
[0010] Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings in which:
[0011] - [Fig.l] is a diagram illustrating the annotation of training data and machine learning of a neural network;
[0012] - [Fig.2] is a diagram illustrating the use of a pre-trained neural network in accordance with the invention;
[0013] - [Fig.3] illustrates the general principle according to which images acquired during the vehicle movement presents traversable image regions corresponding to the areas where the vehicle has moved;
[0014] - [Fig.4] is a diagram illustrating different stages of an annotation process training data in accordance with the invention;
[0015] - [Fig.5] illustrates the projection of vehicle footprints on the ground in different images;
[0016] - [Fig.6] illustrates the annotation of an intermediate zone between several prints on the ground projected into an image.
[0017] DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS
[0018] The invention relates to a computer-implemented method for generating training data for machine learning of a model for predicting the traversability of terrain by a mobile system, for example an all-terrain land mobile system such as a robot, a drone or an autonomous vehicle.
[0019] The invention extends to a machine learning method, comprising the generation of training data and the machine learning of a model for predicting the traversability of a terrain by a mobile system, using the training data. With reference to [Fig.l], this method comprising the ANN annotation of raw data RDS to provide training data ADS and the TRAIN training of an untrained neural network UNN using the annotated training data ADS to provide a trained neural network TNN. The raw data RDS are here terrain images and the training data are annotated terrain images where each pixel is identified as being traversable or not by the mobile system. For example, if the pixel is traversable, it takes the value 1, otherwise it takes the value 0. We thus end up with a binary image which represents the traversability of the terrain.
[0020] The neural network can be a convolutional network (CNN for “Convolutional Neural Network”) capable of making dense prediction, for example a network based on the U-net architecture.
[0021] With reference to [Fig.2], the invention further extends to the use INFER of the trained neural network TNN to determine, from an image of a scene IE, a traversability map of this scene CT, for example a binary map in which each point of the imaged scene is identified as being traversable or not by the mobile system or a map in which a probability of traversability is associated with each point of the scene. The image of the scene IE is acquired by a camera on board the mobile system, preferably a monocular camera.
[0022] It will be noted that the use of a single monocular camera to estimate such a traversability map has the advantages of low consumption for an on-board system, reduced cost compared to a stereovision system or a LiDAR type system and of making it possible to carry out such an estimation without an emissive sensor.
[0023] The images successively acquired by the camera are typically RGB images of the terrain, ensuring functionality in visible light. In an alternative embodiment, nighttime operation is ensured by exploiting another wavelength range (typically infrared).
[0024] The invention uses images forming the raw RDS data which have been acquired by means of a camera on board a vehicle moving in the environment. This movement is for example carried out by a human driver. Different movements are preferably carried out in order to cover various environments.
[0025] Assuming that the vehicle navigates a priori on the ground, the invention assumes that the traversable zones of an environment correspond to the ground planes on which the vehicle has moved. It therefore proves possible to automatically annotate the images forming the RDS raw data by identifying the zones on which the vehicle has been caused to move. This approach makes it possible to go directly from movement, which is a mandatory step for acquiring raw data, to the generation of a database automatically annotated with the traversable zones identified on the images, without going through a phase of manual annotation of the raw data.
[0026] [Fig. 3] illustrates the implementation of this approach. Here, we consider that the vehicle 1 is moving forward (from right to left in [Fig. 3]) and that the on-board camera 2 images the environment in front of the vehicle. [Fig. 3] thus represents the vehicle at two different times during the movement. On the right, the camera images a region of the ground on which the vehicle will then be moved. The invention proposes to determine the footprint of the vehicle EN when it is traveling on this region (i.e. in its left position in [Fig. 3]) and then to annotate, as an area traversable by the mobile system, an image region RI corresponding to the projection of said footprint in an image acquired by the camera (in this example, an image previously acquired when the vehicle is in its right position in [Fig. 3]).
[0027] With reference to [Fig. 4] which illustrates different steps that can be carried out during the ANN annotation of raw RDS data, the method according to the invention comprises an OBT step of obtaining a sequence of images of an environment acquired by a camera on board a vehicle moving in the environment. The method further comprises an EMP step of determining a footprint of the vehicle during the acquisition of an image of the sequence forming a first reference image (i.e. when the vehicle is in the left position in [Fig. 3]). In order to determine this footprint, the ground can be considered as being locally flat under the vehicle. This determination of the footprint of the vehicle during the acquisition of the first reference image is followed by a PROJ step of projecting said footprint into another image of the sequence forming a first projection image (i.e.the image acquired when the vehicle is at the right position in [Fig.3]) and an annotation step, as a traversable area. by the mobile system, of an image region in the first projection image corresponding to the projection of said imprint.
[0028] For the sake of clarity, the above has taken the case of a first reference image and a first projection image. In reality, the footprint of the vehicle on the ground during the acquisition of the first reference image can be projected into a plurality of images of the sequence, typically the images immediately preceding the reference image (for example when the images were acquired with the vehicle moving forward and the camera pointing forward) in the sequence or immediately following the reference image in the sequence (for example when the images were acquired with the vehicle moving forward and the camera pointing forward). Furthermore, the method can be repeated by taking each of the images of the sequence in turn as the first reference image.
[0029] [Fig. 5] illustrates this principle with the footprint of the vehicle EN during the acquisition of an image at time tN projected into the images IN_i, IN 2 and IN. 3 previously acquired at times tN _btN 2 and tN _3, with the footprint of the vehicle En_i during the acquisition of the image IN i at time tN i projected into the images IN. 2 and IN_ 3 previously acquired at times tN 2 and tN _3 and with the footprint of the vehicle EN 2 during the acquisition of the image IN_ 2 at time tN 2 projected into the image IN. 3 previously acquired at time tN _3. In the image IN. 3, the pixels corresponding to the projections in this image of the footprints EN, EN i and EN 3 are thus annotated as traversable pixels.
[0030] In order to enable the determination of the footprint on the ground, in one possible embodiment, the method comprises obtaining inertial data acquired by an inertial unit on board the vehicle moving in the environment, these inertial data being acquired synchronously with the images of the image sequence. Then with reference to [Fig.4], the method comprises a step of SLAM determination, by simultaneous localization and mapping (known in English as "simultaneous localization and mapping") using the images of the sequence and the inertial data, of the position of the camera in a reference frame of the environment during the acquisition of the images of the sequence. This reference frame, also referred to as a world reference frame, forms a global reference frame for any positioning.The SLAM step allows more specifically to reconstruct the trajectory of the vehicle by replacing the different successive poses of the camera in the reference frame of the environment.
[0031] The footprint of the vehicle during the acquisition of the first reference image is then determined by exploiting the position of the camera in the reference frame of the environment during the acquisition of the first reference image as well
[0032]
[0033]
[0034]
[0035] than a priori knowledge of the dimensions of the vehicle (typically its length and width) and the spatial position of the camera in the vehicle. This footprint characterizes an area of the environment taken by the vehicle in the reference frame of the environment at a given time (in this case the time of acquisition of the first reference image). By characterizing the footprint associated with each of the images in the sequence, the entire path taken by the vehicle can be determined. Once the path taken by the vehicle has been determined, it is necessary to determine this path in the images in order to be able to annotate them on the images. As previously indicated, this is done during the PROJ step by projecting the 3D coordinates of the footprint during the acquisition of an image into a plurality of other images in the sequence (for example the N previous images assuming that the vehicle is traveling forward and that the camera is looking in front of the vehicle). This projection makes it possible to move from the coordinates of the vehicle's footprint in the environment frame to the pixels corresponding to this footprint seen in the previous images. This projection is performed using the intrinsic matrix and the distortion parameters of the camera obtained by calibration as well as the different camera poses estimated by simultaneous localization and mapping during the SLAM step. For example, the projection [i, j] of a 3D point in an image is given by the following equation: he' V .w. 'X' y Z .1. Or w ' Lv. [x, y. z] correspond to the coordinates of the 3D point in the coordinate system of the environment ; R is a rotation matrix andf is a translation vector, which allows us to represent the position of the camera in the environment frame; P denotes the camera's projection matrix, determined by prior camera calibration. Note that the camera's distortion parameters are not used, as the images are rectified. The method may further comprise a step of filtering FILT the images in which the image region corresponding to the projection of the vehicle's footprint on the ground during the acquisition of the first reference image is not in reality representative of this footprint on the ground. For example, if a surface on which the vehicle rolled (position associated with the first reference image) is not visible, for example due to occultation or a turn on the path taken, in an image acquired at a previous position of the camera, then this image is discredited.
[0036] To do this, the filtering step FILT may comprise the calculation of a similarity score between the image region corresponding to the projection of the footprint on the ground during the acquisition of the first reference image in the first projection image and the image region corresponding to the projection of said footprint in another image of the sequence forming a second projection image. For example, the first reference image is the image IN, the first projection image is the image IN.i acquired immediately before the image IN and the second projection image is any one of the other images of the plurality of images in which the projection of the footprint on the ground is carried out during the acquisition of the first reference image. When the similarity score is less than a threshold, the image region corresponding to the projection of said footprint in the second projection image is not the subject of an annotation.
[0037] The problem to be addressed is thus the following: having two pixels coming from two different images, we want to know if these two pixels see the same 3D point of the scene. One way to answer this question is to use a descriptor of an image region around each of the two pixels and then to calculate a distance between these two descriptors. Examples of descriptors are the following: SIFT (for "Scale-Invariant Feature Transform"), SURF (for "Speeded Up Robust Features"), BRIEF (for "Binary Robust Independent Elementary Features"), ORB (for "Oriented FAST and rotated BRIEF"), color histogram. The distance between the descriptors can be the Euclidean distance, a Hamming distance, a cosine distance, a correlation coefficient, etc. The measured distance is compared to a threshold in order to declare whether the two pixels see the same point or not.
[0038] As indicated previously, the determination of a vehicle footprint is typically repeated to determine a vehicle footprint when acquiring another image of the sequence taken as a second reference image and projecting this footprint into the first projection image. The method then also comprises annotating as a traversable area an image region corresponding to the projection into the first projection image of the vehicle footprint when acquiring the second reference image.
[0039] It is also possible (in particular due to a low image acquisition frequency in view of the forward speed of the vehicle) that the profitable areas visible on an image are disjointed in places. For example, as shown in [Fig. 6], the RI region of the first projection image corresponding to the projection of the vehicle's footprint on the ground during the acquisition of the first image reference image and the region R2 of the first projection image corresponding to the projection of the vehicle's footprint during the acquisition of the second reference image are disjoint. In such a case, the method may comprise a step INT consisting of annotating as a traversable zone an intermediate region RI in the first projection image between the two disjoint image regions RI, R2 mentioned above. This intermediate zone may be automatically delimited, for example by joining as shown the edges of the 2D shape of the vehicle envelope.
[0040] The invention is not limited to the methods previously described but also extends to a system comprising a processor configured to implement one or other of these methods as well as to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of one or other of these methods. The invention also extends to a system comprising a first processor configured to implement the steps of generating the training and learning data and a second processor configured to implement the steps of obtaining the images of the scene and predicting the traversability of the scene by means of the model which has been the subject of the training.
Claims
1. Claims A computer-implemented method for assisting in the navigation of a mobile system, comprising: - the generation of training data for machine learning of a model for predicting the traversability of a terrain by the mobile system, said generation comprising: • obtaining (OBT) a sequence of images of an environment acquired by a monocular camera (2) on board a vehicle (1) moving in the environment; • when acquiring an image from the sequence forming a first reference image and while the vehicle is driving over a region, determining (EMP) a footprint of the vehicle (EN) on this region; • automatic annotation, as an area traversable by the mobile system, of an image region (RI) corresponding to the projection of said print into another image of the sequence forming a first projection image; - machine learning of the model using training data; - obtaining images of a scene acquired by a camera on board the mobile system; and - prediction of scene traversability by providing images of the scene to the machine-learned model, wherein generating the training data further comprises: - obtaining inertial data acquired by an inertial unit on board the vehicle moving in the environment; - the determination (SLAM), by simultaneous localization and mapping using the images of the sequence and the inertial data, of the position of the monocular camera in a reference frame of the environment during the acquisition of the images of the sequence; and in which the determination of the vehicle's footprint during the acquisition of the first reference image is carried out from the position of the camera in the reference frame of the environment during the acquisition of the first reference image.
2. The method of claim 1, wherein generating the training data further comprises calculating a similarity score (FILT) between the image region corresponding to the projection of said fingerprint in the first projection image and an image region corresponding to the projection of said fingerprint in another image of the sequence forming a second projection image and, when the similarity score is less than a threshold, the image region corresponding to the projection of said fingerprint in the second projection image is not annotated.
3. Method according to one of claims 1 and 2, wherein during the generation of the training data the determination of a footprint of the vehicle is reiterated to determine a footprint of the vehicle during the acquisition of another image of the sequence forming a second reference image and further comprising the annotation as a traversable zone of an image region corresponding to the projection in the first projection image of the footprint of the vehicle during the acquisition of the second reference image.
4. The method of claim 3, wherein generating the training data further comprises annotating (INT) as a traversable area an intermediate region (RI) in the first projection image between the image region (RI) corresponding to the projection of the vehicle footprint when acquiring the first reference image and the image region (R2) corresponding to the projection of the vehicle footprint when acquiring the second reference image.
5. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to one of claims 1 to 4.
6. Data processing system comprising a first processor configured to implement the steps of generating training data and learning the method according to one of the claims 1 to 4 and a second processor configured to implement the steps of obtaining images of the scene and predicting the method according to one of claims 1 to 4.