Navigation assistance device and method based on monocular imaging

Abstract

The invention relates to a navigation assistance device intended to be installed on board a moving system. This device comprises: a monocular camera capable of simultaneously acquiring a first image of a scene (In) with a first depth of field and at least one second image of a scene (If) with a second depth of field smaller than the first depth of field, a depth-estimating unit (23) configured to determine a depth map (Dm) of the scene from the first image of the scene and the at least one second image of the scene; a computer vision unit (24) configured to calculate a navigation path from the first image of the scene and from the depth map of the scene.

Description

[0001] Device and method for assisting navigation based on monocular imaging

[0002] TECHNICAL FIELD

[0003] The field of the invention is that of navigation assistance for a mobile system of the robot or autonomous vehicle type. The invention relates more particularly to the calculation of a navigation trajectory for the mobile system from an RGB-D image of a scene, namely the combination of a color image of the scene and a depth map characterizing the distance of the objects seen in the image.

[0004] PRIOR ART

[0005] The calculation of a navigation trajectory can be implemented using different computer vision algorithms that take as input an image of a scene and a depth map of the same scene to provide as output a navigation solution for example with obstacle and / or collision avoidance.

[0006] Methods for estimating a depth map are generally divided into two categories: active methods where a light source is projected into the scene and passive methods which are based only on acquiring images of the scene illuminated by ambient light.

[0007] Among the passive methods, we distinguish approaches based on multi-view geometry (e.g. stereovision in the case of two views) and approaches based on monocular images.

[0008] One of the major difficulties of multi-view systems concerns the complexity of matching points between images from different views in the case of poorly textured scenes. In addition, the accuracy of this type of system strongly depends on the distance between the acquisition points of the different images (stereo base).

[0009] In monocular image-based approaches, a single view of the scene is considered to contain sufficient cues to estimate the depth map. On the one hand, these depth cues are not directly accessible in the image and on the other hand, the transformation between these measurements and the depth map is not trivial. Thus, neural networks have been used to solve these two tasks.

[0010] It has been shown that the performance of this monocular approach is improved when using a blur index produced by the camera optics. However, using a camera that focuses on a given plane to acquire an image with depth-dependent defocus blur does not provide an all-in-focus image that can be used in other vision tasks. In addition, using another all-in-focus camera for other tasks raises the problem of alignment. Another solution is to use several images that focus on different planes. The disadvantage of these methods lies in the acquisition of these images, which requires alignment. Indeed, when using a single variable-focal camera, the images cannot be aligned when the camera is embedded in a mobile system.The case of using multiple cameras poses the problem of the presence of parallax.

[0011] Furthermore, in this type of approach, the all-in-focus image is estimated and not acquired by the camera, which is a source of error in the case of weakly textured areas. There are other approaches that are based on the difference in blur between two (or more) images. The disadvantage of this type of approach is the lack of use of other depth indices that allow improving depth estimation performance in addition to the absence of all-in-focus images. These indices can be, like blur, geometric in nature (the perspective in the image, or the distance of objects from the horizon line) as well as semantic in nature (textures, relative size of objects, occlusions).

[0012] In summary, it is demonstrated that the focus blur of a camera is an index allowing to significantly improve the performances of the depth estimation. However, an image containing the blur degrades the performances of the other computer vision tasks (segmentation, detection, etc.) necessary for the calculation of the navigation trajectory. The use of two cameras to acquire two images, one with a focus blur and the other sharp everywhere does not offer a relevant solution because the alignment of two images with a parallax effect between them, one of which is blurred, is a difficult task which is a source of error. The other solution which consists of acquiring two images by the same camera by changing the focal parameters, cannot be considered in mobile systems, because the two images will not be acquired at the same time and therefore not aligned.

[0013] STATEMENT OF THE INVENTION

[0014] The invention aims to propose a solution based on a single camera mounted on a mobile system which can improve the performance of depth estimation without degrading the vision tasks necessary for calculating the navigation trajectory of the mobile system.

[0015] To this end, the invention proposes a navigation assistance device intended to be embedded on a mobile system, comprising a computer vision unit configured to calculate a navigation trajectory from a first image of a scene and a depth map of the scene. This device further comprises a monocular camera capable of simultaneously acquiring the first image of the scene with a first depth of field and at least one second image of the scene with a second depth of field less than the first depth of field. This device also comprises a depth estimation unit configured to determine the depth map of the scene from the first image of the scene and the at least one second image of the scene.

[0016] Some preferred but non-limiting aspects of this device are as follows: to determine the depth map of the scene the depth estimation unit uses a machine learning model, for example a pre-trained neural network; the depth estimation unit comprises two different feature extraction branches for calculating feature maps respectively of the first image and of the at least one second image and an encoder-decoder taking as input the feature maps calculated by the two feature extraction branches to determine the depth map; the monocular camera comprises a lens having a first focal length, a lens having a second focal length greater than the first focal length and a splitter capable of directing an input light flux towards each of the lens having the first focal length and the lens having the second focal length;to calculate the navigation trajectory the computer vision unit also uses the at least one second image of the scene acquired by the monocular camera; the monocular camera is capable of simultaneously acquiring the first image of the scene and a plurality of second images of the scene, the second images having a focus in different planes of the scene; the first image is a sharp image at all points and the at least one second image has a depth-dependent defocus blur.;

[0017] The invention also relates to a method for assisting the navigation of a mobile system, comprising a step of calculating a navigation trajectory from a first image of a scene and a depth map of the scene. This method further comprises a step of simultaneously acquiring the first image of the scene with a first depth of field and at least one second image of the scene with a second depth of field less than the first depth of field. This method also comprises a step of determining the depth map of the scene from the first image of the scene and the at least one second image of the scene.

[0018] Some preferred but non-limiting aspects of this device are as follows: the step of determining the depth map is executed by means of a machine learning model taking as input the first image of the scene and the at least one second image of the scene and providing as output the depth map of the scene; the machine learning model comprises two different feature extraction branches for calculating feature maps respectively of the first image and of the at least one second and an encoder-decoder taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

[0019] The invention extends to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the above-mentioned method of determining the depth map and calculating the navigation trajectory.

[0020] The invention also relates to a method of training a machine learning model taking as input a pair of images and providing as output a depth map, comprising:

[0021] - the acquisition, by the monocular camera of the aforementioned device, of a pair of training images consisting of simultaneously acquired images of a scene;

[0022] - receiving a depth map of the scene imaged by the pair of training images;

[0023] - processing the training image pair by the machine learning model;

[0024] - calculating a representative value of the performance of the machine learning model by comparing the depth map of the scene provided as output from the machine learning model and the depth map of the scene associated with the pair of training images.

[0025] The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to carry out the processing and calculation steps of the training method

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings in which: - figure 1 is a diagram illustrating the device according to the invention mounted on a mobile system;

[0028] - figure 2 is a diagram of a device according to the invention;

[0029] - Figure 3 is a diagram illustrating how the blur index makes it possible to perform a depth estimation;

[0030] - figure 4 is a diagram of a first exemplary embodiment of the monocular camera of the device according to the invention;

[0031] - figure 5 is a diagram of a second exemplary embodiment of the monocular camera of the device according to the invention;

[0032] - Figure 6 is a diagram of a possible embodiment of the depth estimation unit.

[0033] DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0034] With reference to Figure 1, the invention relates to a navigation aid device intended to be embedded on a mobile system 20, for example a robot-type system or a drone dedicated to the reconnaissance of an area, the exploration of buildings or the transport of materials. The navigation aid device comprises a monocular camera 21 and a data processing module 22 configured to take as inputs the images acquired by the monocular camera.

[0035] With reference to FIG. 2, the data processing module comprises a computer vision unit 24 configured to calculate a navigation trajectory from a first image In of a scene imaged by the monocular camera 21 and a depth map of the scene Dm. The navigation trajectory may integrate obstacle or collision avoidance.

[0036] The data processing module further comprises a depth estimation unit 23 configured to determine the depth map Dm of the scene.

[0037] According to the invention, the monocular camera 21 is capable of simultaneously acquiring the first image of the scene In with a first depth of field and at least one second image of the scene If with a second depth of field less than the first depth of field. The first image of the scene and the at least one second image of the scene being simultaneously acquired by a monocular camera, they image the scene from the same point of view.

[0038] The depth estimation unit 23 is configured to determine the depth map Dm of the scene from the first image of the scene In and the at least one second image of the scene If. In a possible embodiment, the computer vision unit 24 also uses the at least one second image of the scene If to calculate the navigation trajectory.

[0039] With a second depth of field smaller than the first depth of field, the second If image forms a blurred image of the scene while the first In image forms a sharp image of the scene.

[0040] The first depth of field is preferably selected such that the first image forms an all-in-focus image that is sharp at all points. In particular, for the acquisition of the first image, the monocular camera may be adapted to focus at the hyperfocal distance. In this way, the first image has a sharpness range that extends from half of this hyperfocal distance to infinity. The second depth of field is such that the second image has depth-dependent defocus blur.

[0041] The monocular camera 21 thus makes it possible to acquire a pair of images of the scene, one of which is sharp and the other of which has a focus blur. Furthermore, since the two images are acquired from the same point of view and at the same time, the alignment between the two images is obtained directly. The images In, If acquired by the camera are typically RGB images of the scene. The device is then functional in visible light. In an alternative embodiment, the functionality of the device is extended to nighttime operation by exploiting another wavelength range (typically infrared).

[0042] The first image In is used for various vision tasks requiring a good quality image such as localization and mapping, semantic segmentation or detection and tracking. The first image In is thus used by the computer vision unit 24 in order to develop the navigation trajectory. The second image, which focuses on a plane and contains the blur which varies according to the depth, makes it possible to significantly improve the performance of the depth estimation. Furthermore, according to the invention, the sharp image is also used in the depth estimation.The combination of the blur index present in the second image showing the focus blur with the depth indices present in the sharp image (which are geometrical such as the perspective or elevation of the objects relative to the horizon line of the image, or semantic such as the indices of level of detail of the textures as a function of the distance, of the relative size of the objects in the scene relative to their distance from the camera or even of occultations of the objects between them) makes it possible to significantly improve the performance of the depth estimation task.

[0043] Figure 3 illustrates the focus blur effect. It is noted that points 11 and 12 located at different distances from an optic 9 produce optical spots of different diameters 13 and 14 on a photosensitive sensor 10. Thus the blur diameter of a point on the sensor 10 contains an important indication of the depth of this point in the scene. However, the relationship between the blur diameter and the distance of a point is not bijective. Indeed, there are two points, one in front of the focusing plane and the other behind the focusing plane, which produce two blur spots of the same diameter. This ambiguity is removed implicitly by the depth estimation unit 23.

[0044] Figure 4 is a diagram illustrating a possible embodiment of the monocular camera of the navigation aid device according to the invention. The camera 1 comprises an acquisition system which includes an input optic 3 making it possible to image the surface of an object 2 of the scene inside the acquisition system, a splitter 4 (for example a semi-reflecting mirror) which makes it possible to direct an input light flux towards two lenses 5 and 7, one of which has a shorter focal length than the other. The light flux is then integrated by the two photosensitive sensors 6 and 8, thus making it possible to provide the sharp image In and the blurred image If. The advantage of this acquisition system is that it makes it possible to produce two images of the same scene captured at the same time and without parallax effect between them. Figure 5 is a diagram illustrating another possible embodiment of the monocular camera 1 of the navigation aid device according to the invention.In this embodiment, the acquisition system is configured to allow the simultaneous acquisition of N images of the scene including a sharp image and Nl blurred images which have a focus in different planes of the scene. This embodiment proves advantageous in that it makes it possible to obtain more blur gradient measurements and consequently to further improve the accuracy of the results obtained for the estimation of the depth map. In addition, this configuration makes it possible to directly remove the depth ambiguity as a function of the radius of the blur task. With blurred images of various focuses and a sharp image, the depth estimation unit is in fact able to estimate the order relationship between the different sharp planes of an image and the blurred plane of this same image.

[0045] In one possible embodiment, the depth estimation unit 24 uses a machine learning model, for example a pre-trained neural network such as a convolutional neural network (CNN).

[0046] This machine learning model takes as input the first image of the scene and the at least one second image of the scene and provides as output the depth map of the scene. The architecture of the machine learning model is preferably adapted to perform a pixel-to-pixel regression task, in this case the calculation, for each pixel, of the distance of the object that is represented at this pixel to the camera.

[0047] Referring to Figure 6, the machine learning model may comprise two different feature extraction branches EXn, EXf to calculate feature maps of the sharp image In and the blurred image(s) If respectively. These image feature extraction branches comprise successive convolution layers followed by nonlinearities such as data normalization functions, dimension reduction functions or nonlinear reprojection functions such as, among others, the sigmoid or the rectified linear unit.The features extracted from the sharp image and the blurred image(s) by each of the branches EXn and EXf are then delivered to an encoder-decoder which comprises an encoder ENC, typically a convolutional neural network, responsible for reducing the dimension of the data and a decoder DEC which takes as input the reduced dimension features produced by the encoder in order to predict the depth map of the scene Dm. This decoder is also typically a convolutional neural network whose purpose is to recover the spatial dimension at the input of the auto-encoder while calculating the features necessary for decoding the features.

[0048] The invention is not limited to the device as described above, but also extends to a method for assisting the navigation of a mobile system. With reference to Figure 2, this method comprises: a step of simultaneous acquisition ACQ. of a first image of a scene In with a first depth of field and at least one second image of the scene If with a second depth of field less than the first depth of field, a step of determining EST-D a depth map of the scene Dm from the first image of the scene In and the at least one second image of the scene If, ​​and a step of calculating NAV of a navigation trajectory from the first image of a scene In and the depth map of the scene Dm.

[0049] The invention also extends to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the above-mentioned steps of determining the depth map and calculating the navigation trajectory.

[0050] The invention further relates to a method for training a machine learning model taking as input a pair of images and providing as output a depth map. This method follows an iterative process which comprises:

[0051] - the acquisition, by the monocular camera of the device according to the invention, of a pair of training images consisting of simultaneously acquired images of a scene; - the reception of a depth map of the scene imaged by the pair of training images, said map having been obtained by a third-party method, for example a depth map acquired by a Lidar, by means of structured light, by a time-of-flight camera, by stereovision, etc.;

[0052] - processing the training image pair by the machine learning model;

[0053] - calculating a value representative of the performance of the machine learning model by comparing the depth map of the scene provided as output from the machine learning model and the depth map of the scene associated with the pair of training images.

[0054] The parameters of the learning model, for example the connection weights in the case of a neural network, are then adjusted to reduce the prediction error of the depth map. For example, the gradient of the error can be calculated to determine a direction of variation and a displacement in a direction opposite to the gradient is then carried out.

[0055] This training can be done using a database of image pairs, each associated with a depth map, divided into training image pairs and test image pairs. As mentioned earlier, training a neural network consists of determining the value of each of its weights. The neural network processes a pair of training images and makes a prediction as output. Knowing that the pixel-to-pixel depth of each of the training images is known, it is possible to check whether this prediction is correct. Depending on the accuracy of this prediction, the network's weights are updated, for example, using the error gradient backpropagation algorithm. This process is repeated with all the training image pairs.Once training is complete, it is possible to evaluate the trained model by presenting it with the test image pairs and comparing the model's outputs with the depth maps associated with the test image pairs.

[0056] The invention also extends to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the processing and calculation steps of the method for training the machine learning model.

[0057] The invention offers the following advantages: low cost compared to an active system for determining a depth map, such as a system using a Lidar, low cost compared to the Lidar-based system; better estimation performance using a key monocular depth index, namely blur, using all the depth indices combined, thanks to the sharp image;

[0058] Elimination of the image alignment step thanks to the acquisition system (images acquired at the same time and without parallax effect);

[0059] The acquisition, not the estimation, of an all-in-focus image usable in other vision tasks.

Claims

DEMANDS 1. Navigation aid device intended to be carried on a mobile system, comprising a computer vision unit (24) configured to calculate a navigation trajectory from a first image of a scene (In) and a depth map of the scene (Dm), characterized in that it further comprises: a monocular camera capable of simultaneously acquiring the first image of the scene (In) with a first depth of field and at least a second image of the scene (If) with a second depth of field less than the first depth of field, and a depth estimation unit (23) configured to determine the depth map of the scene (Dm) from the first image of the scene and at least a second image of the scene.

2. Device according to claim 1, wherein to determine the depth map of the scene the depth estimation unit (23) uses a machine learning model, for example a pre-trained neural network.

3. Device according to claim 2, wherein the depth estimation unit comprises two different feature extraction branches (EXn, EXf) for calculating feature maps respectively of the first image (In) and of at least one second image (If) and an encoder-decoder (ENC, DEC) taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

4. Device according to any one of claims 1 to 3, wherein the monocular camera comprises a lens (5) having a first focal length, a lens (7, 15, N) having a second focal length greater than the first focal length and a separator (4) capable of directing an input light flux towards each of the lens having the first focal length and the lens having the second focal length.

5. Device according to any one of claims 1 to 4, wherein to calculate the navigation trajectory the computer vision unit (24) also exploits at least one second image of the scene acquired by the monocular camera.

6. Device according to any one of claims 1 to 5, wherein the monocular camera is capable of simultaneously acquiring the first image of the scene and a plurality of second images of the scene, the second images having focus on different planes of the scene.

7. Device according to any one of claims 1 to 6, wherein the first image is a point-clear image and at least a second image exhibits depth-defocusing blur.

8. Method for aiding the navigation of a mobile system, comprising a calculation step (NAV) of a navigation trajectory from a first image of a scene (In) and a depth map of the scene (Dm), characterized in that it further comprises a simultaneous acquisition step (ACQ) of the first image of the scene (In) with a first depth of field and of at least a second image of the scene (If) with a second depth of field less than the first depth of field and a step of determining the depth map of the scene (EST-D) from the first image of the scene (In) and of at least a second image of the scene (If).

9. Method according to claim 8, wherein the depth map determination step is performed by means of a machine learning model taking as input the first image of the scene (In) and at least one second image of the scene (If) and providing as output the depth map of the scene.

10. A method according to claim 9, wherein the machine learning model comprises two different feature extraction branches (EXn, EXf) for computing feature maps of the first image (In) and the second image (In), respectively. less a second image (If) and an encoder-decoder (ENC, DEC) taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

11. Product: computer program comprising instructions which, when the program is executed by a computer, cause the computer to implement the steps of the method according to any one of claims 8 to 10 for determining the depth map and calculating the navigation trajectory.

Images