Device and method for navigation assistance based on monocular imaging

Abstract

The invention relates to a navigation aid device for use on a mobile system. This device comprises: a monocular camera capable of simultaneously acquiring a first image of a scene (In) with a first depth of field and at least one second image of the scene (If) with a second depth of field less than the first depth of field; a depth estimation unit (23) configured to determine a depth map (Dm) of the scene from the first image of the scene and at least one second image of the scene; and a computer vision unit (24) configured to calculate a navigation path from the first image of a scene and the depth map of the scene. Figure 2

Description

Title of the invention: Device and method for navigation assistance based on monocular imaging technical field

[0001] The field of the invention is that of navigation assistance for a mobile system such as a robot or autonomous vehicle. The invention relates more particularly to the calculation of a navigation trajectory for the mobile system from an RGB-D image of a scene, namely the combination of a color image of the scene and a depth map characterizing the distance of objects seen in the image. Previous technique

[0002] The calculation of a navigation trajectory can be implemented using different computer vision algorithms which take as input an image of a scene and a depth map of the same scene to provide as output a navigation solution for example with obstacle and / or collision avoidance.

[0003] Depth map estimation methods are generally divided into two categories: active methods where a light source is projected into the scene and passive methods which are based only on the acquisition of images of the scene illuminated by ambient light.

[0004] Among passive methods, a distinction is made between approaches based on multi-view geometry (for example stereovision in the case of two views) and approaches based on monocular images.

[0005] One of the major difficulties with multi-view systems concerns the complexity of matching points between images from different views in the case of sparsely textured scenes. Furthermore, the accuracy of this type of system depends heavily on the distance between the acquisition points of the different images (stereo basis).

[0006] In monocular image-based approaches, a single view of the scene is assumed to contain sufficient cues to estimate the depth map. On the one hand, these depth cues are not directly accessible in the image, and on the other hand, the transformation between these measurements and the depth map is not trivial. Thus, neural networks have been used to solve these two tasks.

[0007] It has been shown that the performance of this monocular approach is improved when using a blur index produced by the camera optics. However, using a camera that focuses on a given plane to acquire an image exhibiting depth-defocus blur does not allow The goal is to have a sharp image at every point (also called an all-in-focus image) that can be used in other vision tasks. Furthermore, using a separate all-in-focus camera for other tasks presents the problem of alignment. Another solution is to use multiple images that focus on different planes. The drawback of these methods lies in the image acquisition process, which requires alignment. Indeed, when using a single varifocal camera, the images cannot be aligned when the camera is mounted in a mobile system. Using multiple cameras introduces the problem of parallax.

[0008] Furthermore, in this type of approach, the all-in-focus image is estimated and not acquired by the camera, which is a source of error in the case of weakly textured areas. Other approaches exist that are based on the difference in blur between two (or more) images. The drawback of this type of approach is the lack of use of other depth cues that improve depth estimation performance, in addition to the absence of all-in-focus images. These cues can be, like blur, geometric in nature (perspective in the image, or the distance of objects from the horizon line) as well as semantic in nature (textures, the relative size of objects, occlusions).

[0009] In summary, it has been shown that camera focus blur is a useful indicator for significantly improving depth estimation performance. However, an image containing blur degrades the performance of other computer vision tasks (segmentation, detection, etc.) necessary for calculating the navigation trajectory. Using two cameras to acquire two images, one with focus blur and the other sharp throughout, does not offer a suitable solution because aligning two images with a parallax effect, one of which is blurry, is a difficult task and a source of error. The alternative solution of acquiring two images with the same camera by changing the focal length parameters cannot be considered in mobile systems because the two images will not be acquired at the same time and therefore will not be aligned. Description of the invention

[0010] The invention aims to provide a solution based on a single camera mounted on a mobile system that can improve the performance of depth estimation without degrading the vision tasks necessary for calculating the navigation trajectory of the mobile system.

[0011] To this end, the invention proposes a navigation aid device intended to be embedded on a mobile system, comprising a computer vision unit configured to calculate a navigation trajectory from a first image of a scene and a scene depth map. This device further includes a monocular camera capable of simultaneously acquiring the first scene image with a first depth of field and at least one second scene image with a second depth of field less than the first depth of field. This device also includes a depth estimation unit configured to determine the scene depth map from the first scene image and at least one second scene image.

[0012] Some preferred but not limiting aspects of this device are as follows: - to determine the depth map of the scene the depth estimation unit uses a machine learning model, for example a pre-trained neural network; - the depth estimation unit includes two different feature extraction branches to calculate feature maps respectively of the first image and at least one second image and an encoder-decoder taking as input the feature maps calculated by the two feature extraction branches to determine the depth map; - the monocular camera includes a lens with a first focal length, a lens with a second focal length greater than the first focal length and a splitter capable of directing an incoming light flux towards each of the lens with the first focal length and the lens with the second focal length; - to calculate the navigation trajectory, the computer vision unit also uses at least one second image of the scene acquired by the monocular camera; - the monocular camera is capable of simultaneously acquiring the first image of the scene and a plurality of second images of the scene, the second images presenting a focus on different planes of the scene; - the first image is a sharp image at all points and at least one second image shows a depth-of-focus blur.

[0013] The invention also relates to a method for assisting the navigation of a mobile system, comprising a step of calculating a navigation trajectory from a first image of a scene and a depth map of the scene. This method further comprises a step of simultaneously acquiring the first image of the scene with a first depth of field and at least one second image of the scene with a second depth of field less than the first depth of field. This method also comprises a step of determining the depth map of the scene from the first image of the scene and at least one second image of the scene.

[0014] Some preferred but not limiting aspects of this device are as follows: - the depth map determination step is executed using a machine learning model that takes as input the first image of the scene and at least one second image of the scene and provides as output the depth map of the scene; - the machine learning model includes two different feature extraction branches to calculate feature maps respectively of the first image and at least one second and an encoder-decoder taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

[0015] The invention extends to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of the above-mentioned process of determining the depth chart and calculating the navigation trajectory.

[0016] The invention also relates to a method for training a machine learning model that takes as input a pair of images and provides as output a depth map, comprising: - the acquisition, by the monocular camera of the aforementioned device, of a pair of training images consisting of images simultaneously acquired from a scene; - the reception of a depth map of the scene imaged by the pair of training images; - the processing of the pair of training images by the machine learning model; - the calculation of a representative value of the performance of the machine learning model by comparing the depth map of the scene provided as output of the machine learning model and the depth map of the scene associated with the pair of training images.

[0017] The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the processing and calculation steps of the training method. Brief description of the drawings

[0018] Other aspects, objects, advantages and features of the invention will become clearer upon reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the drawings attached to which:

[0019] - Fig. 1 is a diagram illustrating the device according to the invention mounted on a mobile system;

[0020] - [Fig.2] is a diagram of a device according to the invention;

[0021] - [Fig. 3] is a diagram illustrating how the blur index makes it possible to achieve an es depth estimation;

[0022] - [Fig. 4] is a diagram of a first example of an embodiment of the camera mo nocular of the device according to the invention;

[0023] - [Fig. 5] is a diagram of a second embodiment of the camera mo nocular of the device according to the invention;

[0024] - [Fig. 6] is a diagram of a possible embodiment of the estimation unit depth.

[0025] DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0026] With reference to [Fig. 1], the invention relates to a navigation aid device intended to be mounted on a mobile system 20, for example, a robot-type system or a drone dedicated to area reconnaissance, building exploration, or the transport of materials. The navigation aid device comprises a monocular camera 21 and a data processing module 22 configured to take as input the images acquired by the monocular camera.

[0027] With reference to [Fig. 2], the data processing module includes a computer vision unit 24 configured to calculate a navigation path from a first image In of a scene imaged by the monocular camera 21 and a depth map of the scene Dm. The navigation path may incorporate obstacle or collision avoidance.

[0028] The data processing module also includes a depth estimation unit 23 configured to determine the depth map Dm of the scene.

[0029] According to the invention, the monocular camera 21 is capable of simultaneously acquiring the first image of the scene In with a first depth of field and at least a second image of the scene If with a second depth of field less than the first depth of field. Since the first image of the scene and at least one second image of the scene are simultaneously acquired by a monocular camera, they image the scene from the same viewpoint.

[0030] The depth estimation unit 23 is configured to determine the depth map Dm of the scene from the first image of the scene In and at least one second image of the scene If. In one possible embodiment, the computer vision unit 24 also uses at least one second image of the scene If to calculate the navigation path.

[0031] With a second depth of field less than the first depth of field, the second image If forms a blurry image of the scene while the first image In forms a sharp image of the scene.

[0032] The first depth of field is preferably selected so that the first image forms an all-in-focus image that is sharp at every point. In particular, for the acquisition of the first image, the monocular camera can be adapted to achieve focus at the hyperfocal distance. In this way, the first image has a range of sharpness extending from half this hyperfocal distance to infinity. The second depth of field is such that the second image exhibits a depth-defocusing blur.

[0033] The monocular camera 21 thus makes it possible to acquire a pair of images of the scene, one of which is sharp and the other out of focus. Furthermore, since both images are acquired from the same viewpoint and simultaneously, alignment between the two images is achieved directly. The In, If images acquired by the camera are typically RGB images of the scene. The device is therefore functional in visible light. In an alternative embodiment, the functionality of the device is extended to nighttime operation by exploiting a different wavelength range (typically infrared).

[0034] The first image In is used for various vision tasks requiring a good quality image, such as localization and mapping, semantic segmentation, or detection and tracking. The first image In is thus used by the computer vision unit 24 to develop the navigation trajectory.

[0035] The second image, which focuses on a plane and contains blur that varies with depth, significantly improves depth estimation performance. Furthermore, according to the invention, the sharp image is also used in depth estimation. Combining the blur index present in the second image, which displays the focus blur, with the depth indices present in the sharp image (which are geometric, such as perspective or the elevation of objects relative to the image's horizon line, or semantic, such as indices of texture detail level as a function of distance, the relative size of objects in the scene with respect to their distance from the camera, or the occlusion of objects between them) significantly improves the performance of the depth estimation task.

[0036] Figure 3 illustrates the focus blur effect. It can be seen that points 11 and 12 located at different distances from a lens 9 produce optical spots of different diameters 13 and 14 on a photosensitive sensor 10. Thus, the blur diameter of a point on the sensor 10 contains an important indication of the depth of that point in the scene. However, the relationship between the blur diameter and the distance of a point is not bijective. Indeed, there are two points, one in front of the focal plane. The two areas are located behind the focal plane, producing two blur patches of the same diameter. This ambiguity is implicitly resolved by the depth estimation unit 23.

[0037] Figure 4 illustrates a possible embodiment of the monocular camera of the navigation aid device according to the invention. The camera 1 comprises an acquisition system including an input optic 3 for imaging the surface of an object 2 in the scene within the acquisition system, and a beam splitter 4 (for example, a semi-reflective mirror) that directs an incoming light beam to two lenses 5 and 7, one of which has a shorter focal length than the other. The light beam is then integrated by the two photosensitive sensors 6 and 8, thus providing the sharp image In and the blurred image If. The advantage of this acquisition system is that it allows the production of two images of the same scene captured simultaneously without any parallax effect between them.

[0038] Figure 5 illustrates another possible embodiment of the monocular camera 1 of the navigation aid device according to the invention. In this embodiment, the acquisition system is configured to allow the simultaneous acquisition of N images of the scene, including one sharp image and Nl blurred images that are in focus at different planes of the scene. This embodiment is advantageous because it allows for more blur gradient measurements and consequently further improves the accuracy of the results obtained for depth map estimation. Moreover, this configuration allows for the direct resolution of depth ambiguity as a function of the radius of the blur spot. With blurred images at various focus points and one sharp image, the depth estimation unit is indeed able to estimate the order relationship between the different sharp planes of an image and the blurred plane of that same image.

[0039] In one possible embodiment, the depth estimation unit 24 uses a machine learning model, for example a pre-trained neural network such as a convolutional neural network (CNN for Convolutional Neural Networks).

[0040] This machine learning model takes as input the first image of the scene and at least one second image of the scene and provides as output the depth map of the scene. The architecture of the machine learning model is preferably adapted to performing a pixel-by-pixel regression task, in this case calculating, for each pixel, the distance of the object represented at that pixel from the camera.

[0041] With reference to [Fig. 6], the machine learning model may include two different feature extraction branches EXn, EXf to compute feature maps respectively of the net image In and of the image(s) blurry If branches. These image feature extraction branches include successive convolutional layers followed by nonlinearities such as data normalization functions, dimensionality reduction functions, or nonlinear reprojection functions like, among others, the sigmoid or the rectified linear unit. The features extracted from the sharp image and the blurred image(s) by each of the EXn and EXf branches are then delivered to an encoder-decoder that includes an ENC encoder, typically a convolutional neural network, responsible for reducing the dimensionality of the data, and a DEC decoder that takes as input the reduced-dimension features produced by the encoder in order to predict the scene depth map Dm.This decoder is also typically a convolutional neural network whose goal is to recover the spatial dimension as input to the autoencoder while calculating the features necessary for decoding the features.

[0042] The invention is not limited to the device as described above, but also extends to a method for assisting navigation in a mobile system. With reference to [Fig. 2], this method comprises: - a simultaneous acquisition step (ACQ) of a first image of a scene In with a first depth of field and at least a second image of the scene If with a second depth of field less than the first depth of field, - an EST-D determination step of a depth map of scene Dm from the first image of scene In and at least one second image of scene If, ​​and - a NAV calculation step of a navigation trajectory from the first image of a scene In and the depth map of the scene Dm.

[0043] The invention also extends to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the aforementioned steps of determining the depth map and calculating the navigation trajectory.

[0044] The invention also relates to a method for training a machine learning model that takes as input a pair of images and provides as output a depth map. This method follows an iterative process which includes: - the acquisition, by the monocular camera of the device according to the invention, of a pair of training images consisting of images simultaneously acquired of a scene; - the reception of a depth map of the scene imaged by the pair of training images, said map having been obtained by a third-party method, for example a depth map acquired by a Lidar, by means of of structured light, by a time-of-flight camera, by stereovision, etc.; - the processing of the pair of training images by the machine learning model; - the calculation of a representative value of the performance of the machine learning model by comparing the depth map of the scene provided as output of the machine learning model and the depth map of the scene associated with the pair of training images.

[0045] The parameters of the learning model, for example the connection weights in the case of a neural network, are then adjusted to reduce the prediction error of the depth map. For example, the error gradient can be calculated to determine a direction of variation, and a shift in the opposite direction to the gradient is then made.

[0046] This training can be performed using a database of image pairs, each associated with a depth map, divided into training image pairs and test image pairs. As previously mentioned, training a neural network consists of determining the value of each of its weights. The neural network processes a pair of training images and makes a prediction as output. Knowing the pixel-by-pixel depth of each training image, it is possible to verify whether this prediction is correct. Depending on the accuracy of this prediction, the network's weights are updated, for example, according to the backpropagation algorithm. This process is repeated with all the training image pairs.Once the training is complete, it is possible to evaluate the trained model by presenting it with the test image pairs and comparing the model's outputs with the depth maps associated with the test image pairs.

[0047] The invention also extends to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the processing and calculation steps of the machine learning model training method.

[0048] The invention offers the following advantages: - a low cost compared to an active depth map determination system, such as a Lidar-based system; - improved estimation performance using a key monocular depth cue, namely blur, - the use of all combined depth indices, thanks to the clear image; - Eliminating the image alignment step thanks to the system acquisition (images acquired at the same instant and without parallax effect); The acquisition, and not the estimation, of an all-in-focus image usable in other vision tasks.

Claims

Claims

1. Navigation aid device intended to be embedded on a mobile system, comprising a computer vision unit (24) configured to calculate a navigation trajectory from a first image of a scene (In) and a depth map of the scene (Dm), characterized in that it further comprises: a monocular camera capable of simultaneously acquiring the first image of the scene (In) with a first depth of field and at least one second image of the scene (If) with a second depth of field less than the first depth of field, and a depth estimation unit (23) configured to determine the depth map of the scene (Dm) from the first image of the scene and the at least one second image of the scene.

2. Device according to claim 1, wherein to determine the depth map of the scene the depth estimation unit (23) uses a machine learning model, for example a pre-trained neural network.

3. Device according to claim 2, in which the depth estimation unit comprises two different feature extraction branches (EXn, EXf) for calculating feature maps respectively of the first image (In) and of the at least one second image (If) and an encoder-decoder (ENC, DEC) taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

4. Device according to one of claims 1 to 3, in which the monocular camera comprises a lens (5) having a first focal length, a lens (7, 15, N) having a second focal length greater than the first focal length and a splitter (4) capable of directing an input light flux towards each of the lens having the first focal length and the lens having the second focal length.

5. Device according to one of claims 1 to 4, in which to calculate the navigation trajectory the computer vision unit (24) also uses the at least one second image of the scene acquired by the monocular camera.

6. Device according to one of claims 1 to 5, in which the monocular camera is capable of simultaneously acquiring the first image of the scene and a plurality of second images of the scene, the second images showing focus on different planes of the scene.

7. Device according to one of claims 1 to 6, in which the first image is a sharp image at all points and the at least one second image has a depth-dependent defocus blur.

8. Method for assisting the navigation of a mobile system, comprising a step of calculating (NAV) a navigation trajectory from a first image of a scene (In) and a depth map of the scene (Dm), characterized in that it further comprises a step of simultaneous acquisition (ACQ) of the first image of the scene (In) with a first depth of field and at least one second image of the scene (If) with a second depth of field less than the first depth of field and a step of determining the depth map of the scene (EST-D) from the first image of the scene (In) and the at least one second image of the scene (If).

9. The method of claim 8, wherein the step of determining the depth map is performed by means of a machine learning model taking as input the first image of the scene (In) and the at least one second image of the scene (If) and providing as output the depth map of the scene.

10. Method according to claim 9, in which the machine learning model comprises two different feature extraction branches (EXn, EXf) for calculating feature maps respectively of the first image (In) and of the at least one second image (If) and an encoder-decoder (ENC, DEC) taking as input the feature maps calculated by the two feature extraction branches to determine the depth map (Dm).

11. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to one of claims 8 to 10 for determining the depth map and calculating the navigation trajectory.

Images