Navigation assisting equipment and method based on monocular imaging

Abstract

The invention relates to a navigation assistance device for mounting on a mobile system. The device comprises a monocular camera, a depth estimation unit (23) and a computer vision unit (24). The monocular camera is capable of simultaneously acquiring a first image (In) of the scene having a first depth of field and at least one second image (If) of the scene having a second depth of field less than the first depth of field. The depth estimation unit (23) is configured to determine a depth map (Dm) of the scene from a first image of the scene and at least one second image of the scene. The computer vision unit (24) is configured to calculate a navigation path from a first image of the scene and a depth map of the scene.

Description

Technical Field

[0001] The field of the invention is navigation assistance for mobile systems of autonomous robot or vehicle type. The invention more particularly relates to the calculation of navigation trajectories for mobile systems from an RGB-D image of a scene, i.e. a combination of a color image of the scene and a depth map characterizing the distances of objects seen in the image. Background Art

[0002] The computation of the navigation trajectory may be implemented using different computer vision algorithms which take as input an image of a scene and a depth map of the same scene to provide as output a navigation solution, e.g. with obstacle avoidance and / or collision prevention.

[0003] Methods for estimating depth maps generally fall into two categories: active methods that project a light source into the scene, and passive methods that are based solely on acquiring an image of the scene illuminated by ambient light.

[0004] Among passive methods, a distinction is made between methods based on multi-view geometry (e.g. stereo vision in the case of two views) and methods based on monocular images.

[0005] One of the main difficulties of multi-view systems involves the complexity of matching points between images from different views in the case of weakly textured scenes. Furthermore, the accuracy of this type of system strongly depends on the distances between the acquired points of the different images (stereo basis).

[0006] In monocular image based methods, a single view of the scene is assumed to contain enough metrics to estimate the depth map. On the one hand, these depth metrics are not directly accessible in the image, on the other hand, the conversion between these measurements and the depth map is non-trivial. Therefore, neural networks have been used to handle both tasks.

[0007] It has been demonstrated that the performance of this monocular approach improves in the case of using blur indicators generated by the optics of the camera. However, using a camera focused at a given plane to acquire images with deep defocus blur does not allow having sharp images at all points (the image is also called an all-in-focus image), which can be exploited in other vision tasks. In addition, using another all-in-focus camera for other tasks poses alignment problems. Another solution consists in using multiple images focused on different planes. The disadvantage of these methods is that the acquisition of these images requires alignment. In fact, in the case of using a single variable focal length camera, the images cannot be aligned when the camera is embedded on a mobile system. If multiple cameras are used, there will be parallax problems.

[0008] Furthermore, in this type of approach, the fully focused image is estimated rather than acquired by the camera, which is a source of error in the case of weakly textured areas. There are other approaches based on blurred differences between two (or more) images. A drawback of this type of approach is that, in addition to the absence of a fully focused image, other depth metrics that could improve the depth estimation performance are not used. Like blur, these metrics can be both geometric in nature (perspective in the image, or distance of an object relative to the horizon) and semantic in nature (texture, relative size of objects, occlusion).

[0009] In conclusion, it was demonstrated that the focus blur of the camera constitutes an indicator that can significantly improve the performance of depth estimation. However, images containing blur reduce the performance of other computer vision tasks (segmentation, detection, etc.) necessary to calculate the navigation trajectory. Using two cameras to acquire two images, one blurred in focus and the other completely sharp, does not provide a relevant solution, since it is difficult to align two images with a parallax effect between the two images, one of which is blurred, which is a source of error. Another solution consists in acquiring two images by the same camera by varying the focal length parameters, which is not conceivable in mobile systems, since the two images will not be acquired simultaneously and therefore will not be aligned. Summary of the invention

[0010] The present invention aims to propose a solution based on a single camera mounted on a mobile system, which can improve the depth estimation performance without compromising the visual task required to calculate the navigation trajectory of the mobile system.

[0011] To this end, the present invention proposes a navigation assistance device for embedding in a mobile system, the navigation assistance device comprising a computer vision unit, the computer vision unit being configured to calculate a navigation trajectory based on a first image of a scene and a depth map of the scene. The device also comprises a monocular camera, the monocular camera being capable of simultaneously acquiring a first image of the scene having a first depth of field and at least one second image of the scene having a second depth of field less than the first depth of field. The device also comprises a depth estimation unit, the depth estimation unit being configured to determine a depth map of the scene based on the first image of the scene and at least one second image of the scene.

[0012] Some preferred but non-limiting aspects of the device are as follows:

[0013] -To determine a depth map of the scene, the depth estimation unit uses a machine learning model, such as a pre-trained neural network;

[0014] - the depth estimation unit comprises two different feature extraction branches and an encoder-decoder. The two different feature extraction branches are used to respectively calculate a feature map of the first image and a feature map of at least one second image. The encoder-decoder takes the feature maps calculated by the two feature extraction branches as input to determine the depth map;

[0015] - a monocular camera including a lens having a first focal length, a lens having a second focal length greater than the first focal length, and a beam splitter capable of directing input light flux to each of the lens having the first focal length and the lens having the second focal length;

[0016] - for calculating the navigation trajectory, the computer vision unit also uses at least one second image of the scene acquired by the monocular camera;

[0017] - a monocular camera is capable of simultaneously acquiring a first image of a scene and a plurality of second images of the scene. The second images have focal points at different planes of the scene;

[0018] The first image is a sharp image at all points and at least one of the second images has a deep defocus blur.

[0019] The present invention also relates to a navigation assistance method for a mobile system, comprising the step of calculating a navigation trajectory based on a first image of a scene and a depth map of the scene. The method also comprises the step of simultaneously acquiring a first image of the scene having a first depth of field and at least one second image of the scene having a second depth of field less than the first depth of field. The method also comprises the step of determining a depth map of the scene based on the first image of the scene and the at least one second image of the scene.

[0020] Some preferred but non-limiting embodiments of the device are as follows:

[0021] - the step of determining the depth map is performed by means of a machine learning model which takes as input a first image of the scene and at least one second image of the scene and provides as output a depth map of the scene;

[0022] - The machine learning model includes two different feature extraction branches and an encoder-decoder. The two different feature extraction branches respectively calculate a feature map of a first image and a feature map of at least one second image. The encoder-decoder takes the feature maps calculated by the two feature extraction branches as input to determine a depth map (Dm).

[0023] The invention extends to a computer program product comprising instructions which, when executed by a computer, cause the computer to implement the steps of the above-described method for determining a depth map and calculating a navigation trajectory.

[0024] The present invention also relates to a method for training a machine learning model that takes an image pair as input and provides a depth map as output. The method comprises:

[0025] - acquiring a training image pair consisting of images of a scene acquired simultaneously by a monocular camera of the aforementioned device;

[0026] - receiving a depth map of a scene imaged by a training image pair;

[0027] - The machine learning model processes the training image pairs;

[0028] A value representing the performance of the machine learning model is calculated by comparing a depth map of the scene provided as an output of the machine learning model with a depth map of the scene associated with a training image pair.

[0029] The invention also relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the processing and calculation steps of the training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other aspects, objects, advantages and features of the invention will better emerge on reading the following detailed description of a preferred embodiment given by way of non-limiting example and made with reference to the accompanying drawings, in which:

[0031] Figure 1 is a diagram of a device according to the present invention mounted on a mobile system;

[0032] Figure 2 is a schematic diagram of an apparatus according to the present invention;

[0033] Figure 3 is a diagram showing how the blur index makes it possible to perform depth estimation;

[0034] Figure 4 is a diagram of a first exemplary embodiment of a monocular camera of an apparatus according to the present invention;

[0035] Figure 5 is a diagram of a second exemplary embodiment of a monocular camera of an apparatus according to the present invention;

[0036] Figure 6 is a diagram of one possible embodiment of a depth estimation unit. DETAILED DESCRIPTION

[0037] refer to Figure 1The present invention relates to a navigation assistance device to be embedded in a mobile system 20, such as a robot-type system or a drone dedicated to identifying areas, exploring buildings or transporting materials. The navigation assistance device comprises a monocular camera 21 and a data processing module 22, which is configured to take as input an image acquired by the monocular camera.

[0038] refer to Figure 2 The data processing module includes a computer vision unit 24, which is configured to calculate a navigation trajectory based on a first image In of a scene imaged by a monocular camera 21 and a depth map Dm of the scene. The navigation trajectory may include avoiding obstacles or preventing collisions.

[0039] The data processing module further comprises a depth estimation unit 23 configured to determine a depth map Dm of the scene.

[0040] According to the present invention, the monocular camera 21 can simultaneously acquire a first image In of a scene having a first depth of field and at least one second image If of the scene having a second depth of field less than the first depth of field. The first image of the scene and the at least one second image of the scene are acquired simultaneously by the monocular camera, and they image the scene according to the same viewpoint.

[0041] The depth estimation unit 23 is configured to determine a depth map Dm of the scene based on the first image In of the scene and the at least one second image If of the scene. In one possible embodiment, the computer vision unit 24 further calculates the navigation trajectory using the at least one second image If of the scene.

[0042] When the second depth of field is smaller than the first depth of field, the second image If forms a blurred image of the scene, while the first image In forms a clear image of the scene.

[0043] The first depth of field is preferably selected so that the first image forms a fully focused image that is sharp at all points. In particular, to acquire the first image, the monocular camera can be adapted to focus at a hyperfocal distance. In this way, the first image has a sharpness range extending from half of the hyperfocal distance to infinity. The second depth of field is such that the second image has a deep defocus blur.

[0044] Thus, the monocular camera 21 can acquire a pair of images of the scene, one of which is sharp and the other out of focus. Furthermore, since the two images are acquired simultaneously from the same viewpoint, the alignment between the two images is directly obtained. If the image In is acquired by the camera, the image In is typically an RGB image of the scene. The device then functions in visible light. In a variant, the functionality of the device is extended to nighttime operation by utilizing another wavelength range, typically infrared.

[0045] The first image In is used for various vision tasks that require high quality images, such as localization and mapping, semantic segmentation or detection and tracking. Therefore, the computer vision unit 24 utilizes the first image In in order to describe the navigation trajectory in detail.

[0046] A second image focused on a plane and containing a blur that varies according to depth can significantly improve the performance of depth estimation. In addition, according to the present invention, the clear image is also used for depth estimation. The combination of blur indicators present in the second image with focused blur and depth indicators present in the clear image (depth indicators have a geometric order, such as the perspective or elevation of an object relative to the horizon of the image, or depth indicators have a semantic order, such as an indicator of the functional relationship between the level of detail of the texture and the distance, the relative size of objects in the scene relative to their distance from the camera, or occlusion between objects) can significantly improve the performance of the depth estimation task.

[0047] Figure 3 The effect of focus blur is shown. It can be seen that points 11 and 12 at different distances from the optics 9 produce light spots 13 and 14 of different diameters on the light-sensitive sensor 10. Therefore, the blur diameter of a point on the sensor 10 contains an important indication about the depth of the point in the scene. However, the relationship between the blur diameter and the distance from the point is not bijective. Indeed, there are two points, one in front of the focus plane and one behind the focus plane, which produce two blur tasks of the same diameter. This ambiguity is implicitly removed by the depth estimation unit 23.

[0048] Figure 4 A diagram showing a possible embodiment of a monocular camera of a navigation aid according to the invention. The camera 1 comprises an acquisition system comprising an input optical device 3 and a beam splitter 4 (e.g. a half mirror) for imaging the surface of an object 2 of a scene within the acquisition system. The beam splitter 4 makes it possible to direct the input light flux to two lenses 5 and 7, one of which has a shorter focal length than the other. The light flux is then integrated by two light-sensitive sensors 6 and 8, making it possible to provide a sharp image In and a blurred image If. The advantage of this acquisition system is that it makes it possible to produce two images of the same scene captured simultaneously and without parallax effects between them.

[0049] Figure 5is a diagram showing another possible embodiment of a monocular camera 1 of a navigation aid according to the present invention. In this embodiment, the acquisition system is configured to allow the simultaneous acquisition of N images of a scene, including N-1 blurred images and one clear image with focus at different planes of the scene. The advantage demonstrated by this embodiment is that it can obtain more blur gradient measurements and thus further improve the accuracy of the results obtained for estimating the depth map. In addition, this configuration allows the direct removal of depth blur as a function of the radius of the blur task. For blurred images and clear images of various focal points, the depth estimation unit is indeed able to estimate the sequential relationship between different clear planes of the image and the blurred plane of the same image.

[0050] In one possible embodiment, the depth estimation unit 24 uses a machine learning model, for example a pre-trained neural network, such as a convolutional neural network (CNN).

[0051] The machine learning model takes as input a first image of the scene and at least one second image of the scene and provides as output a depth map of the scene. The architecture of the machine learning model is preferably adapted to perform a pixel-by-pixel regression task, in which case the distance of the object represented at that pixel to the camera is calculated for each pixel.

[0052] refer to Figure 6 , the machine learning model may include two different feature extraction branches EXn, EXf to respectively calculate the feature maps of the clear image In and one or more blurred images If. These image feature extraction branches include consecutive convolutional layers followed by nonlinearities such as data normalization functions, dimensionality reduction functions, or nonlinear reprojection functions such as sigmoid or rectified linear units. The features extracted from the clear image and one or more blurred images by each of the branches EXn and EXf are then transmitted to an encoder-decoder, which includes an encoder (encoder, ENC) and a decoder (decoder, DEC). The encoder is usually a convolutional neural network responsible for reducing the dimensionality of the data, and the decoder takes the reduced dimensional features produced by the encoder as input in order to predict the depth map of the scene Dm. The decoder is usually also a convolutional neural network, whose purpose is to restore the spatial dimensions at the input of the autoencoder while calculating the features required for decoding the features.

[0053] The present invention is not limited to the above-mentioned devices, but can also be extended to a navigation assistance method for a mobile system. Figure 2 , the method comprises the following steps:

[0054] - simultaneously acquiring a first image In of a scene having a first depth of field and at least one second image If of the scene having a second depth of field smaller than the first depth of field;

[0055] - determining a depth map Dm of the EST-D scene from a first image In of the scene and at least one second image If of the scene;

[0056] - Calculate the navigation NAV trajectory based on the first image In of the scene and the depth map Dm of the scene.

[0057] The present invention also extends to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the above-mentioned steps of determining a depth map and calculating a navigation trajectory.

[0058] Furthermore, the present invention relates to a method for training a machine learning model that takes an image pair as input and provides a depth map as output. The method follows an iterative process that comprises:

[0059] - acquiring, by a monocular camera of a device according to the invention, pairs of training images consisting of images of a scene acquired simultaneously;

[0060] - receiving a depth map of the scene imaged by the training image pair, said map having been obtained by a third party method, such as a depth map acquired by LiDAR, by structured light, by a time-of-flight camera, by stereo vision, etc.;

[0061] - The training image pairs are processed by the machine learning model;

[0062] -Calculating a value representing the performance of the machine learning model by comparing a depth map of the scene provided at the output of the machine learning model with a depth map of the scene associated with a training image pair.

[0063] The parameters of the learning model, such as the connection weights in the case of a neural network, are then adjusted to reduce the depth map prediction error. For example, the gradient of the error can be calculated to determine the direction of change, and then a displacement in the direction opposite to the gradient is performed.

[0064] This training can be performed using a database of image pairs, each image pair being associated with a depth map, divided into training image pairs and test image pairs. As previously mentioned, the training of a neural network consists in determining the value of each of its weights. The neural network processes the training image pairs and makes a prediction at the output. Since the pixel-to-pixel depth of each training image is known, it is possible to check whether this prediction is correct. According to the accuracy of this prediction, the weights of the network are updated, for example according to an error gradient back propagation algorithm. This process is repeated for all training image pairs. Once the training is completed, the model can be evaluated by presenting a test image pair to the model thus trained and by comparing the output of the model with the depth map associated with the test image pair.

[0065] The present invention also extends to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to implement the processing and computing steps of the method for training a machine learning model.

[0066] The present invention provides the following advantages:

[0067] - low cost compared to active systems for determining depth maps, such as systems using Lidar, and low cost compared to Lidar-based systems;

[0068] -Better estimation performance by using the key monocular depth metric, namely blur;

[0069] -Use depth metrics of all combinations thanks to clear images;

[0070] - Elimination of the image alignment step thanks to the acquisition system (the images are acquired simultaneously and without parallax effects);

[0071] - Acquire rather than estimate fully focused images that can be exploited in other vision tasks.

Claims

1. A navigation assistance device for embedding in a mobile system, comprising a computer vision unit (24), the computer vision unit being configured to calculate a navigation trajectory based on a first image (In) of a scene and a depth map (Dm) of the scene, characterized in that: The navigation assistance device also includes: a monocular camera capable of simultaneously acquiring the first image (In) of the scene having a first depth of field and at least one second image (If) of the scene having a second depth of field less than the first depth of field; and A depth estimation unit (23) is configured to determine the depth map (Dm) of the scene based on the first image of the scene and the at least one second image of the scene.

2. The device according to claim 1, wherein: To determine the depth map of the scene, the depth estimation unit (23) uses a machine learning model, such as a pre-trained neural network.

3. The device according to claim 2, wherein: The depth estimation unit comprises two different feature extraction branches (EXn, EXf) for respectively calculating a feature map of the first image (In) and a feature map of the at least one second image (If), and an encoder-decoder (ENC, DEC), wherein the two different feature extraction branches (EXn, EXf) are used to respectively calculate a feature map of the first image (In) and a feature map of the at least one second image (If), and the encoder-decoder takes the feature maps calculated by the two feature extraction branches as input to determine the depth map (Dm).

4. The device according to any one of claims 1 to 3, wherein: The monocular camera comprises a lens (5) having a first focal length, a lens (7, 15, N) having a second focal length greater than the first focal length, and a beam splitter (4) capable of directing input light flux to each of the lens having the first focal length and the lens having the second focal length.

5. The device according to any one of claims 1 to 4, wherein: In order to calculate the navigation trajectory, the computer vision unit (24) also uses the at least one second image of the scene acquired by the monocular camera.

6. The device according to any one of claims 1 to 5, wherein: The monocular camera is capable of simultaneously acquiring the first image of the scene and a plurality of second images of the scene, wherein the second images have focal points at different planes of the scene.

7. The device according to any one of claims 1 to 6, wherein: The first image is a sharp image at all points and the at least one second image has a deep defocus blur.

8. A navigation assistance method for a mobile system, comprising the step of calculating a navigation (NAV) trajectory based on a first image (In) of a scene and a depth map (Dm) of the scene, characterized in that: The method also includes the following steps: simultaneously acquiring (ACQ) a first image (In) of the scene having a first depth of field and at least one second image (If) of the scene having a second depth of field smaller than the first depth of field, and determining a depth map (EST-D) of the scene based on the first image (In) of the scene and the at least one second image (If) of the scene.

9. The method according to claim 8, wherein: The step of determining the depth map is performed with the aid of a machine learning model, which takes as input the first image (In) of the scene and the at least one second image (If) of the scene and provides as output the depth map of the scene.

10. The method according to claim 9, wherein: The machine learning model includes two different feature extraction branches (EXn, EXf) and an encoder-decoder (ENC, DEC), wherein the two different feature extraction branches (EXn, EXf) are used to respectively calculate the feature map of the first image (In) and the feature map of the at least one second image (If), and the encoder-decoder uses the feature maps calculated by the two feature extraction branches as input to determine the depth map (Dm).

11. A computer program product comprising instructions, which, when executed by a computer, enable the computer to implement the steps of the method for determining the depth map and calculating the navigation trajectory according to any one of claims 8 to 10.

Images