Method and device for determining a depth by a convolutional neural network associated with a self-supervised vision system.

Abstract

Method and device for determining the depth of a pixel of an image by a convolutional neural network associated with a vision system comprising a camera embedded in a vehicle. The neural network is learned in a learning phase comprising the reception (21) of data representative of a first and second image acquired by the camera at two time instants with a first definition, the prediction (23) by the neural network of depth maps associated with each image, the depth maps being determined with a second definition lower than the first definition, the generation (24) of images having a definition equal to the second definition and the generation (25) of images from the depth maps. The neural network is learned (26) by minimizing a loss error determined from errors determined by comparing the generated images. Figure for abstract: Figure 2

Description

Title of the invention: Method and device for determining a depth by a convolutional neural network associated with a self-supervised vision system. Technical field

[0001] The present invention relates to methods and devices for determining a depth by a vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring a distance separating an object from a vehicle carrying the vision system. Technological background

[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.

[0005] Vision systems comprising one or more cameras may encounter difficulties when faced with images having untextured areas, i.e. areas lacking regular patterns or distinct characteristics. Indeed, the absence of texture makes it difficult to extract features. Many algorithms associated with a vision system often rely on the detection of patterns, contours or textures to identify objects. In the absence of these visual clues, the task becomes difficult, because conventional methods may lack reference points to perform accurate recognition.

[0006] In addition, untextured images may lack significant contrast and gradients. This complicates edge detection and object segmentation in the image.

[0007] Another major challenge lies in data ambiguity. Untextured images can often be subject to multiple interpretations, as the lack of detail can make it difficult to distinguish between different classes of objects. A vision system must therefore be able to handle this uncertainty and make decisions despite the lack of detailed information.

[0008] In addition, lighting conditions can have a significant impact on the perception of untextured images. Lighting variations can create indistinct shadows and impair image quality, thus complicating the task of computer vision algorithms.

[0009] In conclusion, the main difficulty encountered by a vision system when faced with non-textured images lies in the complexity of extracting significant information in the absence of regular patterns.

[0010] The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by a vision system. Summary of the present invention

[0011] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0012] Another object of the present invention is to improve the quality of data from a vision system acquiring images presenting areas devoid of texture.

[0013] Another object of the present invention is to improve road safety, in particular by improving the operational safety of AD AS systems supplied by data obtained from at least one camera.

[0014] According to a first aspect, the present invention relates to a method for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising at least one camera arranged so as to acquire an image of a three-dimensional scene from a determined point of view, the method being implemented by at least one processor, and being characterized in that the convolutional neural network is learned in a learning phase comprising the following steps: - reception of data representative of a first and second acquired images by the camera at two distinct acquisition time instants, the first and second images having a first definition; - prediction of a movement of the camera between the two acquisition time instants from the first and second images; - prediction of a first depth map associated with the first image and a second depth map associated with the second image by the convolutional neural network from the first and second images, the first and second depth maps being determined with a second definition lower than the first definition; - generation of a third image from the first image and a fourth image from the second image by reduction of definition, the third and fourth images having a definition equal to the second definition; - generating a fifth image from the third image, the first depth map and the motion and generating a sixth image from the fourth image, the second depth map and the motion; - training the convolutional neural network by minimizing a loss error, the loss error being determined from first errors determined by comparing pixels of the sixth and third images and second errors determined by comparing pixels of the fifth and fourth images.

[0015] Such a method thus makes it possible to work on images whose definition is reduced and equal to the definition of a depth map associated with this image. The change in definition makes it possible in particular to reduce the sensitivity of the depth prediction model to non-textured areas present in images acquired by the vision system.

[0016] According to a variant of the method, the fifth and sixth images are generated using the following function: P, = tt(K[(p^k\ Dmt(p^} ) With : • Ps a pixel of a generated image, • n a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • an intrinsic matrix of the camera, • the movement of the camera between the two acquisition time instants, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, • K' an intrinsic matrix of the camera, and * P^m^P ) is a Pr°f°n(ieur of the pixel Pt determined by the neural network convolutional.

[0017] According to another variant of the method, a determination of the first and second errors comprises a determination of a photometric error determined by the following function: ^(p)~ (l"a) ' R(p) I(p) ) j With : - L(p) a first error for a pixel p of the third image, respectively a second error for a pixel p of the fourth image; - I(p) a value of the pixel p in the third image, respectively the fourth image; - a value of the pixel p in the sixth image, respectively the fifth picture ; - SSIM (from the English “structural similarity index measure”) a function which takes into account a local structure; and - has a weighting factor depending in particular on the type of environment.

[0018] According to a further variant of the method, the loss error is determined by the following function: The ^pninÇL / p), L / p)) With : • The loss error, • min a minimum value of an argument, • P a pixel associated with a position in an image, • L / / ?) a first error for a pixel p of the fourth image compared to the fifth image, • Lip) a second error for a pixel p of the third image compared to the sixth image.

[0019] According to yet another variant, the method further comprises the prediction of N pairs of depth maps, N being a strictly positive integer, each pair of depth maps comprising a depth map associated with the first image and a depth map associated with the second image, the depth maps of each pair of depth maps having the same third definition lower than the first definition, the steps of generating the third and fourth images and of generating the fifth and sixth images being repeated for each pair of depth maps and the loss error being determined from N first errors each determined by comparing the third and sixth images generated for each pair of depth maps and N second errors each determined by comparing the fourth and fifth images generated for each pair of depth maps.

[0020] According to yet another variant of the method, the loss error is determined by the following function: L' = ÿ7E„Epmin(L}(p\ L^p)) With : • The loss error, • N the number of pairs of depth maps, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image, • min a minimum value of an argument, • a first error for a pixel p of a fourth image associated with a pair n of depth maps compared to a fifth image associated with the pair n of depth maps, • L / p) a second error for a pixel p of the third image associated with the pair n of depth maps compared to the sixth image associated with the pair n of depth maps.

[0021] According to an additional variant of the method, the received data are representative of M images acquired by the camera, M being an integer strictly greater than two, each image being acquired at a distinct acquisition time instant, one of the images of the plurality of images corresponding to the first image and the other images of the plurality of images each corresponding to the second image and forming a set of second images, the steps of generating the third and fourth images and of generating the fifth and sixth images being repeated for each pair of images comprising the first image and a second image different from the set of second images,and the loss error being determined from Ml first errors each determined by comparing the third and sixth images generated for each pair of images and Ml second errors each determined by comparing the fourth and fifth images generated for each pair of images.

[0022] According to another variant of the method, the loss error is determined by the following function: L = ÂH L2(p)) With : • The loss error, • M - 1 the number of pairs of images including the first image and a second image, • m an index associated with a pair of images varying from 1 to Ml, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair m of images comprising the first image and a second image compared to a fifth image associated with the pair m of images comprising the first image and a second image, • L^p) a second error for a pixel p of the third image associated with the pair m of images comprising the first image and a second image compared to the sixth image associated with the pair m of images comprising the first image and a second image.

[0023] According to a second aspect, the present invention relates to a device for determining a depth by a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0024] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0025] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0026] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0027] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0028] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a ROM memory of the type microelectronic circuit, or even a magnetic recording medium or a hard disk.

[0029] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0030] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0031] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 3, in which:

[0032] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting example of embodiment of the present invention;

[0033] [Fig.2] illustrates a flowchart of the different stages of a learning process of a convolutional neural network associated with the on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0034] [Fig.3] schematically illustrates a device configured for the determination of a depth by a convolutional neural network associated with a vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0035] A method and a device for determining a depth by a vision system on board a vehicle will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description which follows.

[0036] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0037] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth by a depth prediction model implemented by a convolutional neural network associated with a system of On-board vision in a vehicle is for example implemented by a computer in the vehicle's on-board system controlling this vision system.

[0038] The vision system comprises at least one camera arranged to acquire an image of a scene from a defined point of view.

[0039] For this purpose, the convolutional neural network is learned in a learning phase comprising the reception of data representative of a first and second image acquired by the camera at two distinct acquisition time instants, the first and second images having a first definition and the prediction of a movement of the camera between the two acquisition time instants.

[0040] A first depth map associated with the first image and a second depth map associated with the second image are then predicted by the convolutional neural network from the first and second images, the first and second depth maps being determined with a second definition lower than the first definition.

[0041] A third image is generated from the first image and a fourth image is generated from the second image by reducing their definition, the third and fourth images having a definition equal to the second definition.

[0042] A fifth image is then generated from the third image, the first depth map and the motion and a sixth image is also generated from the fourth image, the second depth map and the motion.

[0043] The convolutional neural network is finally learned by minimizing a loss error determined from first errors determined by comparing pixels of the sixth and third images and second errors determined by comparing pixels of the fifth and fourth images.

[0044] The convolutional neural network is then learned by minimizing errors determined by comparing the generated images and the predictions of depth maps by the convolutional neural network associated with the monoscopic vision system are then improved by adjusting the parameters of this convolutional neural network.

[0045] Furthermore, in a texture-free area of the first image, a pixel of the second image corresponding to a pixel of the first image may be located anywhere in the texture-free area of the second image corresponding to the texture-free area of the first image. The prediction of a disparity, for example, is then chaotic, for example the predicted disparity is greater than the true disparity which leads to a depth prediction less than a true depth. The reduction of the image resolution thus aims to reduce the size of the texture-free areas, thereby improving the accuracy of a feature extractor (in English "edge aware smoothness loss”) in an area devoid of texture.

[0046] [Fig. 1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0047] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0048] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0049] The vehicle 10 advantageously comprises at least one on-board camera 11, configured to acquire images of a three-dimensional scene taking place in the environment of the vehicle 10 from a current observation position. The camera 11 forms a monoscopic vision system if it is used alone as illustrated in [Fig.l]. The present invention is however not limited to a monoscopic vision system comprising a single camera but extends to any vision system comprising at least one camera, for example 1, 2, 3 or 5 cameras.

[0050] The camera 11 has intrinsic parameters, in particular: - a focal length, - distortions which are due to imperfections in the optical system of the camera 11, - a direction of the optical axis of the camera 11, - an angle a of the field of vision, - an intrinsic matrix K, and - a resolution.

[0051] The intrinsic parameters characterize the transformation which associates, for an image point, subsequently called “point”, its three-dimensional coordinates in the frame of reference of the camera 11 with the pixel coordinates in an image acquired by the camera 11. These parameters do not change if the camera is moved.

[0052] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of acquisition of an image or at the time of calibration.

[0053] The camera 11 is arranged so as to acquire an image of a three-dimensional scene according to a defined point of view, the point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10 as illustrated in [Fig.l].

[0054] The camera 11 for example acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

[0055] It is obvious that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

[0056] An image acquired by the camera 11 at an acquisition time instant is presented in the form of data representing pixels characterized by: - coordinates in the image; and - data relating to the colors and brightness of the objects in the observed scene in the form, for example, of RGB (Red Green Blue) or TSL (Tone, Saturation, Brightness) colorimetric coordinates.

[0057] Each pixel of the acquired image is representative of an object in the three-dimensional scene present in the field of vision of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When the light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor of the camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details.The combination of millions of pixels creates an image faithfully representing the physical object observed by the camera 11. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene.

[0058] When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene- sional are found for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0059] Each image is for example sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0060] A method for determining a depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, that is to say by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the vision system of the vehicle 10.

[0061] A method for determining a depth by a depth prediction model implemented by a convolutional neural network associated with a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, that is to say by a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the monoscopic vision system of the vehicle 10.

[0062] Such a convolutional neural network is known to those skilled in the art, and is for example described in the following documents: - “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019, and - “HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation” by Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu and Xinxin Chen, Yi Yuan published in December 2020.

[0063] Such neural networks must be trained during a learning phase to predict accurate depths adapted to the vision system on board the vehicle 10. Such a learning phase is thus described with reference to [Fig.2],

[0064] [Fig. 2] illustrates a flowchart of the different steps of a method 2 for learning a convolutional neural network associated with a vision system embedded in a vehicle, for example with the vision system embedded in the vehicle 10 of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention.

[0065] In a step 21, data representative of a first and second image acquired by the camera 11 at two distinct acquisition time instants are received. The first and second images have the same first definition, that is to say they have the same number of horizontal and vertical lines of pixels.

[0066] The two received images correspond to two views of the same three-dimensional scene taking place around the vehicle 10 at two distinct acquisition time instants. If the vehicle 10 is moving, then the first and second images are acquired from two distinct viewpoints of the three-dimensional scene, that defined by the first position of the camera 11 at the first acquisition time instant and that defined by the second position of the camera 11 at the second acquisition time instant.

[0067] The movement between the first position and the second position is defined by the movement of the vehicle 10 between the first and second acquisition time instants. This movement of the first camera 11 is therefore linked to the speed and direction of movement of the vehicle 10 during the time separating the first and second acquisition time instants.

[0068] The camera 11, in the two positions defined at the first and second acquisition time instants, forms a monoscopic vision system. The intrinsic parameters of this system remain those defined previously linked to the camera 11. The extrinsic parameters of this monoscopic vision system are the following parameters: - 3 translations in the x, y and z directions: Tx', Ty' and Tz' constituting the translation vector T'; and - 3 rotations around the x, y and z axes: Rx', Ry' and Rz', constituting the rotation matrix R'.

[0069] In a step 22, the movement of the camera 11 between the two acquisition time instants is predicted from the first and second images. The extrinsic parameters of the monoscopic vision system are, for example, determined by a computer associated with this same monoscopic vision system. The determination of the extrinsic parameters of the monoscopic vision system is known to those skilled in the art and presented, for example, in the document “Unsupervised Learning of Depth and Ego-Motion from Video” by Tinghui Zhou, Matthew Brown, Noah Snavely and David G. Lowe published on August 1, 2017.

[0070] In a step 23, a first depth map associated with the first image and a second depth map associated with the second image are predicted by the convolutional neural network from the first and second images, the first and second depth maps being determined with a second definition lower than the first definition.

[0071] Such depth maps are known to those skilled in the art and presented for example in the previously cited documents as well as in the document “3D Packing for Self-Supervised Monocular Depth Estimation » by Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos and Adrien Gaidon published in March 2020.

[0072] The depth maps are predicted for each image acquired by the vision system, but since their definition is lower than that of the acquired images, it is not possible to assign a depth to each pixel of the acquired image.

[0073] In a step 24, a third image is generated from the first image and a fourth image is generated from the second image by reduction of definition, the third and fourth images having a definition equal to the second definition, therefore lower than the first definition.

[0074] According to a particular exemplary embodiment, the value of a pixel of an image generated by reducing the definition is defined according to a bilinear model. The value of each pixel of the generated image is the weighted average value of the i closest pixels, for example with i = 4, respectively in the horizontal direction and the vertical direction; the weighting coefficient of each value is a function of the inverse of the distance of the pixels neighboring the pixel of the generated image.

[0075] In a step 25, a fifth image is generated from the third image, the first depth map and the motion and a sixth image is generated from the fourth image, the second depth map and the motion.

[0076] According to a particular exemplary embodiment, the fifth and sixth images are generated using the following function: [Math.l] P s= 77 ( K [( P^' D^t ( PJ ) ] )

[0077] With: • Px a pixel of a generated image, the fifth image, respectively the sixth image, •77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K an intrinsic matrix of the camera 11, • Tr-*s the movement of the camera 11 between the two acquisition time instants, • 0 a projection function in the three-dimensional scene of a pixel of the third image, respectively the fourth image, as a function of its depth obtained from the first depth map, respectively the second depth map, and * Dmt(p ) is a P°f°n(ieur of the pixel Pt determined by the convolutional neural network in the first depth map, respectively in the second depth map.

[0078] The principle of image generation consists of projecting a pixel of an image generated by reducing the definition of an image acquired by the first camera 11, the third image (generated from the first image), respectively the fourth image (generated from the second image), towards a point in the three-dimensional scene according to its depth obtained from the first depth map, respectively second depth map, then reprojecting this point into an image as it would be acquired from the other point of view of the first camera 11 before or after its movement. Thus, if the depth value associated with a pixel of an image is correct, then the point of the scene is well defined and the other image should have, at the location of this projected point of the scene, a pixel of the same value.

[0079] In a step 26, the convolutional neural network is learned by minimizing a loss error, the loss error being determined from first errors determined by comparing pixels of the sixth and third images and second errors determined by comparing pixels of the fifth and fourth images.

[0080] According to a first particular exemplary embodiment, the first and second errors comprise a determination of a photometric error determined by the following function:

[0081] [Math.2] L(p)= (l-«) ■ \I(p)-î(p)\+a\l-±SSIM{l(p),î(p)))

[0082] With: - L{p) a first error for a pixel p of the third image, respectively a second error for a pixel p of the fourth image; - I(p) a value of the pixel p in the third image, respectively the fourth image; - a value of the pixel p in the sixth image, respectively the fifth picture ; - SSIM (from the English “structural similarity index measure”) a function which takes into account a local structure; and - has a weighting factor depending in particular on the type of environment.

[0083] This function is based on the photometric error. Once an image is generated, it is compared to the image generated by reducing the definition of the image acquired by the monoscopic vision system.

[0084] According to a second particular exemplary embodiment, the first and second errors comprise a determination of a reconstruction error of a generated image further determined by the following function: [Math.3] ^mOoth(D^ o) = LP£d&Jw(Pl)

[0085] With: • Lmworth (O, W, o) a first error for a pixel p of the third image, respectively a second error for a pixel p of the fourth image, • Dt ( p J is a depth obtained from the first depth map, respectively obtained from the second depth map; • W is a parameter matrix; • ° is the order of a smoothing gradient; • an L1 norm of second-order depth gradients is calculated with W =1, and ° =2; • A and are the dimensions of the third image, respectively fourth image; • P is an environment-dependent hyperparameter; and * P ) is the value of pixel Pt in the sixth image, respectively fifth image.

[0086] This second function is generally used to deal with discontinuity at the edge of objects (in English “edge aware smoothness”).

[0087] The reconstruction error is thus defined, for example, from the photometric errors and reconstruction errors previously defined.

[0088] According to a particular exemplary embodiment, the loss error is determined by the following function:

[0089] [Math.4] L' = L / p))

[0090] With: • The loss error, • min a minimum value of an argument, • a first error for a pixel p of the fourth image compared to the fifth image, • Lo(p) a second error for a pixel p of the third image compared to the sixth image.

[0091] The use of the minimum value of the errors thus makes it possible not to be sensitive to occlusion. Indeed, a pixel associated with an object in the three-dimensional scene appears in at least one of the two acquired images and at different positions in the two images.

[0092] According to a first particular embodiment, the method further comprises a prediction of N pairs of depth maps, N being a strictly positive integer. Each pair of depth maps includes a depth map associated with the first image and a depth map associated with the second image, the depth maps of each pair of depth maps having the same third definition lower than the first definition. the steps 24 of generating the third and fourth images and 25 of generating the fifth and sixth images are then repeated in this first particular embodiment for each pair of depth maps and the loss error is determined from: - First N errors each determined by comparing the third and sixth images generated for each pair of depth maps, and - N second errors each determined by comparing the fourth and fifth images generated for each pair of depth maps. The number N is, for example, equal to 4.

[0093] According to a particular exemplary embodiment, in this first particular embodiment, the loss error is determined by the following function:

[0094] [Math.5] L' = L2(p) )

[0095] With: • The loss error, • N the number of pairs of depth maps, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair 11 of depth maps compared to a fifth image associated with the pair n of depth maps, • L / p) a second error for a pixel p of the third image associated with the pair n of depth maps compared to the sixth image associated with the pair n of depth maps.

[0096] According to a second particular embodiment, the received data are representative of M images acquired by the camera 11, M being an integer strictly greater than two. Each image being acquired at a distinct acquisition time instant, one of the images of the plurality of images corresponds to the first image and the other images of the plurality of images each correspond to the second image and form a set of second images. In other words, the plurality of images comprises the first image and other images acquired at other time instants, prior and / or subsequent to the time instant of acquisition of the first image. The other images thus each correspond to a second image. For example, the data are representative of a first image acquired at a first instant acquisition time, of two images acquired at two acquisition time instants prior to the first acquisition time instant and of two images acquired at two acquisition time instants subsequent to the first acquisition time instant.

[0097] In this second particular embodiment, the steps 24 of generating the third and fourth images and 25 of generating the fifth and sixth images are repeated for each pair of images comprising the first image and a second image different from the set of second images, and the loss error is determined from: - M-1 first errors each determined by comparing the third and sixth images generated for each pair of images, and - Ml second errors each determined by comparing the fourth and fifth images generated for each pair of images

[0098] According to a particular exemplary embodiment, in this second particular embodiment, the loss error is determined by the following function:

[0099] [Math.6] vMa»)

[0100] With: • The loss error, • M - 1 the number of pairs of images including the first image and a second image, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair m of images comprising the first image and a second image compared to a fifth image associated with the pair m of images comprising the first image and a second image, • a second error for a pixel p of the third image associated with the pair m of images comprising the first image and a second image compared to the sixth image associated with the pair m of images comprising the first image and a second image.

[0101] The first and second particular embodiments are also combinable. Thus, a training data set comprises several images making it possible to obtain several pairs of images always comprising the same first image and several depth maps determined for each pair of images. The loss error is then, according to a particular exemplary embodiment, determined by the following function:

[0102] [Math.6] L = Mp) )

[0103] With: • The loss error, • M - 1 the number of pairs of images including the first image and a second image, • m an index associated with a pair of images varying from 1 to Ml, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair m of images comprising the first image and a second image compared to a fifth image associated with the pair m of images comprising the first image and a second image, • L^p) a second error for a pixel p of the third image associated with the pair m of images comprising the first image and a second image compared to the sixth image associated with the pair m of images comprising the first image and a second image.

[0104] It should be noted that this function corresponds to an average of the different errors determined. According to other variants, the loss error corresponds to a sum of the errors or to a combination of sums and averages. For example, the loss error corresponds to a sum and is then obtained by the following function:

[0105] [Math.7] L = £,XX / nin (L^p), L2(p))

[0106] With: • The loss error, • M - 1 the number of pairs of images including the first image and a second image, • m an index associated with a pair of images varying from 1 to Ml, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair m of images comprising the first image and a second image compared to a fifth image associated with the pair m of images comprising the first image and a second image, • L^p) a second error for a pixel p of the third image associated with the pair m of images comprising the first image and a second image compared to the sixth image associated with the pair m of images comprising the first image and a second image.

[0107] This learning also makes it possible to find or refine the correct values of the parameters (called “weights” in English) of a chosen convolutional neural network in order to make the depths predicted by this convolutional neural network more reliable. The convolutional neural network associated with the vision system on board the vehicle 10 thus predicts reliable depths in line with images acquired by the camera 11 of the vision system.

[0108] If the ADAS uses a depth predicted by this convolutional neural network as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases sharply, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident, including if the other user is seen by only one camera.

[0109] [Fig. 3] schematically illustrates a device 3 configured for the determination of a depth by a convolutional neural network associated with a vision system on board a vehicle, for example in the vehicle 10 of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention. The device 3 corresponds for example to a device on board the first vehicle 10, for example a computer.

[0110] The device 3 is for example configured for the implementation of the steps described with regard to [Fig.l] or [Fig.2]. Examples of such a device 3 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 3, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 3 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0111] The device 3 comprises one (or more) processor(s) 30 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 3. The processor 30 can include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 3 further comprises at least one memory 31 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0112] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the memory 31.

[0113] According to various particular and non-limiting embodiments, the device 3 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0114] According to a particular and non-limiting exemplary embodiment, the device 3 comprises a block 32 of interface elements for communicating with external devices. The interface elements of the block 32 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (LTE) “Long-Term Evolution” or in French “Long-Term Evolution”), LTE-Advanced (or in French LTE-advanced); - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0115] According to another particular and non-limiting exemplary embodiment, the device 3 comprises a communication interface 33 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 330. The communication interface 33 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 330. The communication interface 33 corresponds for example to a wired network of the CAN type (from the English “Controller Area Network (or in French “Controller Area Network”), CAN FD (from the English “Controller Area Network Flexible Data-Rate”), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3).

[0116] According to a particular and non-limiting exemplary embodiment, the device 3 can provide output signals to one or more external devices, such as a display screen 340, touch-sensitive or not, one or more speakers 350 and / or other peripherals 360 (projection system) via the output interfaces 34, 35, 36 respectively. According to a variant, one or other of the external devices is integrated into the device 3.

[0117] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining a depth and measuring such a quantity by a vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0118] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 3 of [Fig.3].

Claims

Claims

1. Method for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising at least one camera (11) arranged so as to acquire an image of a three-dimensional scene from a determined point of view, said method being implemented by at least one processor, and being characterized in that the convolutional neural network is learned in a learning phase comprising the following steps: - reception (21) of data representative of a first and second image acquired by the camera (11) at two distinct acquisition time instants, the first and second images having a first definition; - prediction (22) of a movement of the camera (11) between the two acquisition time instants from the first and second images; - prediction (23) of a first depth map associated with the first image and of a second depth map associated with the second image by said convolutional neural network from the first and second images, said first and second depth maps being determined with a second definition lower than the first definition; - generation (24) of a third image from the first image and of a fourth image from the second image by reduction of definition, the third and fourth images having a definition equal to the second definition; - generation (25) of a fifth image from the third image, the first depth map and the movement of the camera (11) and generation of a sixth image from the fourth image, the second depth map and the movement; - training (26) the convolutional neural network by minimizing a loss error, the loss error being determined from first errors determined by comparing pixels of the sixth and third images and second errors determined by comparing pixels of the fifth and fourth images.

2. A method according to claim 1, wherein the fifth and sixth images are generated using the following function: P s = ( P^, Dmt ( PJ ) ] ) With : • Ps a pixel of a generated image, • 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K an intrinsic matrix of the camera (11), • The movement of the camera (11) between the two acquisition time instants, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and • Dmt(p ) is a depth of the pixel P( determined by the convolutional neural network.

3. A method according to claim 1 or 2, wherein a determination of the first and second errors comprises a determination of a photometric error determined by the following function: (1-a) ■ \I(p) -I(p)\+a- With : - L(p) a first error for a pixel p of the third image, respectively a second error for a pixel p of the fourth image; - I(p) a value of the pixel p in the third image, respectively the fourth image; ' l(p) a value of pixel p in the sixth image, respectively the fifth image; - SSIM (from the English “structural similarity index measure”) a function which takes into account a local structure; and - G a weighting factor depending in particular on the type of environment.

4. Method according to one of claims 1 to 3, for which the loss error is determined by the following function: L ' = £pmin ( L^p), L^p) ) With : • The loss error, • P a pixel associated with a position in an image, • min a minimum value of an argument, • L^p) a first error for a pixel p of the fourth image compared to the fifth image, • L^p) a second error for a pixel p of the third image compared to the sixth image.

5. Method according to one of claims 1 to 4, further comprising a prediction of N pairs of depth maps, N being a strictly positive integer, each pair of depth maps comprising a depth map associated with the first image and a depth map associated with the second image, the depth maps of each pair of depth maps having the same third definition lower than the first definition, the steps of generating (24) the third and fourth images and of generating (25) the fifth and sixth images being repeated for each pair of depth maps and the loss error being determined from N first errors each determined by comparing the third and sixth images generated for each pair of depth maps and N second errors each determined by comparing the fourth and fifth images generated for each pair of depth maps.

6. Method according to claim 5, for which the loss error is determined by the following function: = iE„Eprnin(LL / p)) With: • L1 the loss error, • N the number of pairs of depth maps, • min a minimum value of an argument, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image,* L^p) a first error for a pixel p of a fourth image associated with a pair n of depth maps compared to a fifth image associated with the pair fl of depth maps, • ^2( / 2) a second error for a pixel p of the third image associated with the pair n of depth maps compared to the sixth image associated with the pair 11 of depth maps.

7. Method according to claim 6, for which the received data are representative of M images acquired by the camera (11), M being a integer strictly greater than two, each image being acquired at a distinct acquisition time instant, one of the images of the plurality of images corresponding to the first image and the other images of the plurality of images each corresponding to the second image and forming a set of second images, the steps of generating (24) the third and fourth images and of generating (25) the fifth and sixth images being repeated for each pair of images comprising the first image and a second image different from said set of second images, and the loss error being determined from Ml first errors each determined by comparing the third and sixth images generated for each pair of images and Ml second errors each determined by comparing the fourth and fifth images generated for each pair of images.

8. The method of claim 7, wherein the loss error is terminated by the following function: L = ÂH ^XXUX^C^)' L 2(P)) With : • The loss error, • M -1 the number of pairs of images including the first image and a second image, • m an index associated with a pair of images varying from 1 to Ml, • n an index associated with a pair of depths varying from 1 to N, • P a pixel associated with a position in an image, • min a minimum value of an argument, • L^p) a first error for a pixel p of a fourth image associated with a pair m of images comprising the first image and a second image compared to a fifth image associated with the pair m of images comprising the first image and a second image, • L / p) a second error for a pixel p of the third image associated with the pair m of images comprising the first image and a second image compared to the sixth image associated with the pair m of images comprising the first image and a second image.

9. Device (3) for determining a depth by a vision system on board a vehicle (10), said device (3) comprising a memory (31) associated with at least one processor (30) configured for implementing the steps of the method according to any one of the claims 1 to 8.

10. Vehicle (10) comprising the device (3) according to claim 9

Images