Method and device for determining depth using a monoscopic vision system with metric accuracy

Abstract

A method or device implementing a method for determining depth using a vision system mounted in a vehicle (10), the vision system comprising at least two cameras (11, 12) arranged so as to each acquire an image of a scene from a different viewpoint. The method includes the steps of predicting first depths using a stereoscopic vision system, second and third depths using a monoscopic vision system, as well as generating images from the second and third depths and training the monoscopic vision system based on an error determined by comparing the generated images with the acquired images. The method then includes adjusting the parameters of a neural network associated with the monoscopic vision system based on reconstruction errors and depths, thereby determining a scaling factor for the monoscopic vision system.Figure for the abbreviation: Figure 1.

Description

Title of the invention: Method and device for determining depth using a monoscopic vision system with metric accuracy technical field

[0001] The present invention relates to methods and devices for determining depth using a monoscopic vision system mounted in a vehicle, for example, in a motor vehicle. The present invention also relates to a method and device for measuring such depth. The present invention further relates to a method and device for controlling one or more ADAS systems mounted in a vehicle based on the determined depth. Technological background

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] The ADAS systems installed in a vehicle are powered by data obtained from one or more onboard sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or any obstacles present around a vehicle in order to, for example: - to adapt the vehicle's lighting according to the presence of other road users; - to automatically regulate the vehicle's speed; - to act on the braking system in case of risk of impact with an object.

[0005] The quality of the data emitted by a vision system therefore determines the proper functioning of the driving assistance devices using this data. Summary of the present invention

[0006] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0007] Another object of the present invention is to improve the accuracy of depths or distances predicted by a monoscopic vision system.

[0008] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from at least one camera.

[0009] According to a first aspect, the present invention relates to a method for determining depth using a monoscopic vision system mounted in a vehicle, the vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the method being characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same first temporal instant of acquisition; - prediction of first depths associated with a first set of pixels from the first image as a function of optical flux values ​​determined from the first set of pixels and a second set of pixels from the second image corresponding to the first set of pixels; - reception of third data representative of a third image acquired by the first camera at a second time instant of acquisition different from the first time instant of acquisition; - determination of data representative of a movement of the first camera between the first temporal instant of acquisition and the second temporal instant of acquisition as a function of the first and third images; - prediction of second depths associated with a third set of pixels from the first image and of third depths associated with a fourth set of pixels from the third image, the fourth set of pixels corresponding to the third set of pixels, as a function of optical flux values ​​determined from the third and fourth sets of pixels, the second and third depths being predicted by a convolutional neural network associated with a monoscopic vision system including the first camera 11; - generation of a fourth image from the first image and the second depths and of a fifth image from the third image and the third depths by applying an image generation function; - learning of the convolutional neural network by minimizing a third error determined as a function of a first reconstruction error determined by comparing the fifth and first images and a second reconstruction error construction determined by comparison of the fourth and third images; - determination of a scaling factor of the monoscopic vision system by adjusting parameters of the convolutional neural network as a function of the first and second reconstruction errors and the first and second depths, the adjustment being obtained by minimizing an overall error determined by the following first loss function: LT!al = ^nin(Lmlp), L^p)) + ~Dst(p) | + - l|With: * a global error, • L^p) the first reconstruction error for a pixel P of the first image, • Lms(p) the second reconstruction error for a pixel P of the third image, * ( P ) a second depth of a pixel p of the first image, • D^^p) a first depth of a pixel p of the first image, and • P a factor depending on the number of iterations carried out during the second learning.

[0010] Such a method makes it possible to predict second depths for the monoscopic vision system more reliable by self-learning, that is to say by learning from data determined by the system itself, via the minimization of the third error.

[0011] Such a method also makes it possible to determine a scale factor for the monoscopic vision system by comparing depths predicted by this monoscopic vision system with depths predicted by a stereoscopic vision system that is metrically accurate. The prediction of second depths by the monoscopic vision system then achieves metric accuracy thanks to the definition of the scale factor for pixels associated with objects present in the common field of view of the first and second cameras.

[0012] A driving assistance system using second depths as input data then has absolute data regarding the distance separating the vision system from the objects present in the field of vision of one of the cameras of this vision system.

[0013] According to a variant of the method, the third error is determined by a first loss function defined by: Lm = E / ?avg(Lmi(p), Lms(p))^c: • The third mistake, • avS an average of arguments, • Lmt(p) the first reconstruction error for a pixel P of the first image, and • Lms(p) the second reconstruction error for a pixel P of the third image.

[0014] According to another variant of the method, the third error is determined by a second loss function defined by: Lm = L / ninLm^p))^c : • r ' the third error, • min a minimum value of an argument, • the first reconstruction error for a pixel P of the first image, and • Lm^p) the second reconstruction error for a pixel P of the third image.

[0015] According to a further variant of the method, the image generation function is defined by the following function for the monoscopic vision system: Ps = Æ(^[^{p^ Dmt(pt) ) ] ) With : • Ps a pixel of a generated image, •77 a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K an intrinsic matrix of the first camera (11), • T a matrix representing the movement of the first camera (11) between two acquisition times, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and * Dmkp) is a pixel depth Pf predicted by the convolutional neural network.

[0016] According to yet another variant of the method, the first and second reconstruction errors are further determined by the following function: L m (p) = E p [(l-«) ■ \I(p)-I(p)\+a- (14SSZW( / (p), / (p)))] With : • Lm(p) the first reconstruction error, respectively the second reconstruction error, • I(p) a value of pixel P in the first image, respectively third image, • a value of pixel P in the fifth image, respectively fourth picture' • SSIM is a function that takes into account a local structure, and • H is a weighting factor that depends in particular on the type of environment.

[0017] According to a further variant of the method, the first and second reconstruction errors are determined, in addition, by the following function: With : • L th{ O, W, o) the first reconstruction error, respectively the second reconstruction error • Dt p} is the second predicted depth Dt of a pixel Pt, respectively the third predicted depth Dt of a pixel Pt, • W is a parameter matrix, • 0 is the order of a smoothing gradient, • an L1 norm of the second-order depth gradients is calculated with W = 1, et0 = 2, • x and y are the dimensions of the first image, respectively the third image, • P is an environment-dependent hyperparameter, and • / ( nj is the value of the pixel Pt in the fifth image, respectively the fourth image.

[0018] According to yet another variant, the method includes a step of determining disparity values ​​induced by a relative position of optical axes of the first camera at the first and second time instants of acquisition, the predictions of second and third depths being, moreover, a function of the induced disparity.

[0019] According to a further embodiment, the method includes a step of rectifying the first and third images according to extrinsic parameters of the monoscopic vision system

[0020] According to a second aspect, the present invention relates to a device for determining depth by a monoscopic vision system mounted in a vehicle, the device comprising a memory associated with at least one processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0021] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0022] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0023] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0024] According to a fifth aspect, the present invention relates to a support computer-readable recording on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0025] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0026] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0027] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0028] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 3, in which:

[0029] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0030] [Fig.2] illustrates a flowchart of the different stages of a process for determining a depth by a monoscopic vision system mounted in the vehicle of the [Fig.1], according to a particular and non-limiting example of the present invention;

[0031] [Fig. 3] schematically illustrates a device configured for determining depth using a vision system mounted in the vehicle of [Fig. 1], according to a particular and non-limiting embodiment of the present invention. Description of embodiment examples

[0032] A method and device for determining depth by means of a monoscopic vision system mounted in a vehicle will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

[0033] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to allow identification and to distinguish different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0034] According to a particular and non-limiting example of an embodiment of the present invention, a method for determining depth by means of a monoscopic vision system embedded in a vehicle is for example implemented by a computer of the vehicle's embedded system controlling this vision system.

[0035] The vision system comprises a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view.

[0036] To this end, the method of determining a depth by a monoscopic vision system mounted in a vehicle includes the steps of receiving first, second and third data respectively representative of first, second and third images.

[0037] The first and second images are acquired respectively by the first and second cameras at the same initial acquisition time instant. The first and second cameras then form a stereoscopic vision system.

[0038] The third image is acquired by the first camera at a second acquisition time that differs from the first acquisition time. The first camera then forms a monoscopic vision system, the first camera having different viewpoints of the three-dimensional scene depending on the movement of the vehicle 10.

[0039] According to a first particular embodiment, the second temporal instant of acquisition is prior to the first temporal instant of acquisition.

[0040] According to a second particular embodiment, the second temporal instant of acquisition is subsequent to the first temporal instant of acquisition.

[0041] Initial depths are predicted for pixels of the first image for the stereoscopic vision system from the first and second images.

[0042] Second and third depths are predicted by a convolutional neural network associated with the monoscopic vision system for pixels of the first image and the third image, respectively. A fourth image is then generated from the first image and the second depths by applying a similar image generation function, and a fifth image is generated by this same image generation function from the third image and the third depths. The convolutional neural network is then learned based on a third error determined from a first reconstruction error determined by comparing the fifth image to the first image and from a second reconstruction error determined by comparing the fourth image to the third. picture.

[0043] A scaling factor of the monoscopic vision system is then determined by an adjustment of parameters of the convolutional neural network as a function of the first and second reconstruction errors and the first and second depths.

[0044] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0045] Such an environment 1 corresponds, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0046] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say, a vehicle operating according to a predetermined level of autonomy or under the total supervision of the driver.

[0047] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereoscopic vision system. Two cameras 11 and 12 are illustrated in [Fig. 1]. The present invention is not limited, however, to a stereoscopic vision system comprising two cameras but extends to any vision system comprising two or more cameras, for example, two, three, four, or five cameras.

[0048] The two cameras 11, 12 have known intrinsic parameters. These parameters include, in particular: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the C2 direction of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.

[0049] The intrinsic parameters characterize the transformation that associates, for an image point, the camera coordinates to the pixel coordinates, in each camera. These parameters do not change if the camera is moved.

[0050] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely the dis Radial, decentering, and prismatic distortions are induced by defects in curvature, lens parallelism, and optical axis coaxiality. In this example, the cameras are assumed to be perfect, meaning that distortions are either not considered or their correction is addressed during image acquisition.

[0051] These two cameras 11, 12 are arranged so that each acquires an image of a scene from a different viewpoint. The first viewpoint is, for example, located on or in the left-hand rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. The second viewpoint is, for example, located on or in the right-hand rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. If both cameras are located at the top of the windshield of the vehicle, they are then positioned at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, and the second camera 12 is located in the right-hand rearview mirror of the vehicle 10.

[0052] A first marker is associated with the first camera 11: - the direction of the y-axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y-axis of the first camera 11. The distance B separating the two cameras 11,12 is called the reference base (in English "baseline") and the direction separating the two cameras 11,12 is that of the y-axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - The direction of the z-axis is defined as orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal coordinate system.

[0053] The extrinsic parameters related to the position of cameras 11, 12 are the following parameters: - 3 translations in the x, y, and z directions: Tx, Ty, and Tz forming the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.

[0054] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.

[0055] A key constraint of stereoscopic vision systems used in automobiles is, for example, the large distance between the two cameras. Indeed, to cover a measurement range of 200 meters, the reference base must be 60 cm for cameras commonly used in this field.

[0056] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 alone covering a first acquisition field 13, the second camera 12 alone covering a second acquisition field 14 and both cameras 11, 12 together covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0057] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0058] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system composed of the second camera 12.

[0059] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable with the monoscopic vision system composed of the first camera 11.

[0060] Finally, field 19 is not visible from any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0061] According to one embodiment, the directions Cl, C2 of the optical axes representing an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0062] According to another embodiment, the directions Cl, C2 of the optical axes representing an orientation of the field of vision of each camera are oriented parallel.

[0063] It is evident that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently.

[0064] The images acquired by cameras 11, 12 at a given acquisition time t1 are in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene, for example in the form of RGB (Red Green Blue) or HSL (Hint, Saturation, Luminosity) colorimetric coordinates.

[0065] The images acquired by cameras 11 and 12 represent views of the same scene taken from different viewpoints, the camera positions being distinct. For example, this scene includes: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0066] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0067] A process for determining depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, i.e. by a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer or computers in charge of the stereoscopic vision system of the vehicle 10.

[0068] In a first operation, the computer receives first data representative of a first image acquired by a first camera 11 of the set of cameras at a first time instant of acquisition.

[0069] In a second operation, the computer receives second data representing a second image acquired by a second camera 12 of the set of cameras at the same first time instant of acquisition.

[0070] The two images received correspond to two views of the same three-dimensional scene taking place around the vehicle 10 at the same first given time instant of acquisition.

[0071] In a third operation, first depths associated with a first set of pixels of the first image are predicted as a function of optical flux values ​​determined from the first set of pixels and a second set of pixels of the second image corresponding to the first set of pixels.

[0072] Since the fields of vision of the first 11 and the second 12 camera are concurrent, a part of the pixels of the first image and a part of the pixels of the second image are associated with the same objects present in the three-dimensional scene.

[0073] The third operation is carried out by implementing a method called optical flow calculation. Such a method is notably described in "PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume" by Deqing Sun, Xiaodong Yang, Ming-Yu Liu and Jan Kautz of June 25, 2018 and in "RAFT: Recurrent Ail-Pairs Field Transforms for Optical Flow" by Zachary Teed and Jia Deng of August 25, 2020.

[0074] It should be noted that the initial depths are accurate to the meter. Indeed, the depth of a pixel is determined based on the optical flow and the reference frame. The reference frame is the distance between the first 11 and second 12 cameras. It is known to the stereoscopic vision system via a calibration for example, and expressed in centimeters for example. A first depth of a pixel being proportional to the reference base and inversely proportional to a norm of the optical flux, this first depth is metrically precise.

[0075] In a fourth operation, third data representing a third image acquired by the first camera 11 at a second time instant of acquisition prior to the first time instant of acquisition of the first image are received.

[0076] The third data has, for example, been saved in a memory associated with the computer or in a memory of a device embedded in the vehicle 10 and accessible to the computer implementing the process.

[0077] If the vehicle 10 is in motion, then the third image corresponds to a third view of the scene taken from a third point of view, that of the first camera 11 at its position at the second time instant of acquisition.

[0078] The position of the first camera 11 at the second acquisition time instant is defined by the movement of the vehicle 10 between the first and second acquisition times. This movement of the first camera 11 is therefore linked to the speed and direction of movement of the vehicle 10 during the time interval between the first and second acquisition times.

[0079] The first camera 11, in the two positions defined at the first and second acquisition times, forms a monoscopic vision system. The intrinsic parameters of this system remain the same as those previously defined for the first camera 11 for the stereoscopic vision system. The extrinsic parameters of this monoscopic vision system are as follows: - 3 translations in the x, y and z directions: Tx', Ty' and Tz' constituting the translation vector T'; and - 3 rotations around the x, y and z axes: Rx', Ry' and Rz', constituting the rotation matrix R'.

[0080] In a fifth operation, data representing the movement of the first camera 11 between the first and second acquisition times are determined based on the first and third images. The extrinsic parameters of the monoscopic vision system are, for example, determined by a computer associated with this same monoscopic vision system. The determination of the extrinsic parameters of the monoscopic vision system is known to those skilled in the art and is presented, for example, in the document "Unsupervised Leaming of Depth and Ego-Motion from Video" by Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe, published on August 1, 2017.

[0081] In a sixth operation, second depths associated with a third The third pixel set of the first image is predicted based on optical flux values ​​determined from the third pixel set and a fourth pixel set from the third image corresponding to the third pixel set. Similarly, in a seventh operation, third depths associated with the fourth pixel set of the third image are predicted based on optical flux values ​​determined from the fourth pixel set and the third pixel set of the third image.

[0082] According to a particular embodiment, in order to facilitate the analysis of the received images, the first and third images are rectified in an eighth operation, following a method known to those skilled in the art. Such a method is described, for example, in "Projective Rectification of Uncalibrated Infrared Stereo Images with Global Consideration of Distortion Minimization" by Benoit Ducarouge, Thierry Sentenac, Florian Bugarin and Michel Devy, dated July 16, 2009.

[0083] The rectification method consists of reorienting the epipolar lines so that they are parallel with the horizontal axis of the image. This method is described by a transformation that projects the epipoles to infinity and whose corresponding points are necessarily on the same ordinate.

[0084] A rectification algorithm consists, for example, of 4 steps: - Rotate (virtually) the first camera 11 so that the epipole goes to infinity along the horizontal axis of the frame associated with it; - Apply the same rotation to the second camera to return to the initial geometric configuration; - Rotate the second camera by the rotation associated with the rotation matrix 'R', corresponding to the extrinsic parameter of the starting stereoscopic vision system; - Adjust the scale in both camera reference points.

[0085] According to another embodiment, the first and third images are not rectified. The rotation and translation to go from the optical axis of the first camera 11 at the first time instant to that of the first camera 11 at the second time instant creates in particular an induced disparity which is, moreover, included in the optical flow determined from the first and third images.

[0086] According to this particular embodiment, in order not to take into account this induced disparity which may distort the prediction of second and third depths, a second fully convolutional neural network of type FCN (from the English "Fully Convolutional network"), hereafter called an FCN, is implemented to determine the induced disparity and filter the optical flux value before predicting the depth of a pixel.

[0087] The input data of such an FCN is a one-dimensional matrix HxWxN with H the The height of an image, W the width of an image, and N the number of channels including the unaffected elements of the rotation matrix (maximum 9), the unaffected elements of the translation vector (maximum 3), and the two optical fluxes (horizontal and vertical). The maximum number of channels is therefore 14. The number of intermediate layers and the number of channels in the matrix at each layer depend on the deviation of the parallel system.

[0088] The structure of this FCN is constructed experimentally in such a way as to minimize the size of the FCN while offering an acceptable result.

[0089] The structure of this FCN is constructed experimentally in such a way as to minimize the size of the FCN while offering an acceptable result.

[0090] Thus, according to this particular embodiment, in a ninth operation, an induced disparity is determined by the FCN from the first and third images allowing the induced disparity to be subtracted from the optical flux to obtain a corrected optical flux, the second and third depths are then predicted as a function of this corrected optical flux, a second and / or third depth being proportional to a norm of an optical flux from which the induced disparity is subtracted.

[0091] In a tenth operation, a fourth image is generated from the first image and the second depths by applying an image generation function.

[0092] Similarly, in an eleventh operation, a fifth image is generated from the third image and the third depths by applying this same image generation function.

[0093] According to a particular embodiment, the image generation function is defined by the following function for the monoscopic vision system: [Math.l] Ps “ TT ( K [ ( p^\ Dmt ( pJ ) ] )

[0094] With: • Ps one pixel of the fourth image, respectively fifth image, • 77 a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K an intrinsic matrix of the first camera 11, • M is a matrix representing the movement of the first camera 11 between the first and second time instants, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and • Dml(p ) is a second depth of a pixel Pt of the first image, respectively third depth of a pixel Pt of the third image.

[0095] The principle of generating the fourth image consists of projecting a pixel from the first image acquired by the first camera 11 at the first acquisition time instant onto a point in the three-dimensional scene according to its second depth, and then reprojecting this point into an image as it would be acquired from the viewpoint of the first camera 11 from its position at the second acquisition time instant. Thus, if the second depth value associated with a pixel of the first image is correct, then the point in the scene is well-defined and the third image has, at the location of this projected point in the scene, a pixel of the same value.

[0096] In a twelfth operation, the convolutional neural network associated with the monoscopic vision system comprising the first camera 11 is learned by minimizing a third error determined as a function of a first reconstruction error determined by comparison of the fifth and first images and a second reconstruction error determined by comparison of the fourth and third images.

[0097] According to the first particular embodiment, the first reconstruction error and respectively the second reconstruction error are determined, furthermore, by the following function: [Math.2] Lm(p) = -?O)I+«■ (l-iSSIM{l(p)Mp))}]

[0098] With: • Lm(p) the first reconstruction error, respectively the second reconstruction error, • I(p) a value of pixel P in the first image, respectively third image, • a value of pixel P in the fifth image, respectively fourth picture' • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0099] This first function is based on photometric error. Once an image is generated, it is compared to the true image acquired by the monoscopic vision system.

[0100] According to a second particular embodiment, the first reconstruction error and respectively the second reconstruction error are further determined by the following function: [Math.3]

[0101] With: • L th{ O, W, o) the first reconstruction error, respectively the second reconstruction error • Dt p} is the second predicted depth Dt of a pixel Pt, respectively the third predicted depth Dt of a pixel Pt, • W is a parameter matrix, • 0 is the order of a smoothing gradient, • an L1 norm of the second-order depth gradients is calculated with W = 1, et0 = 2, • x and y are the dimensions of the first image, respectively the third image, • P is an environment-dependent hyperparameter, and • / ( nj is the value of the pixel Pt in the fifth image, respectively the fourth image.

[0102] This function is generally used to deal with the discontinuity at the edge of objects (in English, "edge aware smoothness").

[0103] According to a third particular embodiment, the first reconstruction error and respectively second reconstruction error are determined by a function combining the function presented in the first particular embodiment and by the function presented in the second particular embodiment.

[0104] The third error is then determined from these first and second reconstruction errors.

[0105] According to a particular embodiment, the third error is determined by a second loss function defined by: [Math.4] Lm = Lms(p)}

[0106] With: • J, the third mistake, • avS an average of arguments, • Lmt(p) is the first reconstruction error for a pixel P in the first image, and • Lms(.p) the second reconstruction error for a pixel P of the third image.

[0107] By calculating the average of the two reconstruction errors, the two predictions are treated equally, which improves the prediction reliability when one of the predictions is better in one direction than the other (from the first image to the third image or vice versa), that is, when the reliability of the second and third depths is not equivalent.

[0108] According to another particular embodiment, the third error is determined by a third loss function defined by: [Math.5] ( ^-'mt^P)' ^mÂP) )

[0109] With: • / ' the third mistake, • min a minimum value of an argument, • L^p} the first reconstruction error for a pixel P of the first image, and • the second reconstruction error for a pixel P of the third image.

[0110] This loss function is faster and allows the number of iterations in the learning operation to be reduced.

[0111] Thus, the operations enabling the prediction of the second and third depths are implemented by a self-supervised convolutional neural network, that is to say, this network is learned using data that it generates. Such a convolutional neural network is known to those skilled in the art as PWCNet (from the English "Pyramidal Processing, Warping, and the use of a Cost volume Network").

[0112] This learning also makes it possible to find or refine the correct values ​​of the parameters (called "weights" in English) of a chosen convolutional neural network in order to make the second predicted depths more reliable.

[0113] In a thirteenth operation, a scaling factor of the monoscopic vision system is determined by fitting parameters of the convolutional neural network as a function of the first and second reconstruction errors and the first and second depths. The fitting is then obtained by minimizing an overall error determined by the following first loss function: [Math.6] = ï^nin (L,M, Lms(p)) + | Dmt (p) - Dst (p) j + | - 11

[0114] With: * a global error, • Lml{p) the first reconstruction error for a pixel P of the first image, • Lms(p) is the second reconstruction error for a pixel P in the third image, • D^^p} a second depth of one pixel P from the first image, • Ds[(p) a first depth of a pixel P of the first image, and • P is a factor dependent on the number of iterations performed during the second learning.

[0115] This overall loss is the result of the following different losses: the first term is the loss related to image generation, the second term is called "supervised loss" related to supervision of the monoscopic vision system by the system of stereoscopic vision and the third term is the loss related to the scale factor. The scale factor is thus defined as the ratio between a first depth and a second depth for a pixel associated with an object present in the field of view common to the first 11 and second 12 cameras.

[0116] The factor V is a variable dependent on the number of iterations (or "epoch") of the training. This factor becomes increasingly important as the training progresses. Indeed, it is preferable to have learned the convolutional neural network before increasing the weight of the scaling factor in this overall loss function.

[0117] Depth prediction by the monoscopic vision system thus achieves greater accuracy. Indeed, the stereoscopic vision system is metrically accurate, meaning that the scale factor in the images is known through the reference frame, whereas the monoscopic vision system cannot determine absolute depth because it defines relative depths. This process allows the monoscopic vision system to achieve metric accuracy, thus the second depth predictions are refined and metrically precise.

[0118] If the ADAS uses second depths as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another road user, the ADAS is then able to determine this distance precisely. For example, if the ADAS is designed to activate the braking system of the vehicle 10 in the event of a risk of collision with another road user, and the distance between the vehicle 10 and that road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of the vehicle 10 to avoid a possible accident, even if the other road user is only seen by a single camera.

[0119] Figure 2 illustrates a flowchart of the different stages of a process 2 of de Determination of depth by means of a monoscopic vision system mounted in a vehicle, for example the vehicle of [Fig. 1], according to a particular and non-limiting embodiment of the present invention. Method 2 is implemented, for example, by a device mounted in the vehicle 10 or by the device 4 of [Fig. 3].

[0120] In a step 21, first representative data of a first image acquired by a first camera 11 of the set of cameras at a first time instant of acquisition are received.

[0121] In a step 22, second data representing a second image acquired by a second camera 12 of the camera set at the same first time instant of acquisition are received.

[0122] In a step 23, the first depths associated with a first set of pixels of the first image are predicted based on optical flux values ​​determined from the first set of pixels and a second set of pixels from the second image corresponding to the first set of pixels.

[0123] In a step 31, third data representing a third image acquired by the first camera 11 at a second time instant of acquisition are received.

[0124] The second time instant of acquisition is different from the first time instant of acquisition of the first image.

[0125] In a step 32, data representing a movement of the first camera 11 between the first time instant of acquisition and the second time instant of acquisition are determined as a function of the first and third images.

[0126] In a step 33, second depths associated with a third set of pixels from the first image are predicted as a function of optical flux values ​​determined from the third set of pixels and a fourth set of pixels from the third image corresponding to the third set of pixels.

[0127] Third depths associated with the fourth set of pixels of the third image are also predicted as a function of optical flux values ​​determined from the fourth and third sets of pixels.

[0128] The second and third depths are predicted by a convolutional neural network associated with a monoscopic vision system including the first camera 11.

[0129] In a step 34, a fourth image is generated from the first image and the second depths by applying an image generation function and a fifth image is generated from the third image and the third depths by applying the same image generation function.

[0130] In a step 35, the convolutional neural network is learned by minimizing a third error determined as a function of a first reconstruction error determined by comparison of the fifth and first images and a second reconstruction error determined by comparison of the fourth and third images.

[0131] In a step 36, a scaling factor of the monoscopic vision system is determined by adjusting parameters of the convolutional neural network as a function of the first and second reconstruction errors and the first and second depths.

[0132] The adjustment is obtained by minimizing an overall error determined by the following first loss function: [Math.7] = Yf / nin(Lmt(p), Lmip)) + Yp\Dmt(p) -Dst(p) | + ÆJ—4 -1|

[0133] With: • an overall error, • Lm^p) the first reconstruction error for a pixel P of the first image, • Lms(p) is the second reconstruction error for a pixel P in the third image, * a second depth of one pixel P from the first image, • Dsi(p) a first depth of a pixel P of the first image, and • P is a factor dependent on the number of iterations performed during the second learning.

[0134] According to one variant, the variants and examples of the operations described in relation to [Fig.1] apply to the steps of process 2 of [Fig.2].

[0135] Figure 3 schematically illustrates a device 4 configured for determining depth by a vision system mounted in a vehicle 10, according to a particular and non-limiting embodiment of the present invention. The device 4 corresponds, for example, to a device mounted in the first vehicle 10, for example a computer.

[0136] Device 4 is, for example, configured to carry out the operations described opposite [Fig. 1] and / or steps described opposite [Fig. 2]. Examples of such a device 4 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of device 4, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 4 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0137] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0138] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 41.

[0139] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0140] According to a particular and non-limiting embodiment, the device 4 includes a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 include one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0141] According to another particular and non-limiting embodiment, the device 4 includes a communication interface 43 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 430. The communication interface 43 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0142] According to a particular and non-limiting embodiment, the device 4 can provide output signals to one or more external devices, such as a screen display 440, touch or not, one or more speakers 450 and / or other peripherals 460 (projection system) via output interfaces 44, 45, 46 respectively. In one variant, one or more of the external devices is integrated into device 4.

[0143] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining depth and measuring such a quantity using a monoscopic vision system learned from a stereoscopic vision system mounted in a vehicle, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0144] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 4 of [Fig.3].

Claims

1. Claims Method for determining a depth by a monoscopic vision system on board a vehicle (10), the vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method being characterized in that it comprises the following steps: - reception (21, 22) of first and second data respectively representative of a first and second image acquired by respectively a first and second camera (11, 12) of said set of cameras at the same first acquisition time instant; - prediction (23) of first depths associated with a first set of pixels of the first image as a function of optical flow values ​​determined from said first set of pixels and a second set of pixels of the second image corresponding to said first set of pixels; - reception (31) of third data representative of a third image acquired by said first camera (11) at a second acquisition time instant different from said first acquisition time instant; - determination (32) of data representative of a movement of the first camera (11) between the first acquisition time instant and the second acquisition time instant as a function of said first and third images; - prediction (33) of second depths associated with a third set of pixels of the first image and of third depths associated with a fourth set of pixels of the third image, said fourth set of pixels corresponding to said third set of pixels, as a function of optical flow values ​​determined from said third and fourth sets of pixels, said second and third depths being predicted by a convolutional neural network associated with a monoscopic vision system comprising the first camera (11); - generation (34) of a fourth image from said first image and the second depths and of a fifth image from said third image and the third depths by application of an image generation function; - training (35) said convolutional neural network by minimizing a third error determined as a function of a first reconstruction error determined by comparison of the fifth and first images and of a second reconstruction error determined by comparison of the fourth and third images;- determining (36) a scale factor of said monoscopic vision system by adjusting parameters of the convolutional neural network as a function of said first and second reconstruction errors and the first and second depths, said adjustment being obtained by minimizing a global error determined by the following first loss function: Lmal = L^nin(Ljji), Lm / p)Dmt(pj-D^p) | + |-11 With: • a global error, • Lm^p) the first reconstruction error for a pixel P of the first image, • Lms{p) the second reconstruction error for a pixel P of the third image, • Dm[(p) a second depth of a pixel P of the first image, • Dst(p) a first depth of a pixel P of the first image, and • P a factor depending on the number of iterations performed during the second learning.;

2. Method according to claim 1, for which said third error is determined by a second loss function defined by: Lm = Epavg(LmZ( / A Lms(p))^c : • J the third error, • avS an average of arguments, • Lm^p) the first reconstruction error for a pixel P of the first image, and • L^p) the second reconstruction error for a pixel P of the third image.

3. Method according to claim 1, for which said third error is determined by a third loss function defined by: Lm = Epmin ( Lm^p\ )With: • J the third error, • min a minimum value of an argument, • Lm^p) the first reconstruction error for a pixel P of the first image, and • L^p) the second reconstruction error for a pixel P of the third image.

4. Method according to one of claims 1 to 3, for which said image generation function is defined by the following function for the monoscopic vision system: Ps = 71 ( K [ T$ ( P^~ Dmt ( Pt ) ) ] ) With: • Ps a pixel of a generated image, • 77 a function for going from homogeneous coordinates to pixel coordinates by removing a dimension of a vector, • an intrinsic matrix of the first camera (11), • T a matrix representing the movement of the first camera (11) between two acquisition time instants, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and * ) is a depth of a pixel Pt predicted by the convolutional neural network.

5. Method according to one of claims 1 to 4, for which said first and second reconstruction errors are further determined by the following function: Lm(p) = Ep[(l-«) ■ R(p)-^(p)l+«' With: • Lm(p) the first reconstruction error, respectively the second reconstruction error, • I(p) a value of the pixel P in the first image, respectively third image, * î(p) a value of the pixel P in the fifth image, respectively fourth image' • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

6. Method according to one of claims 1 to 5, for which said first and second reconstruction errors are determined, in furthermore, by the following function: tp,o} =EXeJ w(pt)) With: • L oth( 0, W, O) the first reconstruction error, respectively the second reconstruction error • D,(p ) is the second predicted depth D, of a pixel Pt, respectively the third predicted depth Dt of a pixel Pt, • W is a parameter matrix, • ° is the order of a smoothing gradient, • an L1 norm of the second order depth gradients is calculated with W =1 , and ° =2, • x and 2 are the dimensions of the first image, respectively the third image, • / > is an environment-dependent hyperparameter, and * If(p) is the value of the pixel Pt in the fifth image, respectively the fourth image.

7. Method according to one of claims 1 to 6, comprising a step of determining disparity values ​​induced by a relative position of optical axes of the first camera (11) at the first and second acquisition time instants, said predictions of second and third depths being, in addition, a function of said induced disparity.

8. Method according to one of claims 1 to 6, comprising a step of rectifying the first and third images as a function of extrinsic parameters of said monoscopic vision system.

9. Device (4) for determining a depth by a monoscopic vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 8.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images