Method and device for determining depth using a self-supervised stereoscopic vision system.

Abstract

Method or device implementing a method for determining depth by stereoscopic vision system mounted in a vehicle (10), the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a scene from a different point of view, the optical axes representing an orientation of the field of vision of each camera being oriented in a non-parallel manner, the method being characterized in that it comprises the steps of cropping images acquired by the stereoscopic vision system, predictions of depths by a convolutional neural network for pixels of each image, generation of images as a function of the predicted depths and an error calculation by comparison of the generated images and the acquired images, the neural network being learned by minimizing an error determined from an average of the errors previously determined.Figure for the abbreviation: Figure 1.

Description

Title of the invention: Method and device for determining depth by a self-supervised stereoscopic vision system. Technical field

[0001] The present invention relates to methods and devices for determining a depth by means of a stereoscopic vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring such a depth. The present invention also relates to a method and a device for controlling one or more AD AS systems on board a vehicle from the determined depth. Technological background

[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.

[0005] The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by a vision system. Summary of the present invention

[0006] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0007] Another object of the present invention is to improve the quality of the data obtained of these cameras.

[0008] Another object of the present invention is to improve road safety, in particular by improving the operational safety of AD AS systems supplied by data obtained from at least one camera.

[0009] According to a first aspect, the present invention relates to a method for determining a depth by a stereoscopic vision system on board a vehicle, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the method being characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same acquisition time instant; - obtaining a first cropped image by cropping the first image and obtaining a second cropped image by cropping the second image, a cropping of the first and second images being a function of fields of vision of the first and second cameras; - prediction of first depths associated with a first set of pixels of the first cropped image from the first set of pixels and a second set of pixels of the second cropped image corresponding to the first set of pixels and prediction of second depths associated with a third set of pixels of the second cropped image from the third set of pixels and a fourth set of pixels of the first cropped image corresponding to the third set of pixels, the first and second depths being predicted by an optical flow method implemented by a convolutional neural network; - generation of a third image from the first cropped image and the first depths and generation of a fourth image from the second cropped image and the second depths, the generations being obtained from the following formula: DA.Pt) ) ] ) With : • Ps a pixel of the third image, respectively fourth image, • 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K an intrinsic matrix of the second camera, respectively of the first camera, associated with the projection of a point of the three-dimensional scene to the co 3-dimensional ordinates in an image with 2-dimensional coordinates, • F an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, • K' an intrinsic matrix of the first camera, respectively of the second camera, associated with the projection of a point of the scene at the 3-dimensional coordinates in the image at the 2-dimensional coordinates, • Dj{p J is a first depth of the pixel Pt of the first cropped image, respectively a second depth of the pixel Pt of the second cropped image; - determination of a first error by comparison of the third image and second cropped image and determination of a second error by comparison of the fourth image and first cropped image; - adjusting input parameters of said convolutional neural network by minimizing a third error, said third error being determined from an average of said first and second errors.

[0010] Such a method allows training of the convolutional neural network from the images acquired by the cameras of the stereoscopic vision system without having to use annotated data. Such training makes it possible to qualify the convolutional neural network as “self-supervised”, that is to say that it is trained from data generated by the stereoscopic vision system. The depths predicted by this stereoscopic vision system are then made reliable.

[0011] According to a variant of the method, the third error is determined by the following loss function: C = E / ?avg(Lrt(p), LJp)) With: • £' the third error, • avS an average of arguments, • Lx£p) the first error for a pixel P of the second cropped image, • Ls£p) the second error for a pixel P of the first cropped image.

[0012] According to another variant of the method, the first and second errors respectively are determined by the following loss function: L.(p) = L[(lo) ■ Ur) -î(p)| +«■ With : • L*(p) the first, respectively second error, • I(p) a value of the pixel P in the second cropped image, respectively the first cropped image, * neither a value of the pixel P in the third image, respectively the fourth picture' • SSIM a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0013] According to another variant of the method, the first and second errors respectively are determined by the following loss function: W'-«) w (p,) I|eW.Wi) With : • L th(O, VV. c») the first, respectively second error, • p) is the first depth, respectively second depth Dtm of a pixel Pt predicted by the stereoscopic vision system; - W is a parameter matrix; - ° is the order of a smoothing gradient; - an L1 norm of second-order depth gradients is calculated with W = 1, and ° =2; - x and are the dimensions of the second image, respectively first image; - / J is an environment-dependent hyperparameter; and - / fp ) is the value of pixel Pt in the third image, respectively fourth image.

[0014] According to a variant of the additional method, the cropping is carried out so as to obtain at least 50% of the pixels of each of said first and second cropped images associated with objects of the three-dimensional scene present in fields of vision of the first and second cameras.

[0015] According to another method variant, the first and second depths are determined by a so-called optical flow method.

[0016] According to another variant, the method comprises a step of determining a disparity induced by a relative position of the first (11) and second (12) cameras, the predictions of first and second depths being, in addition, a function of the induced disparity.

[0017] According to a second aspect, the present invention relates to a device for determining a depth by a stereoscopic vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0018] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0019] According to a fourth aspect, the present invention relates to a program computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0020] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0021] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0022] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.

[0023] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0024] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0025] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:

[0026] [Fig-1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

[0027] [Fig.2] illustrates a flowchart of the different steps of a method for determining a visibility mask by a vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0028] [Fig.3] schematically illustrates images obtained from a stereoscopic vision system on board the vehicle of [Fig.l], according to an example of realization particular and non-limiting use of the present invention;

[0029] [Fig.4] schematically illustrates a device configured for the determination of a Visibility mask by on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention. Description of the exemplary embodiments

[0030] A method and a device for determining a depth by means of a stereoscopic vision system on board a vehicle will now be described in the following with joint reference to FIGS. 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.

[0031] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0032] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth by a stereoscopic vision system on board a vehicle is for example implemented by a computer of the on-board system of the vehicle controlling this stereo system.

[0033] The stereo system comprises a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the optical axes representative of an orientation of the field of vision of each camera being oriented in a non-parallel manner. The stereoscopic vision system is thus said to be non-parallel.

[0034] For this purpose, the method for determining a depth by a stereoscopic vision system on board a vehicle comprises receiving first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same acquisition time instant, cropping the first and second images as a function of fields of vision of the first and second cameras, predicting first depths associated with a first set of pixels of the first image and predicting second depths associated with a third set of pixels of the second image, the first and second depths being predicted by a convolutional neural network.

[0035] The method also comprises generating a third image from the first image and the first depths and generating a fourth image from said second image and the second depths, determining a first error by comparing the third and second images and determining of a second error by comparing the fourth and first images.

[0036] Input parameters of the convolutional neural network are then adjusted by minimizing a third error, the third error being determined from an average of the first and second errors.

[0037] [Fig. 1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0038] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0039] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0040] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereo system. Two cameras 11 and 12 are illustrated in [Fig.l]. The present invention is however not limited to a stereo system comprising two cameras but extends to any stereo system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.

[0041] The two cameras 11, 12 have known intrinsic parameters. These parameters consist in particular of: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.

[0042] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.

[0043] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely the dis radial, decentering, and prismatic twists, induced by defects in curvature, lens parallelism, and optical axis coaxiality. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account or their correction is processed at the time of image acquisition.

[0044] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

[0045] A first marker is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is that of the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0046] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.

[0047] The extrinsic parameters are further determined during a calibration phase of the stereoscopic vision system.

[0048] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the reference base must reach 60cm for the cameras commonly used in this field.

[0049] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0050] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereo system composed here of the three fields 16, 17 and 19.

[0051] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system composed of the second camera 12.

[0052] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system composed of the first camera 11.

[0053] Finally, field 19 is not visible from any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0054] According to an exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0055] According to another exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented horizontally, that is to say that each of the first (11) and second (12) cameras has a horizontal optical axis. In other words, the two optical axes are each arranged in a horizontal plane, the two planes each supporting an optical axis being parallel.

[0056] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.

[0057] The images acquired by the cameras 11, 12 at a given acquisition time instant are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0058] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are found for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0059] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0060] A process for determining a depth by a stereoscopic vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, that is to say by a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the stereoscopic vision system of the vehicle 10.

[0061] In a first operation, the computer receives first data representative of a first image 31a acquired by the first camera 11 at an acquisition time instant.

[0062] In a second operation, the computer receives second data representative of a second image 32a acquired by the second camera 12 at the same acquisition time instant.

[0063] The fields of vision of the first and second cameras being concurrent, a part of the pixels of the first image and a part of the pixels of the second image are associated with the same objects present in the three-dimensional scene.

[0064] The prediction of a depth of a pixel of an image acquired by a camera of the stereoscopic vision system via an algorithm associated with stereoscopic vision is done for a pixel of the first image having an associated pixel in the second image, that is to say that the prediction of the distance of an object of the three-dimensional scene by an algorithm associated with stereoscopic vision is reliable if the object is present in the common field of vision of the first 11 and second 12 cameras, the third acquisition field 15.

[0065] In order to keep parts of the first 31a and second 32a images associated with the third acquisition field 15, a first cropped image 31b is obtained in a third operation by cropping the first image 31a and a second cropped image 32b is obtained in a fourth operation by cropping the second image 32a. The cropping of the first and second images is therefore a function of the fields of vision of the first 11 and second 12 cameras.

[0066] According to a particular embodiment illustrated by [Fig.3], the first cropped image 31b is obtained by cropping a left part 311 of the first image 31a and a lower part 312 of this first image 31a.

[0067] According to this particular embodiment, the second cropped image 32b is obtained by cropping a right part 321 of the second image 32a and an upper part 322 of this second image 32a.

[0068] Of course, the parts of the first 31a and second 32a images that are cropped depend on the relative position and relative orientation of the first 11 and second 12 cameras.

[0069] The first image 31a is for example cropped in its right part and the second image 32a cropped in its left part if the positions of the first 11 and second 12 cameras are reversed.

[0070] The optical center of an image is defined by its coordinates in the image, the origin of the image being for example located at the top left of the image.

[0071] The optical center of the first image 31a therefore has the following coordinates:

[0072] ( [Math.l]

[0073] With w the width of the first image 3la and h the height of the first image 31a.

[0074] If the first 31a and second 32a images have the same dimension, i.e. a width w and a height h, the optical centers of the first 31a and second 31b images both have coordinates (w / 2, h / 2).

[0075] The different parts of the first 31a and second 32a images have the following respective optical centers:

[0076] [Math.2] r3 \b - It cy ~ 2

[0077]

[0078]

[0079]

[0080] r32b - LL 2 / ••311 — LL cr — 2 Cy - 2 ^12 = W312_.^ r3\2-lj3n_ A C y — fl q “ 2 / •322 - h

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] £■321 — W32I _ Ç r321 - A C>' “ 2 The index x represents the abscissas in the horizontal direction, the index y the ordinates in the vertical direction. The superscripts represent the part of an image as represented by the example illustrated in [Fig.3]. The optical centers then make it possible to determine the intrinsic matrices K linked to the different parts of the images, the intrinsic matrix K being defined by: [Math.3] With : • K an intrinsic matrix of an image or part of an image, • / the focal length in pixel units in the horizontal direction of the image or part of the image, • f the focal length in pixel units in the vertical direction of the image or part of the image, • U the abscissa of the optical center of the image or part of the image, • c>' the ordinate of the optical center of the image or part of the image. According to a particular exemplary embodiment, the cropping is carried out so as to obtain at least 50%, 60% or 80% of the pixels of each of the first 31b and second 32b cropped images associated with objects of the three-dimensional scene present in fields of vision of the first 11 and second 12 cameras. In a fifth operation, first depths associated with a first set of pixels of the first cropped image 31b are predicted from the first set of pixels and a second set of pixels of the second cropped image 32b corresponding to the first set of pixels. In a sixth operation, second depths associated with a third set of pixels of the second cropped image 32b are predicted from the third set of pixels and a fourth set of pixels of the first cropped image 31b corresponding to the third set of pixels. The first and second depths are predicted by an optical flow method using a convolutional neural network. Thus, the fifth and sixth operations are carried out by implementing a method called optical flow calculation. Such a method is notably described in “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume” by Deqing Sun, Xiaodong Yang, Ming-Yu Liu and Jan Kautz on June 25, 2018 and in “RAFT: Recurrent Ail-Pairs Field Transforms for Optical Flow” by Zachary Teed and Jia Deng on August 25, 2020.

[0092] The optical flow calculation method is performed by a convolutional neural network (CNN). This type of tool is commonly used in image processing.

[0093] An intermediate data item of the fifth operation is an optical flow representative of a displacement between each pixel of the first set of pixels and a second pixel corresponding to each first pixel in the second cropped image 32b, the set of second pixels corresponding to the second set of pixels.

[0094] Similarly, an intermediate data item of the sixth operation is an optical flow representative of a displacement between each pixel of the third set of pixels and a fourth pixel corresponding to each third pixel in the first cropped image 31b, the set of fourth pixels corresponding to the fourth set of pixels.

[0095] Indeed, the optical flow calculation makes it possible to process the first 31b and second 32b cropped images in both directions without additional operation.

[0096] The first and second depths are then determined from these optical flows, a depth being proportional to a norm of an optical flow.

[0097] The rotation and translation to move from the optical axis of the first camera 11 to the second camera 12 create in particular a so-called “induced” disparity. This disparity is, moreover, included in the optical flow determined from the first 31b and second 32b cropped images.

[0098] The rotation is for example due to the fact that the optical axes of the first 11 and second 12 cameras are not parallel. The translation is for example due to the fact that the first 11 and second 12 cameras are not placed at the same height on the vehicle 10.

[0099] According to a particular exemplary embodiment, in order not to take into account this induced disparity likely to distort the prediction of first and second depths, a convolutional neural network of the FCN type (from the English “Fully Convolutional network”), subsequently called an FCN, is implemented to determine the induced disparity and filter the optical flow value before predicting the depth of a pixel.

[0100] Thus, according to this particular exemplary embodiment, in a seventh operation, a first induced disparity is determined by the FCN from the first 31b and second 32b cropped images making it possible to subtract the first induced disparity from the optical flow to obtain a corrected optical flow, the first depths are then predicted as a function of this corrected optical flow.

[0101] Similarly, in an eighth operation, a second induced disparity is determined by the FCN from the second 32b and first 31b cropped images making it possible to subtract the second induced disparity from the optical flow to obtain a corrected optical flow, the second depths are then predicted as a function of this corrected optical flow.

[0102] The input data of such an FCN is a one-dimensional matrix HxWxN with H the height of an image, W the width of an image and N a number of channels including the unnullified elements of the rotation matrix (at most 9), the unnullified elements of the translation vector (at most 3) as well as the two optical flows (horizontal and vertical). The maximum number of channels is therefore 14. The number of intermediate layers and the number of channels of the matrix at each layer depends on the deviation of the parallel system.

[0103] The structure of this FCN is constructed by experimentation in such a way as to minimize the size of the FCN while providing an acceptable result.

[0104] In a ninth operation, a third image is generated from the first cropped image 31b and the first determined depths.

[0105] Similarly, in a tenth operation, a fourth image is generated from the second cropped image 32b and the second determined depths.

[0106] The generations of the third and fourth images are obtained from the following formula: [Math.4] P s = n (K [( P^ 1' ( Pt ) ) )

[0107] With: • Ps a pixel of the third image, respectively fourth image, • 77 a function to go from homogeneous coordinates to pixel coordinates in removing a dimension from a vector, • K an intrinsic matrix of the second camera 12, respectively of the first camera 11, associated with the projection of a point of the three-dimensional scene to the co 3-dimensional ordinates in an image with 2-dimensional coordinates, • P an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, • K' an intrinsic matrix of the first camera 11, respectively of the second camera 12, associated with the projection of a point of the scene at the coordinates in 3 dimensions in the image to the 2-dimensional coordinates, • D^p ) is a first depth of the pixel Pt of the first cropped image 31b, respectively a second depth of the pixel P, of the second cropped image 32b.

[0108] The principle of image generation consists of projecting a pixel of an image acquired by the first camera 11 towards a point in the three-dimensional scene as a function of its first depth, then reprojecting this point into an image as it would be acquired from the point of view of the second camera 12 and, respectively, projecting a pixel of an image acquired by the second camera 12 towards a point in the three-dimensional scene as a function of its second depth, then reprojecting this point into an image as it would be acquired from the point of view of the first camera 11.

[0109] In an eleventh operation, a first error is determined by comparing the third image and the second cropped image 32b.

[0110] Similarly, in a twelfth operation, a second error is determined by comparing the fourth image and the first cropped image 31b.

[0111] According to a first particular exemplary embodiment, the first and respectively second errors are determined by the following loss function: [Math.5] L^p) = Ep[ (la) ■ \I(p)-I(p) \+a-(l-jSSIM(I(p),I(p) ) ) ]

[0112] With: • L*(p) the first, respectively second error, • I(p) a value of the pixel P in the second cropped image (32b), respectively the first cropped image (31b), • a value of the pixel P in the third image, respectively the fourth picture' • SSIM a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0113] This loss function is based on the photometric error. Once a third image, or respectively fourth image, is generated, it is compared to the true image acquired by the stereoscopic vision system, the second cropped image 32b or respectively the first cropped image 31b.

[0114] According to a second particular exemplary embodiment, the first and respectively second errors are determined by the following loss function: [Math.6] «) = L>L, a , y ( w WI4) I 1 )

[0115] With: • L oth(0, IV, o) the first, respectively second error, • Dst ( p ) is the first depth, respectively second depth D!m of a pixel Pt predicted by the stereoscopic vision system; - W is a parameter matrix; - 0 is the order of a smoothing gradient; - an L1 norm of second-order depth gradients is calculated with W =1, and0 =2; - x and ? are the dimensions of the second image, respectively first image; - / î is an environment-dependent hyperparameter; and - it) is the value of the pixel Pt in the third image, respectively fourth image.

[0116] This second loss function is generally used to deal with discontinuity at the edge of objects (in English “edge aware smoothness”).

[0117] According to a third particular exemplary embodiment, the first and second errors are determined by a function combining the function presented in the first particular exemplary embodiment and by the function presented in the second particular exemplary embodiment, for example by performing the sum of the two functions or by applying one or the other of the functions according to a location of the pixels of the first 31b and second 32b cropped images.

[0118] The operations for predicting the first and second depths are further implemented by a self-supervised convolutional neural network. This learning from the data obtained from the stereoscopic vision system makes it possible to refine the parameters of the convolutional neural network and thus make the intermediate data such as the disparities or optical flows or output data such as the first and second depths more reliable.

[0119] Such a convolutional neural network is known to those skilled in the art, such as PWCNet (from the English “Pyramidal processing, Warping, and the use of a Cost volume Network”).

[0120] According to a particular exemplary embodiment, to find the correct values ​​of the input parameters (called “weights” in English) of a chosen convolutional neural network, the first and second errors are minimized in a thirteenth operation.

[0121] In a fourteenth operation, the input parameters of the convolutional neural network are adjusted by minimizing a third error, the third error being determined from an average of the first and second errors.

[0122] According to a particular exemplary embodiment, the third error is determined by the following loss function: [Math.7] Lx = Hpavg(Lst(p\ LXi(p))

[0123] With: • £ . the third error, • avS an average of arguments, • L^p) the first error for a pixel P of the second cropped image (32b), • Li^p) the second error for a pixel P of the first cropped image (31b).

[0124] The averaging of the first and second errors further allows each of the first and second errors to be taken into consideration in an equivalent manner.

[0125] [Fig.2] illustrates a flowchart of the different steps of a method 2 for determining a depth by a stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention.

[0126] The method 2 is for example implemented by one or more processors of one or more computers on board the vehicle 10, for example by a computer controlling the stereo system.

[0127] In a first step 21, first data representative of a first 31a image acquired by a first camera 11 are received.

[0128] In a second step 22, second data representative of a second 32a image acquired by a second camera 12 are received.

[0129] The two images received correspond to two views of the same scene taking place around the vehicle 10 at the same acquisition time instant.

[0130] In a third step 23a, a first cropped image 31b is obtained by cropping the first image 31a.

[0131] In a fourth step 23b, a second cropped image 32b is obtained by cropping the second image 32a.

[0132] A cropping of the first and second images is a function of fields of vision of the first 11 and second 12 cameras.

[0133] In a fifth step 24a, first depths associated with a first set of pixels of the first cropped image 31b are predicted from the first set of pixels and a second set of pixels of the second cropped image 32b corresponding to the first set of pixels.

[0134] In a sixth step 24b, second depths associated with a third set of pixels of the second cropped image 32b are predicted from the third set of pixels and a fourth set of pixels of the first cropped image 31b corresponding to the third set of pixels.

[0135] The first and second depths are predicted by an optical flow method implemented by a convolutional neural network.

[0136] In a seventh step 25a, a third image is generated from the first cropped image 31b and the first predicted depths.

[0137] In an eighth step 25b, a fourth image is generated from the second cropped image 32b and second depths.

[0138] The image generations are obtained from the following formula: [Math. 8] \ Dst(pf) ) ] )

[0139] With: • Ps a pixel of the third image, respectively fourth image, •77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K an intrinsic matrix of the second camera 12, respectively of the first camera 11, • P an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, • K' an intrinsic matrix of the first camera 11, respectively of the second camera 12, • is a first depth of the pixel Pt of the first cropped image 31b, respectively a second depth of the pixel P / of the second cropped image 32b.

[0140] In a ninth step 26a, a first error is determined by comparing the third image and second cropped image 32b.

[0141] In a tenth step 26b, a second error is determined by comparing the fourth image and first cropped image 31b.

[0142] In an eleventh step 27, input parameters of the convolutional neural network are adjusted by minimizing a third error, the third error being determined from an average of the first and second errors

[0143] Thus, the predictions of the first and second depths are refined, improving the reliability of the output data of the stereoscopic vision system.

[0144] An ADAS using the first or second depths as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases sharply, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.

[0145] [Fig.4] schematically illustrates a device 4 configured for the determination of a visibility mask by a vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device on board the first vehicle 10, for example a computer.

[0146] The device 4 is for example configured for the implementation of the operations described with regard to figures 1 and 3 and / or the steps described with regard to [Fig.2]. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0147] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0148] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.

[0149] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0150] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0151] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0152] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

[0153] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining a depth by a stereoscopic vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0154] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.4].

Claims

1. Claims Method for determining a depth by a stereoscopic vision system on board a vehicle (10), the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method being characterized in that it comprises the following steps: - reception (21, 22) of first and second data respectively representative of a first (31a) and second (32a) images acquired by respectively a first and second camera (11, 12) of said set of cameras at the same acquisition time instant; - obtaining (23a) a first cropped image (31b) by cropping the first image (31a) and obtaining (23b) a second cropped image (32b) by cropping the second image (32a), a cropping of the first and second images being a function of fields of vision of said first (11) and second (12) cameras; - prediction (24a) of first depths associated with a first set of pixels of the first cropped image (31b) from said first set of pixels and a second set of pixels of the second cropped image (32b) corresponding to said first set of pixels and prediction (24b) of second depths associated with a third set of pixels of the second cropped image (32b) from said third set of pixels and a fourth set of pixels of the first cropped image (31b) corresponding to said third set of pixels, said first and second depths being predicted by an optical flow method implemented by a convolutional neural network; - generation (25a) of a third image from said first cropped image (31b) and the first depths and generation (25b) of a fourth image from said second cropped image (32b) and the second depths, the generations being obtained from the following formula: Ps = tt ((pt\K"\Dst(pt)j) With : • Ps a pixel of the third image, respectively fourth image, •77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K an intrinsic matrix of the second camera (12), respectively of the first camera (11), • F an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, • K' an intrinsic matrix of the first camera (11), respectively of the second camera (12), • D^p^ is a first depth of the pixel Pt of the first cropped image (31b), respectively a second depth of the pixel Pt of the second cropped image (32b); - determining (26a) a first error by comparing the third image and second cropped image (32b) and determining (26b) a second error by comparing the fourth image and first cropped image (31b); - adjusting (27) input parameters of said convolutional neural network by minimizing a third error, said third error being determined from an average of said first and second errors.

2. The method of claim 1, wherein said third error is determined by the following loss function: Ls = Epavg( L^p), L4p)) With: • p the third error, • avS an average of arguments, • Ls!(p} the first error for a pixel P of the second cropped image (32b), • L,s(p) the second error for a pixel P of the first cropped image (31b).

3. Method according to one of claims 1 to 2, for which said first and respectively second errors are determined by the following loss function: l*(p) = EJ ( i-«) ■ \i(p)-Up) I + «■ (i-±ssim(i(p),î(p) ) ) ] With: • LPp) the first, respectively second error, respectively the first cropped image (31b), * a value of the pixel P in the third image, respectively the fourth image' • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

4. Method according to one of claims 1 to 3, for which said first and respectively second errors are determined by the following loss function: With: • Lsnwoth( 0, W, o) the first, respectively second error, • ) is the first depth, respectively second depth Dtm of a pixel Pt predicted by the stereoscopic vision system; - W is a parameter matrix; - ° is the order of a smoothing gradient; - an L1 norm of the second-order depth gradients is calculated with W =1, and ° =2; - x and 37 are the dimensions of the second image, respectively first image; - is an environment-dependent hyperparameter; and - hip ) is the value of the pixel Pt in the third image, respectively fourth image.

5. Method according to one of claims 1 to 4, for which said cropping is carried out so as to obtain at least 50% of the pixels of each of said first and second cropped images (31b, 32b) associated with objects of the three-dimensional scene present in fields of vision of the first (11) and second (12) cameras.

6. Method according to one of claims 1 to 5, comprising a step of determining a disparity induced by a relative position of the first (11) and second (12) cameras, said predictions of first and second depths being, in addition, a function of the induced disparity.

7. Computer program comprising instructions for implementing the method according to any one of the preceding claims.

8.

9.

10. preceding, when these instructions are executed by a processor. Recording medium readable by a computer on which is recorded a computer program comprising instructions for executing the steps of the method according to claims 1 to 6. Device (4) for determining a depth by a stereoscopic vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 6. Vehicle (10) comprising the device (4) according to claim 9.

Images