Method and device for determining depth using a self-supervised vision system.
Patent Information
- Application Number
- FR2023003341
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-04-04
- Publication Date
- 2025-02-21
- Estimated Expiration
- 2043-04-04
Smart Images
Figure 00000034_0000 
Figure 00000035_0000 
Figure 00000036_0000
Abstract
Description
Title of the invention: Method and device for determining a depth by a self-supervised vision system. Technical field
[0001] The present invention relates to methods and devices for determining a depth by means of a stereoscopic vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring such a depth. The present invention also relates to a method and a device for controlling one or more AD AS systems on board a vehicle from the determined depth. Technological background
[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.
[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.
[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.
[0005] The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by a vision system.
[0006] Summary of the present invention
[0007] An object of the present invention is to solve at least one of the problems of the technological background described above.
[0008] Another object of the present invention is to improve the quality of the data obtained of these cameras.
[0009] Another object of the present invention is to improve road safety.
[0010] According to a first aspect, the present invention relates to a method for determining a depth by a stereoscopic vision system on board a vehicle, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representative of an orientation of the field of vision of each camera being oriented in a non-parallel manner. The method is characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same first acquisition time instant; - prediction of first depths associated with a first set of pixels of the first image as a function of disparity values determined from the first set of pixels of the first image and a set of pixels of the second image corresponding to the first set of pixels of the first image; - projection into the three-dimensional scene of the first set of pixels of the first image in the form of a first set of points as a function of the first depths; - determination of a first visibility mask by reconstruction of a fourth image by reprojection of each point of the first set of points as a function of intrinsic parameters of the first camera and extrinsic parameters of the stereoscopic vision system, when a first plurality of points of the first set of points are projected onto the same pixel of the fourth image, then a point of the first plurality of points having the smallest depth is associated with the same pixel, the other point(s) of the first plurality of points being selected to generate a first visibility mask representative of a second set of pixels of the first image pixels of the first image not visible in the second image; - reception of third data representative of a third image acquired by the first camera at a second acquisition time instant prior to the first acquisition time instant of the first image, the first camera forming a monoscopic vision system; - determination of data representative of a movement of the first camera between the first time instant and the second time instant as a function of the first and third images; - prediction of second depths associated with a third set of pixels of the first image from the third set of pixels of the first image and of a set of pixels of the third image corresponding to the third set of pixels of the first image; - projection into the three-dimensional scene of the third set of pixels of the first image in the form of a second set of points according to the second depths; - determining a second visibility mask by reconstructing a fifth image by reprojecting each point of the second set of points as a function of intrinsic parameters of the first camera and data representative of the movement of the first camera, when a second plurality of points of the second set of points are projected onto the same pixel of the fifth image, then a point of the second plurality of points having the smallest depth is associated with the same pixel, the other point(s) of the second plurality of points being selected to generate a second visibility mask representative of a fourth set of pixels of the first image not visible in the third image; - determination of a mask of dynamic objects associated with the first image and representative of a fifth set of pixels of the first image associated with at least one moving object in the three-dimensional scene by comparing positions of the points of the second set of points and points representative of a set of pixels of the third image projected into the three-dimensional scene in the form of a third set of points as a function of second depths predicted for the set of pixels of the third image; - first automatic learning of the stereoscopic vision system under supervision of the monoscopic vision system by minimizing a first calculation error between the first and second predicted depths for the first image as a function of the first visibility mask, second visibility mask and dynamic object mask.
[0011] According to a method variant, the first learning is obtained, in addition, by minimizing the following function representative of the first calculation error: 1^=14 il d4p,)-d, (p,) ii )av“: - Lnr^s corresponds to a loss of recalibration of the second depth compared to the first depth; - Pt is a pixel of the first image; - V^ / ?) is information on the visibility of the pixel Pt by the stereoscopic vision system; - Vm(p ) is information on the visibility of the pixel Pt by the mono- vision system scopic; - Md( P ) is motion information obtained from the mask of dynamic objects; ■ D,m(p is the second predicted depth for pixel Pt; and ■ DtJ^p is the first predicted depth for pixel Pt.
[0012] According to another variant, the method further comprises a second automatic learning of the monoscopic vision system under supervision of the stereoscopic vision system by minimizing a second calculation error between the first and second depths predicted for the first image as a function of the first visibility mask.
[0013] The method thus gains in reliability, the prediction of the second depths being improved by the supervision of the stereoscopic vision system.
[0014] According to a method variant, the second automatic learning is obtained by minimizing the following function representative of the second calculation error: V s (p) He D tm (p )-D (pj II j '\ ■ 2 /
[0015] with: - L^m corresponds to a loss of recalibration of the first depth compared to the second depth; - Pt is a pixel of the first image; - ^^P ) is information on the visibility of the pixel Pt by the stereoscopic vision system; - Dtm is the second predicted depth for pixel Pt; and - Dts is the first predicted depth for pixel Pt.
[0016] According to an additional method variant, the first depths are predicted via a first self-supervised learning convolutional neural network, the method further comprising the step of obtaining a first reconstruction error by comparing the second and fourth images, the first convolutional neural network being learned in a self-supervised manner as a function of the first reconstruction error.
[0017] According to yet another method variant, the second depths are predicted via a self-supervised learning convolutional neural network, the method further comprising the step of obtaining a second reconstruction error by comparing the third and fifth images, the second convolutional neural network being learned in a self-supervised manner as a function of the second reconstruction error.
[0018] According to another method variant, the first and second reconstruction errors are respectively obtained by the following loss function: = (1-a) ■ |I(p)-î(p)|+a- (14SSIM(ï(p),Î(p)))
[0019] with: - Kp) a value of the pixel P in the second image and the third image respectively; - a value of the pixel P in the reconstructed image corresponding respectively to the fourth image and the fifth image; - SSIM (from the English “structural similarity index measure”) a function which takes into account a local structure; and - a corresponds to a weighting factor depending on the type of road environment in which the vehicle is driven.
[0020] According to yet another method variant, the dynamic object mask is defined by the following formulas: Mh (pt) = Vm (Pt) [T(p^Dsm(ps))-0(pptm(Pt))] = l-eôlMd 11X4^4: - Mh[ p ) corresponds to a static object mask applied to a target pixel Pt of the first image; - Vm(p ) is representative of a visibility of the target pixel Pt of the first image by the monoscopic vision system; - corresponds to a displacement matrix of the camera (11) between the second acquisition time instant and the first acquisition time instant; - corresponds to a backprojection of a pixel P with its corresponding depth D(p); ■ p ) is the second depth of a source pixel Ps of the second image calculated for the monoscopic vision system; - Dtm^P ) is 'a First depth of the target pixel Pt of the first image calculated for the monoscopic vision system; - Md( p ) corresponds to the dynamic object mask applied to the target pixel Pt of the first image; - K corresponds to an intrinsic matrix of the camera (11); is a squared error; 2 transforms a vector of coordinates of the target pixel Pt into a coordinate homogeneous by adding a dimension to the coordinate vector of the target pixel Pt; and - § corresponds to a predefined hyperparameter based on a type of road environment in which the vehicle circulates.
[0021] According to a second aspect, the present invention relates to a device for determining a depth by a stereoscopic vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.
[0022] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.
[0023] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.
[0024] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.
[0025] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.
[0026] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.
[0027] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.
[0028] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures
[0029] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:
[0030] [Fig-1] schematically illustrates a non-parallel stereoscopic vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention;
[0031] [Fig.2] schematically illustrates a device configured for determining a depth by a stereoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention;
[0032] [Fig.3] illustrates a flowchart of the different operations of a process for determining a depth by a stereoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention;
[0033] [Fig.4] illustrates a flowchart of the different steps of a method for determining a depth by a stereoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention.
[0034] Description of the examples of embodiment
[0035] A method and a device for determining a depth by a stereoscopic vision system, hereinafter called a stereo system, on board a vehicle will now be described in what follows with joint reference to Figures 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.
[0036] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth by a stereoscopic vision system on board a vehicle, called a stereo system, is for example implemented by a computer of the on-board system of the vehicle controlling this stereo system.
[0037] The stereo system comprises a set of cameras of at least two cameras arranged so as to each acquire an image of a scene from a different point of view, the optical axes representative of an orientation of the field of vision of each camera being oriented in a non-parallel manner. The stereoscopic vision system is thus said to be non-parallel.
[0038] For this purpose, the method for determining a depth by a stereoscopic vision system on board a vehicle comprises the reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of a set of cameras at the same first acquisition time instant, the prediction of first depths associated with a first set of pixels of the first image, the projection into the three-dimensional scene of the first set of pixels of the first image in the form of a first set of points as a function of the first depths and the determination of a first visibility mask by reconstruction of a fourth image by reprojection of each point of the first set of points as a function of intrinsic parameters of the first camera and extrinsic parameters of the stereoscopic vision system.
[0039] The method also comprises receiving third data representative of a third image acquired by the first camera at a second acquisition time instant prior to the first acquisition time instant of the first image, the first camera forming a monoscopic vision system. Data
[0040] representative of a movement of the first camera between the first time instant and the second time instant are determined as a function of the first and third images, second depths associated with a third set of pixels of the first image are predicted from the third set of pixels of the first image and a set of pixels of the third image corresponding to the third set of pixels of the first image, the third set of pixels of the first image is projected into the three-dimensional scene in the form of a second set of points as a function of the second depths,a second visibility mask is determined by reconstructing a fifth image by reprojecting each point of the second set of points as a function of the intrinsic parameters of the first camera and data representative of the movement of the first camera and a dynamic object mask associated with the first image and representative of a fifth set of pixels of the first image associated with at least one moving object in the three-dimensional scene is determined by comparing positions of the points of the second set of points and points representative of a set of pixels of the third image projected into the three-dimensional scene in the form of a third set of points as a function of second depths predicted for the set of pixels of the third image.
[0041] A first automatic learning of the stereoscopic vision system is carried out under supervision of the monoscopic vision system by minimizing a first calculation error between the first and second predicted depths for the first image as a function of the first visibility mask, second visibility mask and dynamic object mask.
[0042] [Fig. 1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.
[0043] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.
[0044] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.
[0045] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereo system. Two cameras 11 and 12 are illustrated in [Fig. 1]. The present invention is however not limited to a stereo system comprising two cameras but extends to any stereo system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.
[0046] The two cameras 11, 12 have known intrinsic parameters. These parameters consist in particular of: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.
[0047] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.
[0048] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account or that their correction is processed at the time of image acquisition.
[0049] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the rear right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.
[0050] A first marker is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is that of the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.
[0051] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.
[0052] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.
[0053] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the "baseline" must reach 60cm for the cameras commonly used in this field.
[0054] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.
[0055] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereo system composed here of the three fields 16, 17 and 19.
[0056] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system composed of the second camera 12.
[0057] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system composed of the first camera 11.
[0058] Finally, field 19 is not visible from any of the cameras. The part of the scene present in this field 19 is therefore not observable.
[0059] According to an exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.
[0060] According to another exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented parallel.
[0061] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.
[0062] The images acquired by the cameras 11, 12 at a given acquisition time instant tl are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of the objects in the observed scene in the form, for example, of RGB (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness) colorimetric coordinates.
[0063] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are found, for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.
[0064] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.
[0065] [Fig. 2] schematically illustrates a device 4 configured for determining a depth by a stereoscopic vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device on board the first vehicle 10, for example a computer.
[0066] The device 4 is for example configured for the implementation of the operations and / or steps described with regard to figures 1, 3 and 4. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.
[0067] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.
[0068] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.
[0069] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.
[0070] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (LTE) “Long-Term Evolution” or in French “Evolution à long ternie”), LTE-Advanced (or in French LTE-avancé); - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).
[0071] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).
[0072] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.
[0073] [Fig. 3] illustrates a flowchart of the different operations of a process 2 for determining a depth by a stereoscopic vision system on board the vehicle of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention.
[0074] The process is for example implemented by one or more processors of one or more computers on board the vehicle 10, for example by a computer controlling the stereo system.
[0075] In a first operation 21, the computer receives first data representative of a first image acquired by a first camera 11 of the set of cameras at a first acquisition time instant t1.
[0076] In a second operation 22, the computer receives second data representative of a second image acquired by a second camera 12 of the set of cameras at the same first acquisition time instant t1.
[0077] The two images received correspond to two views of the same scene taking place around the vehicle 10 at the same given first acquisition time instant tl.
[0078] In order to facilitate the analysis of the two received images, the first and second images are rectified according to a method known to those skilled in the art. Such a method is described, for example, in “Projective Rectification of Non-Calibrated Infrared Stereo Images with Global Consideration of Distortion Minimization” by Benoit Ducarouge, Thierry Sentenac, Florian Bugarin and Michel Devy of July 16, 2009.
[0079] The rectification method consists of reorienting the epipolar lines so that they are parallel with the horizontal axis of the image. This method is described by a transformation which projects the epipoles to infinity and whose corresponding points are necessarily on the same ordinate.
[0080] A rectification algorithm consists, for example, of 4 steps: - Rotate (virtually) the first camera 11 so that the epipole goes to infinity along the horizontal axis of the reference frame associated with it; - Apply the same rotation to the second camera 12 to end up in the initial geometric configuration; - Rotate the second camera 12 by the rotation associated with the rotation matrix 'R', corresponding to the extrinsic parameter of the starting stereoscopic vision system; - Adjust the scale in both camera markers.
[0081] It should be noted that rectification simplifies the matching of pixels in stereo images, i.e. images obtained by a stereoscopic vision system. The corresponding pixel in the second image to a pixel in the first image (and vice versa) is positioned on the same line. From the knowledge of the epipolar geometry and therefore of a fundamental matrix of the stereo system, the objective is then to determine a pair of projective transformations, called homographies, which reorient the epipolar projections parallel to the lines of the images, therefore to the horizontal axis of the rectified cameras.
[0082] In an operation 23, a set of pixels of the second image corresponding to a first set of pixels of the first image is determined and geometric data associating the pixels of the first set of pixels of the first image with the pixels of the set of pixels of the second image are obtained, this operation is called “stereo matching” (in English “feature matching”).
[0083] Stereo matching or disparity estimation is the process of finding pixels in stereoscopic views that correspond to the same 3D point in the scene. Rectified epipolar geometry simplifies this process of finding correspondences on the same epipolar line. There is no need to calculate the co ordinates of the 3D point to find the corresponding pixel on the same line of the other image. The disparity is the distance d between a pixel and its correspondence in the other image. This disparity is horizontal when the images are rectified, the horizontality being defined according to the x direction of the first image.
[0084] The output data of this operation 23 is a first depth D^p ) of a pixel Pt predicted from a disparity d(p) representative of a displacement between each pixel of the first set of pixels of the first image and the pixel corresponding to each pixel in the second image:
[0085] [Math.l]
[0086] with: - f the focal length of the first camera 11; - B the baseline of the stereo system; and ■ d(p J the disparity of the pixel Pt.
[0087] It is possible that a pixel of the first image does not find a corresponding pixel in the second image. This phenomenon is explained by the fact that areas of the first image may be occluded in the second image. Indeed, the difference in point of view of the two cameras 11, 12 does not allow the two cameras 11, 12 to see all the elements of the scene. An object present in the scene, for example the obstacle 18, may mask a second object of the scene, the second object being visible from the point of view of the first camera 11 but being masked by the obstacle 18 from the point of view of the second camera 12.
[0088] In an operation 24, the first set of pixels of the first image are projected into the three-dimensional scene in the form of a first set of points according to the first depths previously determined. The calculation of the spatial position of a point Pi corresponding to the projection of a pixel P; of the first image is done using the following function:
[0089] [Math.2] p^^pF'D«W)
[0090] with: - K an intrinsic matrix of the camera 11 associated with the projection of a point in space at 3-dimensional coordinates into the first image at 2-dimensional coordinates; - 0 a backprojection function in the scene of a pixel Pf according to its depth; ■ Dt^p ) is a first depth of the pixel Pt of the first image predicted by the stereoscopic vision system.
[0091] In an operation 25, a fourth image is reconstructed by reprojecting each point of the first set of points as a function of intrinsic parameters of the first camera 11 and first depths predicted by the stereoscopic vision system.
[0092] The coordinates of the points Pss projected in the fourth image are defined by the following formula:
[0093] [Math.3]
[0094] or:
[0095] [Math.4]
[0096] with: - 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector; - K an intrinsic matrix of the first camera 11 associated with the projection of a point in space with 3-dimensional coordinates into the image with 2-dimensional coordinates; - K' an intrinsic matrix of the second camera 12 associated with the projection of a point in space with 3-dimensional coordinates into the image with 2-dimensional coordinates; - F a fundamental matrix linked to the extrinsic parameters of the stereoscopic vision system; - 0 a backprojection function in the scene of a pixel Pt according to its depth; - Dtnlp ) is a first depth of the pixel Pt of the first image predicted by the monoscopic vision system.
[0097] When a plurality of points of the set of points are projected onto the same pixel of the fourth image, then a point of the plurality of points having the smallest depth is associated with this same pixel.
[0098] The other point(s) of the plurality of points are selected to generate a first visibility mask representative of a second set of pixels of the first image not visible in the second image, the pixels of the first image referred to as non-visible being those associated with the points not selected during the projection into the fourth image.
[0099] At the output of this operation 25, a first visibility mask representative of pixels visible in the second image and not visible in the first image is determined from the fourth image, a pixel being defined as not visible when none of the points in the point set projects onto the pixel in the fourth image.
[0100] According to a variant, the value l(x, y) of a pixel (RGB or TSL for example) of the third image at the coordinates (a; y) can be defined as a function of the values 'U) of the projected points around the pixel. The value of a pixel is, for example, defined as a function of the values of the projected points weighted by the distance separating the projected points from the pixel. Thus, the definition of the value of a pixel can be carried out using the following function:
[0101] [Math.5] K* y) xl(p.). if |p(x, y) -p.| < 1
[0102] with: -1(x, y) the value of the pixel at location (x, y) in the third image; - p(x, y) the pixel coordinate; - P, the location of the point in the scene projected in the third image; 11 represents the Euclidean distance; - jpj is the RGB or TSL value of the projected 3D point, where / ( x, y ) is the RGB value of the pixel to be calculated.
[0103] A maximum of 4 points are thus used to calculate the value of a pixel, the points being present within a radius of one pixel.
[0104] It should be noted that the value of the point selection radius is defined as being equal to one pixel, but this value is set according to the results obtained during tests. A radius of 0.5 pixels, for example, makes it possible to increase the precision of the calculation but would not make it possible to take into account all the points around the pixel. This radius is therefore a compromise between precision and taking into account a larger number of points.
[0105] According to an exemplary embodiment, the operations of calculating the disparity and searching for associated pixels are carried out by a first convolutional neural network, that is to say that the prediction of the first depths is obtained via a convolutional neural network, the latter being for example trained in a self-supervised manner. This self-supervised learning makes it possible to refine the parameters of the convolutional neural network and thus to make the output data such as the disparity values more reliable.
[0106] Such a convolutional neural network is known to those skilled in the art, such as for example: GCNet (from the English “Global Context Network”, in French “network of global context"), PSMNet (from the English "Pyramid Stereo Matching Network"), PWCNet (from the English "Pyramidal processing, Warping, and the use of a Cost volume Network").
[0107] In an operation 26, to find the correct values of the parameters (called "weights" in English) of a chosen convolutional neural network, the fourth image called the target image (in English "target") is compared to the second image, called the "source" image pixel by pixel,
[0108] The first convolutional neural network is trained to minimize the various reconstruction errors calculated by loss functions, such as the following functions. The performance of the convolutional neural network is thus evaluated and its parameters optimized.
[0109] The first loss function is based on the photometric error. Once the source image is reconstructed from the second image and the disparities calculated, it is compared to the true source image and defined as follows:
[0110] [Math.6] Mo) = o) • s(i(p, ), î s (p f o) )) [YES] with: - Ls is the photometric loss function of the stereo system; - index 5 indicates “stereo”; - Vs(pt, O) is the visibility by the stereo system of the reconstructed pixel; - l(p^ is the value of pixel Pt in the target image; ' I^p 0} is 'a value of the reconstructed pixel for the stereo system defined from the pixel Pt and O; - O is the first depth Dts of a pixel Pt calculated for the stereo system;
[0112] Function 5 is the photometric error defined as follows:
[0113] [Math.7] s(l(p),î(p)) = (1-a) ■ |I(p) -î(p)|+a- (l-|SSIM(l(p),î(p)))
[0114] with: - Kp) is the value of pixel P in the target image, here the second image; - is the value of pixel P in the reconstructed image, here the fourth image; - SSIM (from the English “structural similarity index measure”) is a function which takes into account a local structure; and - a is a weighting factor depending in particular on the type of environment.
[0115] The second loss function is based on a consistency of the disparity obtained from the first image to the second image and that obtained in the opposite direction, i.e. from the second image to the first image. Minimizing the output value of this function helps in the convergence of the stereo system architecture. The definition of this function is as follows:
[0116] [Math. 8] T -JLY i Lsc - N 2-pj dp j
[0117] with: - Lsc is the coherence loss function for the stereo system; - the index ' indicates “stereoscopic”; - index 6 indicates “coherence” (in English “consistency”); - N is a number of pixels of the first image; - P is a pixel of the first image; and - the superscripts (1) and (2) indicate respectively that the disparity is obtained from the first image to the second image and from the second image to the first image.
[0118] The third loss function is generally used to deal with discontinuity at the edge of objects (in English "edge aware smoothness") and is defined as follows:
[0119] [Math.9] U„<„ h (o, W, o) = w(p t ) |V»o(p t ) I )
[0120] with: - O is the first depth Dts of a pixel Pt predicted by the stereo system; - IL is a parameter matrix; - ° is the order of a smoothing gradient - an L1 norm of second-order depth gradients is calculated with W =1, and() =2; - x and -1' are the dimensions of the first image; - fi is an environment-dependent hyperparameter; and ■ It(p ) is the value of pixel Pt in the target image.
[0121] Thus, using these loss functions, parameters of the convolutional neural network are defined. The output result of operation 26 is a prediction of a first depth Dts by the monoscopic vision system associated with at least one pixel of the first set of pixels of the first image.
[0122] In an operation 31, the calculator receives third representative data of a third image acquired by the first camera 11 at a second acquisition time instant t2.
[0123] This second acquisition time instant t2 is prior to the first acquisition time instant t1.
[0124] The third data have, for example, been saved in a memory associated with the computer or in a memory of a device on board the vehicle 10 and accessible to the computer implementing the process.
[0125] If the vehicle 10 is moving, then the third image corresponds to a third view of the scene taken from a third point of view, that of the first camera 11 at its position at the second acquisition time instant t2.
[0126] This position of the first camera 11 at the second acquisition time instant t2 is defined by the movement of the vehicle 10 between the times t2 and t1. This movement is therefore linked to the speed of the vehicle 10 during the time separating the first and second acquisition time instants t1 and t2.
[0127] The first camera 11 in the two positions defined at the first and second acquisition time instants t1 and t2 forms a monoscopic vision system. The intrinsic parameters of this system remain the same as those defined previously related to the first camera 11 for the stereo system. The extrinsic parameters of this monoscopic vision system are the following parameters: - 3 translations in the x, y and z directions: Tx', Ty' and Tz' constituting the translation vector T'; and - 3 rotations around the x, y and z axes: Rx', Ry' and Rz', constituting the rotation matrix R'.
[0128] In an operation 32, the extrinsic parameters of the monoscopic vision system are, for example, determined by a computer associated with this same monoscopic vision system. The determination of the extrinsic parameters of the monoscopic vision system is known to those skilled in the art and presented, for example, in the document Unsupervised Learning of Depth and Ego-Motion from Video by Tinghui Zhou, Matthew Brown, Noah Snavely and David G. Lowe published on August 1, 2017.
[0129] In an operation 33, second depths associated with a third set of pixels of the first image are predicted from a set of pixels of the third image corresponding to the third set of pixels of the first image and the third set of pixels of the first image by the monoscopic vision system, such an operation 33 is also described in the previously cited document.
[0130] It is possible that a pixel in the first image does not find a corresponding pixel in the third image. This phenomenon is explained by the fact that areas of the first image may be occluded in the third image. Indeed, the difference from the point of view of the first camera 11 at the acquisition time instants tl, t2 does not allow the first camera 11 to see all the elements of the scene. An object present in the scene can mask a second object of the scene, the second object being visible from the point of view of the first camera 11 at the acquisition time instant tl but being masked by an obstacle from the point of view of the first camera 11 at the acquisition time instant t2.
[0131] In an operation 34, pixels of the first image are projected into the three-dimensional scene in the form of a second set of points as a function of the second depths. The calculation of the spatial position of a point P, corresponding to the projection of a pixel Pt of the first image, is done using the following function:
[0132] [Math. 10] p.^pjK'.Djpj)
[0133] with: - K an intrinsic matrix of the camera 11 associated with the projection of a point in space with 3-dimensional coordinates into the image with 2-dimensional coordinates; - 0 a backprojection function in the scene of a pixel Pt according to its depth; - is a second depth of the pixel Pt of the first image predicted by the monoscopic vision system.
[0134] In an operation 35, a fifth image is reconstructed by reprojecting each point of the second set of points as a function of intrinsic parameters of the camera 11 and data representative of the movement of the first camera 11 to generate a fifth image.
[0135] The coordinates of the points Pms projected into the third image from the points P, in the scene are defined by the following formula:
[0136] [Math. 11] Pms = ^[^^Pj] )
[0137] or:
[0138] [Math. 12] p„»="(kMpF'd.J])
[0139] with: - 71 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector; - £ an intrinsic matrix of the camera 11 associated with the projection of a point in space with 3-dimensional coordinates into the image with 2-di- coordinates measurements; - T a displacement matrix between the positions of the camera 11 at the second acquisition time instant t2 and at the first acquisition time instant tl; - 0 a backprojection function in the scene of a pixel Pt according to its depth; - Dtn^P ) is a second depth of the pixel Pt of the first image predicted by the monoscopic vision system.
[0140] When a plurality of points of the second set of points project onto the same pixel of the fifth image, then a point of the plurality of points having the smallest depth is associated with this same pixel.
[0141] The other point(s) of the plurality of points are selected to generate a second visibility mask representative of a fourth set of pixels of the first image not visible in the third image, the pixels of the first image referred to as non-visible being those associated with the points not selected during the projection into the fifth image.
[0142] At the output of operation 35, a second visibility mask representative of pixels visible in the first image and not visible in the third image is determined from the fifth image, a pixel being defined as not visible when none of the points of the set of points projects onto the pixel of the fifth image.
[0143] As presented previously, according to a variant, the value / (xy) of a pixel (RGB or TSL for example) of the fifth image at the coordinates (x, y) can be defined as a function of the values p J of the projected points P, around this pixel. The value of a pixel is, for example, defined as a function of the values of the projected points weighted by the distance separating the projected points from the pixel.
[0144] The monoscopic vision system is not capable of measuring depth with image reconstruction for pixels associated with a moving object in the scene. Therefore, a dynamic object mask must be determined for the monoscopic vision system.
[0145] In an operation 36, a mask of the dynamic objects associated with the first image and representative of a fifth set of pixels of the first image associated with a moving object in the scene is determined by comparing positions of the points of the second set of points and points representative of a set of pixels of the third image projected into the three-dimensional scene in the form of a third set of points as a function of second depths predicted for the set of pixels of the third image.
[0146] The mask of dynamic objects is defined using the following formulas:
[0147] [Math. 13] H(pJ = ) "4>(p t Pt m (P t ) ) ]
[0148] [Math. 14] M d (pJ = [-e^lM^X
[0149] [Math. 15] 4>(p t Am(p t ) ) =D tm (p t )K -1 h(P t )
[0150] with: ■ Mh[^p J is the mask for the static environment applied to the pixel Pt of the first image; - Vm(jP ) is representative of the visibility of the pixel Pt by the monoscopic vision system; - is the displacement matrix of the first camera between the first acquisition time instant tl and the second acquisition time instant t2; - $ is the backprojection of a pixel P with its corresponding depth; ■ ) cst 'a second depth of a source pixel Ps of the third image predicted by the monoscopic vision system; ■ Dfm(p ) is second the depth of a target pixel Pt of the first image predicted by the monoscopic vision system; ■ Md( p ) is the mask for the dynamic environment applied to pixel Pt of the first image; - K is the intrinsic matrix of the first camera 11; - || || ^ is the squared error (L2 norm); ■ h{p ) transforms the pixel vector Pt to the homogeneous coordinate by adding a dimension to the pixel coordinate vector Pt to allow matrix multiplication; and - where one corresponds to a predefined hyperparameter based on a type of road environment of said scene.
[0151] The hypothesis of Mh is that static points in 3D space do not change their positions during the time separating two consecutive images. The mask Md is constructed by an exponential function with a hyperparameter a to find a good criterion for Mh. The static character of the objects is thus considered in a relative manner, the weighting being implemented via the exponential function.
[0152] It is noted here that for the monoscopic vision system, the previously described function allowing the calculation of the photometric error must be multiplied bet.Mrf(pJ-
[0153] According to an exemplary embodiment, the second depths are predicted by a second convolutional neural network, the latter being for example trained in a self-supervised manner. In this embodiment, the self-supervised learning of the convolutional neural network associated with the monoscopic vision system is carried out in a manner similar to that used for the self-supervised learning of the convolutional neural network associated with the stereo system.
[0154] Such a convolutional neural network is known to those skilled in the art, such as for example: GCNet, PSMNet, PWCNet.
[0155] In an operation 37, to find the correct values of the parameters (called “weights” in English) of a chosen convolutional neural network, the fifth image, called the target image (in English “target”) is compared to the third image, called the “source” image pixel by pixel in order to determine a second reconstruction error.
[0156] The second convolutional neural network is trained to minimize the various reconstruction errors calculated by loss functions, such as the following functions. The performance of the convolutional neural network is thus evaluated and its parameters optimized.
[0157] A loss function is based on the photometric error. Once the source image is reconstructed from the first image and the disparities calculated, it is compared to the true source image and defined as follows:
[0158] [Math. 16] L m (O) = E Pf ( v m (p t , O) ■ s(l(pj, î m (p f O) ) )
[0159] with: - Lm is the photometric loss function of the monoscopic vision system; - the index m indicates “monoscopic”; - Vm(p ' is the visibility by the monoscopic vision system of the reconstructed pixel; - J is the value of pixel Pt in the target image; - îm(p is the value of the reconstructed Pixc' for the monoscopic vision system defined from the pixel Pt and O; - O is the second depth Dtm of a pixel Pt predicted by the monoscopic vision system;
[0160] The function ' is the photometric error defined as follows:
[0161] [Math. 17] s(l(p), î(p) ) = ( 1-a) ■ |I(p)-î(p) | +a- (l-|SSIM(l(p),î(p)))
[0162] with: - f(p) is the value of pixel P in the target image, here the third image; - is the value P*XC1 in the reconstructed image, here the fifth image; - SSIM (from the English “structural similarity index measure”) is a function which takes into account a local structure; and - a is a weighting factor depending in particular on the type of environment.
[0163] The second loss function is generally used to deal with edge-aware smoothness and is defined as follows:
[0164] [Math. 18] ^smooth(O. W, o) |V»o(p t ) |e-fWrlpj i)
[0165] with: - O is the second depth Dtm of a pixel Pt predicted by the monoscopic vision system; - W is a parameter matrix; - 0 is the order of a smoothing gradient; - an L1 norm of second-order depth gradients is calculated with W =1, and0 =2; - x and 3 are the dimensions of the first image; - is an environment-dependent hyperparameter; and ■ ft( p ) is the value of pixel Pt in the target image.
[0166] Thus, using these loss functions, parameters of the second convolutional neural network are defined. The result at the output of the calculation is a prediction of a second depth D[m by the monoscopic vision system associated with at least one pixel of the third set of pixels of the first image.
[0167] The prediction of the first depths by the stereo vision system is not accurate in occluded areas. Therefore, it is possible to use the monoscopic vision system to supervise the stereoscopic vision system.
[0168] In an operation 38, a first automatic learning of the stereoscopic vision system under supervision of the monoscopic vision system is carried out by minimizing a first calculation error between the first and second depths as a function of the first visibility mask, second visibility mask and dynamic object mask.
[0169] Taking into account the second visibility mask and the mask of dynamic objects thus makes it possible to exploit the depths calculated for the vision system. monoscopic using them as annotated values during supervised training of the stereo system. Thus, the parameters of the first convolutional neural network for calculating the first depths can be adjusted during operation 38 using the loss function defined as:
[0170] [Math. 19] i^=iJv4p3[i^(pJ](ia^(p,)) ||D tm (p t )-D (p) It ) \ ts 2 /
[0171] with: - Lnr^s corresponds to a loss of recalibration of the second depth compared to the first depth; - Pt is a pixel of the first image; - ) is information on the visibility of the pixel Pt by the stereo system; - ) is information on the visibility of the pixel Pt by the mono vision system scopic; - ) is motion information obtained from the mask of dy objects namics; - Dlm is the second depth associated with the pixel Pt; and - Dts is the first depth associated with the pixel Pt.
[0172] Thus, the first depth calculations are refined, the second depths corresponding to the annotations allowing the learning of the stereoscopic vision system.
[0173] The prediction of the depth by the monoscopic vision system is not of metric precision. It is therefore useful, in order to improve the relevance of the second calculated depths, to set up in an operation 39 a second supervised automatic learning (in English “knowledge distillation”) of the second convolutional neural network making it possible to define the second depth via the monoscopic vision system.
[0174] According to an exemplary embodiment, the second convolutional neural network making it possible to calculate the second depth is trained under the supervision of the stereo system so as to minimize the calculation error defined by the following loss function:
[0175] [Math.20] U™ = Ep t ( V S ( P1 ) He D tm (p t ) -DJpJ II j
[0176] with: - L^m corresponds to a loss of recalibration of the first depth compared to the second depth; - Pt is a pixel of the first image; - VA(p) is information on the visibility of pixel Pt by the stereo system; - Dtin is the second depth calculated for pixel Pt; and - Dts is the first depth calculated for pixel Pt.
[0177] Thus, the second depth calculations are refined, the first depths corresponding to the annotations allowing the learning of the monoscopic vision system.
[0178] If the ADAS uses the first or second depths as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.
[0179] [Fig.4] illustrates a flowchart of the different steps of a method 3 for determining a depth by a stereoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting example of the present invention. The method is for example implemented by a device on board the first vehicle 10 or by the device 4 of [Fig.2].
[0180] In a first step 21, first data representative of a first image acquired by a first camera 11 of said set of cameras at a first acquisition time instant t1 are received.
[0181] In a second step 22, second data representative of a second image acquired by a second camera 12 of the set of cameras at the same first acquisition time instant t1 are received.
[0182] In a third step 23, first depths associated with a first set of pixels of the first image are predicted.
[0183] In a fourth step 24, the pixels of the first set of pixels are projected into the three-dimensional scene.
[0184] In a step 25, a first visibility mask is determined by reconstructing a fourth image by reprojecting each point of the first set of points as a function of intrinsic parameters of the first camera 11 and extrinsic parameters of the stereoscopic vision system.
[0185] In a step 31, third data representative of a third image acquired by the first camera (11) at a second acquisition time instant prior to the first time instant of acquisition of the first image are received.
[0186] In a step 32, data representative of a movement of the first camera 11 between the first time instant and the second time instant are determined as a function of the first and third images.
[0187] In a step 33, second depths associated with a third set of pixels of the first image are predicted.
[0188] In a step 34, the third set of pixels of the first image is projected into the three-dimensional scene in the form of a second set of points as a function of the second depths.
[0189] In a step 35, a second visibility mask is determined by reconstructing a fifth image by reprojecting each point of the second set of points as a function of intrinsic parameters of the first camera and data representative of the movement of the first camera 11.
[0190] In a step 36, a mask of dynamic objects associated with the first image and representative of a fifth set of pixels of the first image associated with at least one moving object in the scene is determined by comparing positions of the points of the second set of points and points representative of a set of pixels of the third image projected into the three-dimensional scene in the form of a third set of points.
[0191] In a step 38, the stereoscopic vision system is learned under supervision of the monoscopic vision system.
[0192] According to a variant, the variants and examples of the operations described in relation to Figures 1 and 3 apply to the steps of the method of [Fig.4].
[0193] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining a depth by a stereoscopic vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.
[0194] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.2].
Claims
1. Claims Method for determining a depth by a stereoscopic vision system on board a vehicle (10), the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method being characterized in that it comprises the following steps: - reception (21, 22) of first and second data respectively representative of a first and second image acquired by respectively a first and second camera (11, 12) of said set of cameras at the same first acquisition time instant; - prediction (23) of first depths associated with a first set of pixels of the first image as a function of disparity values determined from said first set of pixels of the first image and a set of pixels of the second image corresponding to said first set of pixels of the first image; - projection (24) into the three-dimensional scene of said first set of pixels of said first image in the form of a first set of points as a function of said first depths; - determination (25) of a first visibility mask by reconstruction of a fourth image by reprojection of each point of said first set of points as a function of intrinsic parameters of the first camera (11) and extrinsic parameters of said stereoscopic vision system, when a first plurality of points of said first set of points are projected onto the same pixel of said fourth image, then a point of said first plurality of points having the smallest depth is associated with said same pixel, the other point(s) of said first plurality of points being selected to generate a first visibility mask representative of a second set of pixels of said first image not visible in said second image; - reception (31) of third data representative of a third image acquired by said first camera (11) at a second acquisition time instant prior to said first acquisition time instant of said first image, said first camera (11) forming a monoscopic vision system; - determination (32) of data representative of a movement of the first camera (11) between the first time instant and the second time instant as a function of said first and third images; - prediction (33) of second depths associated with a third set of pixels of the first image from said third set of pixels of the first image and a set of pixels of the third image corresponding to said third set of pixels of the first image; - projection (34) into the three-dimensional scene of said third set of pixels of said first image in the form of a second set of points as a function of said second depths; - determination (35) of a second visibility mask by reconstruction of a fifth image by reprojection of each point of said second set of points as a function of the intrinsic parameters of the first camera (11) and of said data representative of the movement of the first camera (11), when a second plurality of points of said second set of points are projected onto the same pixel of said fifth image, then a point of said second plurality of points having the smallest depth is associated with said same pixel, the other point(s) of said second plurality of points being selected to generate a second visibility mask representative of a fourth set of pixels of said first image not visible in said third image; - determination (36) of a mask of dynamic objects associated with said first image and representative of a fifth set of pixels of the first image associated with at least one moving object in the three-dimensional scene by comparison of positions of said points of said second set of points and points representative of a set of pixels of said third image projected into the three-dimensional scene in the form of a third set of points as a function of second depths predicted for said set of pixels of the third image; - first automatic learning (38) of said stereoscopic vision system under supervision of said monoscopic vision system by minimizing a first calculation error between said first and second predicted depths for said first image as a function of said first visibility mask, second visibility mask and dynamic object mask.
2. Method according to claim 1, for which said first learning is obtained, in addition, by minimizing the following function representative of the first calculation error: Lm-s = Ep,( Vm(pt) [ 1-Md(pt) ] ( 1-Vs(pt) ) Il -DjpJ II j with: - corresponds to a loss of registration of the second depth with respect to the first depth; - Pt is a pixel of the first image; - V,\pj is information on the visibility of the pixel Pr by the stereoscopic vision system; ' Vn^P ) is information on the visibility of the pixel Pt by the monoscopic vision system; ■ Md( p ) is information on movement obtained from the mask of the dynamic objects; - ( P ) is 'aci'te second depth predicted for the pixel Pt; and ■ DfS p ) is said first depth predicted for the pixel Pt.
3. A method according to claim 1 or 2, further comprising a second automatic learning (39) of said monoscopic vision system under supervision of said stereoscopic vision system by minimizing a second calculation error between said first and second predicted depths for the first image as a function of said first visibility mask.
4. Method according to claim 3, for which said second automatic learning is obtained by minimizing the following function representative of said second calculation error: Ls-m = Ep^Vs(pt) || Dtm(pj-DtJpt) || j with: - Ls_*m corresponds to a loss of registration of the first depth with respect to the second depth; - Pt is a pixel of the first image; - V*(p ) is information on the visibility of the pixel Pt by the stereoscopic vision system; - Dtm is the second depth predicted for the pixel Pt; and - Dts is the first predicted depth for pixel Pt.
5. Method according to one of claims 1 to 4, for which said first depths are predicted via a first convolutional neural network with self-supervised learning, the method further comprising the step (26) of obtaining a first reconstruction error by comparing said second and fourth images, said first convolutional neural network being learned in a self-supervised manner as a function of said first reconstruction error.
6. Method according to one of claims 1 to 5, for which said second depths are predicted via a second convolutional neural network with self-supervised learning, the method further comprising the step of obtaining (37) a second reconstruction error by comparing said third and fifth images, said second convolutional neural network being learned in a self-supervised manner as a function of said second reconstruction error.
7. Method according to one of claims 5 and 6, for which said first and second reconstruction errors are respectively obtained by the following loss function: s(l(p),î(p)) = (1-a) ■ |l(p)-î(p)I +«■ (1-4SSIM(i(p),Î(p))) with: - I(p) a value of the pixel P in respectively the second image and the third image; - a value of the pixel P in the reconstructed image corresponding respectively to the fourth image and to the fifth image; - SSIM corresponds to a function which takes into account a local structure; and - a corresponds to a weighting factor depending on a type of road environment in which the vehicle (10) is traveling.
8. Method according to one of claims 1 to 7, for which the dynamic object mask is defined by the following formulas: Mb(p,) = V^pJ [T^(pjDsm(pJ ) -4>(pjDIm(pt) ) ] ■ Mb(p ) corresponds to a static object mask applied to a target pixel Pt of the first image; - Vm(jP ) is representative of a visibility of the target pixel Pt of the first image by the monoscopic vision system; - corresponds to a displacement matrix of the camera (11) between the second acquisition time instant and the first acquisition time instant; - ÿ corresponds to a backprojection of a pixel P with its corresponding depth D(p); ■ ( p ) is the second depth of a source pixel Ps of the second image calculated for the monoscopic vision system; - Dtm^P ) is 'a First depth of the target pixel Pt of the first image calculated for the monoscopic vision system; - ^d^P ) corresponds to said dynamic object mask applied to the target pixel Pt of the first image; - K corresponds to an intrinsic matrix of the camera (11); - || || is a squared error; ■ h(pj transforms a vector of coordinates of the target pixel Pt into a homogeneous coordinate by adding a dimension to said vector of coordinates of the target pixel Pt; and - 5 corresponds to a predefined hyperparameter based on a type of road environment in which the vehicle circulates (10).
9. Device (4) for determining a depth by a stereoscopic vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 8.
10. Vehicle (10) comprising the device (4) according to claim 9.