Method and device for determining the depth of an object by a self-supervised vision system.

Abstract

Method or device for determining a depth of at least one object by a stereoscopic vision system on board a vehicle comprising at least two cameras, the method comprising receiving (21, 22) first and second data representative of first and second images acquired by the cameras, determining (23, 24) boxes encompassing each object present in the first and second images. A zero value is assigned (25, 26) to each pixel of the two images not included in a bounding box. Depths associated with the pixels of the first image are predicted (27) as a function of disparities, a disparity associated with the other pixels being predicted by a convolutional neural network learned in a self-supervised manner by minimizing a reconstruction error of a set of training images and a disparity equal to zero being assigned to each pixel not included in a bounding box. Figure for abstract: Figure 2

Description

Title of the invention: Method and device for determining the depth of an object by a self-supervised vision system. Technical field

[0001] The present invention relates to methods and devices for determining the depth of an object by means of a stereoscopic vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring such a depth. The present invention also relates to a method and a device for controlling one or more AD AS systems on board a vehicle from the determined depth. Technological background

[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.

[0005] The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by a vision system and its speed of execution. Summary of the present invention

[0006] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0007] Another object of the present invention is to improve the quality of the data from these cameras.

[0008] Another object of the present invention is to improve road safety.

[0009] According to a first aspect, the present invention relates to a method for determining a depth of a set of objects by a stereoscopic vision system on board a vehicle, the set of objects comprising at least one object, the stereoscopic vision system comprising a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene comprising the set of objects from a different point of view, the method being characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of the set of cameras at the same acquisition time instant; - determining for each of the objects in the set a first bounding box in the first image and a second bounding box in the second image to form a first set of first bounding boxes in the first image and a second set of second bounding boxes in the second image, a first set of pixels of the first image being associated with the first set of first bounding boxes and a second set of pixels of the second image being associated with the second set of second bounding boxes; - assigning a zero value to each pixel of a third set of pixels in the first image, the third set of pixels being the complement of the first set of pixels in the first image, and a zero value to each pixel of a fourth set of pixels in the second image, the fourth set of pixels being the complement of the second set of pixels in the second image; and - prediction of depths associated with the pixels of the first image based on disparities, a disparity associated with a pixel of the first set of pixels being predicted by a convolutional neural network learned in a self-supervised manner by minimizing a reconstruction error of a set of training images reconstructed from the first image and the disparities, the reconstruction error being determined by comparing the set of training images to the second image, and a disparity equal to zero being assigned to each pixel in the third set of pixels.

[0010] Such a method makes it possible to precisely determine a depth of an object in the field of vision of the cameras of a vehicle without having to determine a depth for each pixel of an image acquired by one of the cameras.

[0011] According to a variant of the method, the set of training images comprises two training images reconstructed from two disparities predicted for the same pixel of the first set of pixels, a first training image being reconstructed from a first disparity and a second training image being reconstructed from a second disparity.

[0012] The two most probable disparities are then taken into account for a more exact determination of a depth.

[0013] According to another variant of the method, the first depths are predicted by: D “U,)= / ^ avec: • Dt^p ) a predicted depth for a pixel Pt of the first image, • f a focal length of the first camera, • B a distance separating the first and second cameras, • d(p ) a first disparity associated with the pixel Pt.

[0014] A depth is thus determined using the disparities and a known dimension associated with a camera of the vision system.

[0015] According to another variant, the method comprises a phase of learning the convolutional neural network comprising the following steps: - reconstructing a set of training images from the first image and the disparities associated with the first set of pixels, a pixel of a training image of the set of training images being determined being defined by: • Pss a pixel of a training image, • Pt one pixel of the first image, * d(p ) a disparity associated with the pixel Pt of the first image; - determining the reconstruction error by comparing the training image set to the second image; and - minimization of reconstruction error.

[0016] Self-supervised learning allows for the elimination of annotated data while benefiting from a large amount of data received from the vision system itself. The use of bounding boxes allows for faster learning.

[0017] According to an additional variant of the method, the reconstruction error is obtained by a first loss function as follows: s(l(p),î(p)) = ( 1-a) ■ |I(p)'î(p)|+a- (14SSIM(l(p),î(p))) with : - I(p) a value of the pixel P in the second image; - a value of the pixel P in a training image; - SSIM corresponds to a function that compares a local structure of two images; and - a corresponds to a weighting factor depending on a type of road environment in which the vehicle is traveling.

[0018] According to yet another variant of the method, the convolutional neural network is learned in a self-supervised manner by minimizing a second following loss function: Lsmwth ( o, w, o) = w( P[ ) |v;;o( Pt ) : - O is a depth Dts of a pixel Pt predicted by the stereoscopic vision system, - W is a parameter matrix, - 0 is the order of a smoothing gradient, - an L1 norm of the second-order depth gradients is calculated with W =1 and° =2, - x and 2 are the dimensions of the first image, - / ) is a hyperparameter dependent on a type of road environment in which the vehicle travels, and ■ P} is the value of pixel Pt in the first image.

[0019] According to a second aspect, the present invention relates to a device for determining a depth of at least one object by a stereoscopic vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0020] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0021] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0022] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0023] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0024] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.

[0025] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0026] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0027] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 3, in which:

[0028] [Fig-1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

[0029] [Fig.2] illustrates a flowchart of the different steps of a method for determining a depth of at least one object by a stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0030] [Fig.3] schematically illustrates a device configured for determining a depth of at least one object by a stereoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0031] A method and a device for determining the depth of at least one object by means of a stereoscopic vision system on board a vehicle will now be described in the following with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the des- description that will follow.

[0032] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth of at least one object by a stereoscopic vision system on board a vehicle is for example implemented by a computer of the on-board system of the vehicle controlling this stereoscopic vision system.

[0033] The stereoscopic vision system comprises a set of cameras of at least two cameras arranged so as to each acquire an image of a scene comprising the object from a different point of view.

[0034] For this purpose, the method for determining a depth of at least one object by a stereoscopic vision system on board a vehicle, comprises the reception of first and second data respectively representative of a first and second image acquired by respectively a first and second camera of a set of cameras at the same acquisition time instant.

[0035] Bounding boxes are determined for all objects present in the first and second images, forming a first set of first bounding boxes in the first image and a second set of second bounding boxes in the second image.

[0036] A first set of pixels of the first image is associated with the first set of first bounding boxes and a second set of pixels of the second image is associated with the second set of second bounding boxes.

[0037] A zero value is then assigned to each pixel of a third set of pixels in the first image, the third set of pixels being the complement of the first set of pixels in the first image.

[0038] Similarly, a zero value is assigned to each pixel of a fourth set of pixels in the second image, the fourth set of pixels being the complement of the second set of pixels in the second image.

[0039] Depths associated with pixels of the first image are then predicted based on disparities, a disparity associated with a pixel of the first set of pixels being predicted by a convolutional neural network learned in a self-supervised manner by minimizing a reconstruction error of a set of training images reconstructed from the first image and the disparities, the reconstruction error being determined by comparing the set of training images to the second image and a disparity equal to zero being assigned to each pixel of the third set of pixels.

[0040] [Fig. 1] schematically illustrates a stereoscopic vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0041] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0042] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0043] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms the stereoscopic vision system. Two cameras 11 and 12 are illustrated in [Fig.l]. The present invention is however not limited to a stereoscopic vision system comprising two cameras but extends to any stereoscopic vision system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.

[0044] The two cameras 11, 12 have known intrinsic parameters. These parameters consist in particular of: - the focal length f of the first camera 11; - the focal length f' of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.

[0045] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.

[0046] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account or that their correction is processed at the time of image acquisition.

[0047] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view being for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

[0048] A first reference point is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is that of the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0049] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.

[0050] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.

[0051] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the "baseline" must reach 60cm for the cameras commonly used in this field.

[0052] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0053] According to an exemplary embodiment, the directions C1, C2 of the optical axes represent indicative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0054] According to another exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented parallel.

[0055] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.

[0056] The images acquired by the cameras 11, 12 at a given acquisition time instant tl are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of the objects in the observed scene in the form, for example, of RGB (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness) colorimetric coordinates.

[0057] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. In this scene there is at least one object, for example: - a building; - road infrastructure; - another stationary user, for example a parked vehicle; and / or - another mobile user, for example another vehicle, a cyclist or a moving pedestrian.

[0058] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0059] In a first operation, the computer receives first data representative of a first image acquired by the first camera 11 of the set of cameras at a first acquisition time instant t1.

[0060] In a second operation, the computer receives second data representative of a second image acquired by the second camera 12 of the set of cameras at the same first acquisition time instant t1.

[0061] The two images received correspond to two views of the same scene taking place around the vehicle 10 at the same given first acquisition time instant tl.

[0062] In order to facilitate the analysis of the two received images, the first and second images are rectified according to a method known to those skilled in the art. Such a method is described, for example, in “Projective Rectification of Non-Calibrated Infrared Stereo Images with Global Consideration of Distortion Minimization” by Benoit Ducarouge, Thierry Sentenac, Florian Bugarin and Michel Devy of July 16, 2009.

[0063] The rectification method consists of reorienting the epipolar lines so that they are parallel with the horizontal axis of the image. This method is described by a transformation which projects the epipoles to infinity and whose corresponding points are necessarily on the same ordinate.

[0064] A rectification algorithm consists, for example, of 4 steps: - Rotate (virtually) the first camera 11 so that the epipole goes to infinity along the horizontal axis of the reference frame associated with it; - Apply the same rotation to the second camera 12 to end up in the initial geometric configuration; - Rotate the second camera 12 by the rotation associated with the rotation matrix 'R', corresponding to the extrinsic parameter of the starting stereoscopic vision system; - Adjust the scale in both camera markers.

[0065] It should be noted that rectification simplifies the matching of pixels in stereo images, i.e. images obtained by a stereoscopic vision system. The corresponding pixel in the second image to a pixel in the first image (and vice versa) is positioned on the same line. From the knowledge of the epipolar geometry and therefore of a fundamental matrix of the stereoscopic vision system, the objective is then to determine a pair of projective transformations, called homographies, which reorient the epipolar projections parallel to the lines of the images, therefore to the horizontal axis of the rectified cameras.

[0066] In a third operation, a first bounding box is determined for each of the objects in the set in the first image to form a first set of first bounding boxes in the first image

[0067] A first set of pixels of the first image is then associated with the first set of first bounding boxes.

[0068] Such a bounding box, in English "Bounding Box", is for example defined by the coordinates of a pixel corresponding to a corner or the center of an object and by two dimensions, a width and a height. It thus represents a rectangle in which the object is located.

[0069] The definition of a bounding box is for example described in the document "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors", in French "Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors" by Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao of July 6, 2022.

[0070] In the case of the presence of several objects in the three-dimensional scene, therefore in the first image, as many bounding boxes as detected objects are determined, their size being dependent on the size of the detected objects.

[0071] Similarly, in a fourth operation, a second bounding box is determined for each of the objects in the set in the second image to form a second set of second bounding boxes in the second image.

[0072] A second set of pixels of the second image is then associated with the second set of second bounding boxes.

[0073] In a fifth operation, a zero value is assigned to each pixel of a third set of pixels in the first image, the third set of pixels being the complement of the first set of pixels in the first image.

[0074] Similarly, in a sixth operation, a zero value is assigned to each pixel of a fourth set of pixels in the second image, the fourth set of pixels being the complement of the second set of pixels in the second image.

[0075] Thus, in the first and second images, pixels not in a bounding box are assigned a zero value.

[0076] In a seventh operation, disparities are predicted for the pixels of the first set of pixels, i.e. for the pixels of the first image included in at least one bounding box.

[0077] In an eighth operation, a disparity equal to zero is assigned to each pixel of the third set of pixels of the first image.

[0078] The determination of the disparity is for example carried out according to a method known to those skilled in the art, presented in particular in the document “Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation”, in French “Corréler et exciter: correspondance stéréo en temps réel via l’excitation guider du volume des Coûts”, by Antyanta Bangunharcana, Jae Won Cho, Seokju Lee, In So Kweon, Kyung-Soo Kim and Soohyun Kim dated August 12, 2021.

[0079] A probability distribution of the disparity is determined and a sorting algorithm then selects the two most likely disparities for each pixel in the first image, a first disparity and a second disparity. This sorting requires a lot of machine resources, but is only performed on the pixels in the first set of pixels, i.e., only on the pixels present in bounding boxes.

[0080] Thus, several disparities are predicted for the same pixel of the first set of pixels.

[0081] In order to train the convolutional neural network, in a ninth operation, a first training image is reconstructed from the first predicted disparity for the first set of pixels

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] A pixel of the first training image is defined by: [Math.l] Pss = Pt-dM with : • a pixel from the first training image, • Pt one pixel of the first image, • d(p ) a disparity associated with the pixel Pt of the first image, the first disparity for a pixel of the first set of pixels, a zero disparity for a pixel of the third set of pixels. Still to train the convolutional neural network, in a tenth operation, a second training image is reconstructed from the second disparity determined for the first set of pixels and with zero disparity applied to the third set of pixels. A pixel of the second training image is defined by: [Math.2] P^P^M with : • PiX one pixel of the second training image, • Pt one pixel of the first image, • d(p ) a disparity associated with the pixel Pt of the first image, the second disparity for a pixel of the first set of pixels, a zero disparity for a pixel of the third set of pixels. Thus, the training image set comprises two training images reconstructed from two predicted disparities for the same pixel of the first pixel set, the first training image being reconstructed from the first disparity and the second training image being reconstructed from the second disparity. In an eleventh operation, a reconstruction error is determined for the set of training images. According to an exemplary embodiment, the reconstruction error is obtained for each training image by the following first loss function: [Math.3] s(l(p), î(p) ) = ( 1 -«) ■ |I(p)-î(p)|+a- (14SSIM(l(p),î(p))) with : - I(p) a value of the pixel P in the second image; ' a value of pixel P in a training image; - SSIM corresponds to a function which compares a local structure of two images; and - a corresponds to a weighting factor depending on a type of road environment in which the vehicle is traveling (10).

[0094] SSIM is a function that compares a local structure of two images by measuring the structural similarity between the two images rather than a pixel-to-pixel difference.

[0095] Thus, a first reconstruction error El is obtained for the first training image and a second reconstruction error E2 is obtained for the second training image.

[0096] A global reconstruction error E is then obtained by weighted addition of the first and second reconstruction errors. Thus, the global reconstruction error is defined by: E = ax El + (1- a) x E2 with 0.5 <a<l.

[0097] For example, for a = 0.7, the first reconstruction error obtained for the first training image determined using the first disparities corresponding to the most probable disparities weighs more in the calculation of the overall reconstruction error than the second reconstruction error.

[0098] In a twelfth operation, a reconstruction error is minimized for all training images: the global reconstruction error. Parameters of the convolutional neural network are adjusted to minimize the global reconstruction error during training. The convolutional neural network is then learned in a self-supervised manner, without resorting to the use of annotated data.

[0099] Thus, a disparity associated with a pixel of the first set of pixels is predicted by a convolutional neural network learned in a self-supervised manner by minimizing the reconstruction error of the set of training images reconstructed from the first image and the disparities, the reconstruction error being determined by comparing the set of training images to the second image.

[0100] In a thirteenth operation, depths associated with said first set of pixels are predicted based on disparities determined from the first set of pixels and the second set of pixels.

[0101] The disparities used to define the depth are the first and second disparities. A global disparity for a pixel pt of the first set of pixels is for example defined as a weighted sum of the first and second disparities determined for this same pixel of the first set of pixels, for example using the same weighting coefficient a as that used for the global reconstruction error: Overall disparity (pt) = ax first disparity (pt) + (1- a) x second disparity (pt) with 0.5 <a<l.

[0102] Thus the depths are predicted by:

[0103] [Math.4] DJS{p— fx —p---

[0104] with: • p) a predicted depth for a pixel Pt of the first image, • f a focal length of said first camera (11), • j a global disparity associated with the pixel Pt.

[0105] In a fourteenth optional operation, the convolutional neural network is learned in a self-supervised manner by minimizing a second loss function as follows:

[0106] [Math.5] L™«|.(O, wo) )

[0107] with: - O is a depth Dts of a pixel Pt predicted by the stereoscopic vision system, - W is a parameter matrix, - ° is the order of a smoothing gradient, - an L1 norm of second-order depth gradients is calculated with W =1 and ° =2, - x and are the dimensions of the first image, - / ? is a hyperparameter dependent on a type of road environment in which the vehicle is traveling (10), and - / p J is the value of pixel Pt in the first image.

[0108] This second loss function makes it possible to process the discontinuity of the depth at the edge of the objects (in English “Edge Aware Smoothness”).

[0109] Thus, the determination of the depth of an object is made reliable using a method allowing rapid self-supervision of the convolutional neural network making it possible to determine disparities for pixels associated with this same object.

[0110] If an ADAS uses the depths as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.

[0111] [Fig.2] illustrates a flowchart of the different steps of a method 2 for determining a depth of at least one object by a stereoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention. The method is for example implemented by a device on board the first vehicle 10 or by the device 3 of [Fig.3].

[0112] In a step 21, first data representative of a first image acquired by a first camera 11 of the set of cameras at an acquisition time instant are received.

[0113] In a step 22, second data representative of a second image acquired by a second camera 12 of the set of cameras at the same acquisition time instant are received.

[0114] In a step 23, a first bounding box is determined for each of the objects in the first image and a first set of first bounding boxes is formed in the first image from each first bounding box.

[0115] A first set of pixels of the first image is associated with the first set of first bounding boxes.

[0116] In a step 24, a second bounding box is determined for each of the objects in the second image and a second set of second bounding boxes is formed in the second image from each second bounding box.

[0117] A second set of pixels of the second image is associated with the second set of second bounding boxes.

[0118] In a step 25, a zero value is assigned to each pixel of a third set of pixels of the first image, the third set of pixels being the complement of the first set of pixels in the first image.

[0119] In a step 26, a zero value is assigned to each pixel of a fourth set of pixels of the second image, the fourth set of pixels being the complement of the second set of pixels in the second image.

[0120] In a step 27, a disparity associated with a pixel of the first set of pixels is predicted by a convolutional neural network learned in a self-supervised manner by minimizing a reconstruction error of a set of training images reconstructed from the first image and the disparities, the reconstruction error being determined by comparing the set of training images to the second image.

[0121] A disparity equal to zero is assigned to each pixel of the third set of pixels.

[0122] Depths associated with the pixels of the first image are then predicted in function of the disparities associated with the pixels of the first image.

[0123] According to a variant, the variants and examples of the operations described in relation to [Fig.l] apply to the steps of the method of [Fig.2].

[0124] [Fig. 3] schematically illustrates a device 3 configured for determining a depth of at least one object by a stereoscopic vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 3 corresponds for example to a device on board the first vehicle 10, for example a computer.

[0125] The device 3 is for example configured for the implementation of the operations described with regard to [Fig.l] or the steps described with regard to [Fig.2]. Examples of such a device 3 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 3, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 3 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0126] The device 3 comprises one (or more) processor(s) 30 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 3. The processor 30 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 3 further comprises at least one memory 31 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0127] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the memory 31.

[0128] According to various particular and non-limiting embodiments, the device 3 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0129] According to a particular and non-limiting exemplary embodiment, the device 3 comprises a block 32 of interface elements for communicating with external devices. interface elements of block 32 include one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0130] According to another particular and non-limiting exemplary embodiment, the device 3 comprises a communication interface 33 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 330. The communication interface 33 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 330. The communication interface 33 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0131] According to a particular and non-limiting exemplary embodiment, the device 3 can provide output signals to one or more external devices, such as a display screen 340, touch-sensitive or not, one or more speakers 350 and / or other peripherals 360 (projection system) via the output interfaces 34, 35, 36 respectively. According to a variant, one or other of the external devices is integrated into the device 3.

[0132] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining a depth of at least one object by a stereoscopic vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0133] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 3 of [Fig.3].

Claims

1. Claims Method for determining a depth of a set of objects by a stereoscopic vision system on board a vehicle (10), said set of objects comprising at least one object, the stereoscopic vision system comprising a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a three-dimensional scene comprising said set of objects from a different point of view, said method being characterized in that it comprises the following steps: - reception (21, 22) of first and second data respectively representative of a first and second image acquired by respectively a first (11) and second (12) camera of said set of cameras at the same acquisition time instant; - determining (23, 24) for each of the objects of said set a first bounding box in said first image and a second bounding box in said second image to form a first set of first bounding boxes in the first image and a second set of second bounding boxes in the second image, a first set of pixels of the first image being associated with said first set of first bounding boxes and a second set of pixels of the second image being associated with said second set of second bounding boxes; - assigning (25, 26) a zero value to each pixel of a third set of pixels of the first image, said third set of pixels being the complement of said first set of pixels in the first image, and a zero value to each pixel of a fourth set of pixels of the second image, said fourth set of pixels being the complement of said second set of pixels in the second image; and - prediction (27) of depths associated with the pixels of the first image as a function of disparities, a disparity associated with a pixel of said first set of pixels being predicted by a convolutional neural network learned in a self-supervised manner by minimizing a reconstruction error of a set of training images reconstructed from said first image and said disparities, said reconstruction error being determined by comparing said set of training images to said second image, and a disparity equal to zero being assigned to each pixel of said third set of pixels.

2. The method of claim 1, wherein said set of training images comprises two training images reconstructed from two predicted disparities for the same pixel of said first set of pixels, a first training image being reconstructed from a first disparity and a second training image being reconstructed from a second disparity.

3. Method according to claim 1 or 2, for which said depths are predicted by: D^Pt)=f with: • p) a predicted depth for a pixel Pi of the first image, • f a focal length of said first camera (11), • B a distance separating the first (11) and the second (12) cameras, • j a disparity associated with the pixel Pt.

4. Method according to one of claims 1 to 3, which comprises a phase of training said convolutional neural network comprising the following steps: - reconstruction of said set of training images from said first image and the disparities associated with said first set of pixels, a pixel of a training image of said set of training images being determined by: p^p^M with: • Pxs a pixel of a training image, • Pt a pixel of the first image, • d(p ) a disparity associated with the pixel Pt of the first image; - determination of said reconstruction error by comparing said set of training images to said second image; and - minimization of said reconstruction error.

5. Method according to claim 4, for which said reconstruction error is obtained by a first loss function as follows: s(l(p), î(p) ) = (1-a) ■ jl(p)-î(p)|+a- (14SSïm(i(p),Î(p))) with: - I(p) a value of pixel P in the second image; - a value of pixel P in a training image; - SSIM corresponds to a function which compares a local structure of two images; and - a corresponds to a weighting factor depending on a type of road environment in which the vehicle circulates (10).

6. Method according to one of claims 4 to 5, for which said convolutional neural network is learned in a self-supervised manner by minimizing a second following loss function: I^xbta w. o) =^,^( W(p,) |v3O(pt) |e^»Wl ) with: - 0 is a depth PP of a pixel Pt predicted by the stereoscopic vision system, - W is a parameter matrix, - ° is the order of a smoothing gradient, - an L1 norm of the second-order depth gradients is calculated with W =1 and0 =2, - A and are the dimensions of the first image, - / i is a hyperparameter dependent on a type of road environment in which the vehicle (10) is traveling, and - h^P ) is the value of the pixel Pt in the first image.

7. Computer program product comprising instructions adapted for executing the steps of the method according to one of claims 1 to 6, when the computer program is executed by at least one processor.

8. A computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to one of claims 1 to 6.

9. Device (3) for determining a depth of at least one object by a stereoscopic vision system on board a vehicle (10), said device (3) comprising a memory (31) associated with at least one processor (30) configured for implementing the steps of the method according to any one of claims 1 to 6.

10. Vehicle (10) comprising the device (3) according to claim 9.

Images