Method and device for determining the depth of a pixel of an image by a depth prediction model associated with a vision system on board a vehicle

Abstract

Method or device for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system comprising several cameras. Indeed, the depth prediction model is learned in a learning phase comprising the generation of images (35) from images acquired by the vision system, from determined directions and depths (33, 34) for pixels of the received images and from extrinsic parameters of the vision system, making it possible to determine a first and second error by comparing pixels of the acquired images and generated images and an invisibility mask (36) as well. The depth prediction model is learned (37) by minimizing an error determined from the first and second errors associated with pixels not included in the invisibility mask. Figure for abstract: Figure 3

Description

Title of the invention: Method and device for determining the depth of a pixel of an image by a depth prediction model associated with a vision system on board a vehicle Technical field

[0001] The present invention relates to methods and devices for determining a depth by a vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring a distance separating an object from a vehicle carrying a vision system. Technological background

[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0004] The AD AS embedded in a vehicle are supplied with data obtained one or more on-board sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or possible obstacles present around a vehicle in order, for example: • to adapt the vehicle's lighting according to the presence of other users; • automatically regulate the vehicle speed; • to act on the braking system in the event of a risk of impact with an object.

[0005] In order to have an extended view of the vehicle's environment, i.e. a three-dimensional scene taking place around the vehicle, a wide-angle camera, i.e. a camera with a wide field of vision, is recommended. Indeed, the use of a wide-angle camera has many advantages compared to “standard” cameras: • a wider field of vision allowing a larger part of the scene to be captured, which is particularly important in the context of driving, where it is essential to monitor the environment both to the sides and in front of the vehicle, for example, • greater efficiency in perceiving complex environments, such as intersections, tight turns, parking spaces, etc., by minimizing blind spots and providing a more complete view of driving situations, and • increased safety by more easily detecting obstacles, other vehicles, pedestrians and cyclists in areas adjacent to the vehicle.

[0006] Although wide-angle cameras offer many advantages, they can also introduce distortions into the captured images, and these distortions can lead to certain problems such as: • the distortion of straight lines, for example barrel or pincushion, causing curvature of straight lines in an image acquired by the wide-angle camera, making it difficult to estimate the actual distances between objects, particularly towards the edges of the image, • stretching or compressing objects, especially towards the edges of the image, changing the apparent size of objects, which can be problematic when judging the distance or actual size of objects, • changing the proportions of objects, making them larger or smaller than their actual size and more complex to identify, and • the difficulty of rectifying or correcting distortion in post-processing which can be complex and can lead to a loss of information.

[0007] Thus, the processing of an image acquired by a wide-angle camera requires special processing, in particular because of the strong distortion present in the image. Determining a distance separating the vehicle carrying the camera from an object in the scene is then not achievable with the methods commonly used for standard cameras used in certain vision systems.

[0008] Furthermore, images acquired at the same time instant by a stereoscopic vision system, i.e. a vision system comprising several cameras acquiring images of the same three-dimensional scene, sometimes include occluded areas, i.e. areas visible in an image acquired by a camera having pixels associated with an object of the three-dimensional scene not visible in an image acquired by another camera of the stereoscopic vision system. The prediction of a depth associated with a pixel corresponding to an object of the three-dimensional scene not visible by all the cameras of the stereoscopic vision system is complex and is a source of prediction error. Summary of the present invention

[0009] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0010] Another object of the present invention is to improve the quality of the data resulting from the processing of an image acquired by a vision system, in particular by a depth prediction model implemented by a neural network associated with this stereoscopic vision system.

[0011] Another object of the present invention is to improve road safety, in particular by improving the operational safety of AD AS systems supplied with data obtained from a wide-angle camera.

[0012] According to a first aspect, the present invention relates to a method for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the method being implemented by at least one processor, and being characterized in that the depth prediction model is learned in a learning phase comprising the following steps: - reception of data representative of a first image and a second image acquired by the first camera and the second camera respectively at the same acquisition time instant; - determining a first feature map associated with the first image and a second feature map associated with the second image by a feature extractor; - determination of directions associated with pixels of a first set of pixels of the first image, called first pixels, and of directions associated with pixels of a first set of pixels of the second image, called second pixels, by a direction prediction model from the first image and the second image respectively; - determining depths associated with the first pixels and the second pixels by said depth prediction model from the first image and the second image respectively; - generation of a third image from the first image, the directions and depths associated with the first pixels, extrinsic parameters of the stereoscopic vision system and the direction prediction model and generation of a fourth image from the second image and the directions and depths associated with the second pixels, extrinsic parameters of the stereoscopic vision system- reoscopic and direction prediction model; - determining a first error associated with each first pixel by comparing pixels of the first and fourth images and a second error associated with each second pixel by comparing pixels of the second and third images; - determining an invisibility mask from first coordinates of a second set of pixels of the first image associated with at least one object of the three-dimensional scene not visible in the second image and from second coordinates of a second set of pixels of the second image associated with at least one object of the three-dimensional scene not visible in the first image, a zero value being assigned to each first error and each second error associated with a pixel included in the invisibility mask; - training the depth prediction model by minimizing a loss error determined from the first and second errors, a first error and a second error associated with a pixel included in the invisibility mask being zero.

[0013] Such a depth prediction model thus makes it possible to accurately predict the depth of a pixel of an image acquired by one of the cameras of the stereoscopic vision system, i.e. the distance separating the vehicle carrying the camera from a physical object of the three-dimensional scene associated with this pixel.

[0014] According to a variant of the method, the first and second errors are photometric errors determined by the following function: l^p) = EJ ( i-«) ■ U(p)-Hp)\+a- (i455W7(p)4(p)))] With: • LA p) the first reconstruction error noted LÀpL respectively the second reconstruction error noted ( p ), being a pixel defined by its coordinates in an image, • Kp) a value of the pixel P not included in the invisibility mask in the first image, respectively second image, • 2^ a value of the pixel P not included in the invisibility mask in the fourth image, respectively third image, • SSIM a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0015] According to yet another variant of the method, the loss error is determined by the following function: With : • The loss error, • p) the first reconstruction error for a pixel P of the first image, and • L2 ( p ) the second reconstruction error for a pixel P of the second image corresponding to the pixel P of the first image.

[0016] According to a further variant, the third and fourth images are generated using the following function: p.=DM ) ] ) With : • Ps a pixel of a generated image corresponding to the third image, respectively to the fourth image, • 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K a direction prediction model associated with the second camera, respectively with the first camera, • K' a direction prediction model associated with the first camera, respectively with the second camera, • T an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and • D^p ) is a depth of one pixel Pt of the first image, respectively of the second image, determined by the convolutional neural network.

[0017] According to another variant of the method, the invisibility mask comprises coordinates included in the first coordinates and in the second coordinates.

[0018] According to yet another variant of the method, at least one of the cameras of the stereoscopic vision system is a wide-angle camera.

[0019] According to a further variant of the method, a field of view of the first camera covers at least half of a field of view of the second camera and a field of view of the second camera covers at least half of a field of view of the first camera.

[0020] According to a second aspect, the present invention relates to a device for determining a depth by a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0021] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0022] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0023] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0024] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0025] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.

[0026] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0027] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0028] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 5, in which:

[0029] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting example of embodiment of the present invention;

[0030] [Fig.2] illustrates a flowchart of the different steps of a method for determining a depth of a pixel of an image by a depth prediction model associated with a vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0031] [Fig.3] illustrates a flowchart of the different stages of a learning process of the depth prediction model used in the method of [Fig.2], according to a particular and non-limiting exemplary embodiment of the present invention;

[0032] [Fig.4] schematically illustrates an invisibility mask determined from images received during the learning method of [Fig.3], according to a particular and non-limiting exemplary embodiment of the present invention;

[0033] [Fig.5] schematically illustrates a device configured to determine a depth of a pixel of an image by a neural network associated with a vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0034] A method and a device for determining a depth of a pixel of an image by a neural network associated with a vision system on board a vehicle will now be described in what follows with joint reference to Figures 1 to 5. The same elements are identified with the same reference signs throughout the description which follows.

[0035] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0036] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system comprising several cameras.

[0037] Indeed, the depth prediction model is learned in a learning phase comprising the generation of images from images acquired by the vision system, from directions and depths determined for pixels of the received images and from extrinsic parameters of the vision system, then making it possible to determine an invisibility mask as well as a first and second error by comparison of pixels of the acquired images and of the generated images.

[0038] The depth prediction model is learned by minimizing a loss error determined from the first and second errors associated with pixels not included in the invisibility mask.

[0039] [Fig. 1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0040] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0041] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0042] The vehicle 10 advantageously comprises at least two on-board cameras, a first camera 11 and a second camera 12, configured to acquire images of a three-dimensional scene taking place in the environment of the vehicle 10 from separate observation positions. The first camera 11 and the second camera 12 form a stereoscopic vision system when used together as illustrated in [Fig.l]. The first camera 11 forms a monoscopic vision system when used alone, likewise the second camera 12 forms another monoscopic vision system when used alone. The present invention, however, extends to any vision system comprising at least two cameras, for example 2, 3 or 5 cameras.

[0043] The intrinsic parameters of the first camera 11 characterize the transformation which associates, for an image point, subsequently called “point”, its three-dimensional coordinates in the reference frame of the first camera 11 with the pixel coordinates in an image acquired by the first camera 11. These parameters do not change if the first camera 11 is moved. The intrinsic parameters of the first camera 11 include in particular a first focal distance fl associated with the first camera 11.

[0044] The intrinsic parameters of the second camera 12 characterize, for their part, the transformation which associates, for an image point, its three-dimensional coordinates in the reference frame of the second camera 12 with the pixel coordinates in an image acquired by the second camera 12. These parameters do not change if the second camera 12 is moved. The intrinsic parameters of the second camera 12 include in particular a second focal length f2 associated with the second camera 12.

[0045] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account. account, that their correction is processed at the time of image acquisition or at the time of calibration.

[0046] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

[0047] A first marker is associated with the first camera 11: - the direction of the x axis is defined horizontal and normal to the optical axis of the first camera 11. The distance B separating the optical center of the first camera 11 from the projection of the optical center of the second camera 12 on the horizontal plane passing through the optical center of the first camera 11 is called the reference base (in English “baseline”); - the direction of the y axis is defined vertical and normal to the optical axis of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0048] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - three translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - three rotations in the x, y and z directions: 0x, 0y and 0z.

[0049] An extrinsic matrix of the vision system then includes the previously defined extrinsic parameters.

[0050] The extrinsic parameters are determined, for example, during a calibration phase of the stereoscopic vision system comprising the first camera 11 and the second camera 12.

[0051] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the reference base must reach 60cm for the cameras commonly used in this field.

[0052] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0053] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0054] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system comprising the second camera 12.

[0055] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system comprising the first camera 11.

[0056] Finally, field 19 is not visible to any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0057] According to a particular exemplary embodiment, the field of vision of the second camera 12 covers at least half of the field of vision of the first camera 11.

[0058] It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.

[0059] The images acquired by the cameras 11, 12 at an acquisition time instant are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0060] Each pixel in the acquired image is representative of an object in the three-dimensional scene present in the camera's field of vision. Indeed, a pixel in the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor in the camera after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image representing fi delimitation of the physical object observed by the camera. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene.

[0061] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are found for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0062] According to a particular embodiment, the first camera 11 and / or the second camera 12 is of the “wide-angle” type, a wide-angle camera being for example equipped with a lens designed to acquire an image representative of a three-dimensional scene perceived according to a wider field of vision than that of a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene taking place in front of or around the wide-angle camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle a of the field of vision of the wide-angle camera is for example equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, an open field of vision following an angle of 45° or less.Such a wide-angle camera is, for example, a camera equipped with mirrors or a "fisheye" camera. Wide-angle lenses have a shorter focal length than standard lenses, which makes them suitable for capturing images of landscapes, architecture, road intersections or any other subject requiring a wide perspective. Wide-angle cameras are, for example, used to capture immersive and dynamic images with an extended depth of field.

[0063] According to a particular embodiment, an image acquired by the first camera 11 and / or an image acquired by the second camera 12 comprises a distortion equal to 0.5%, 0.8% or greater than 1%. The measurement of such a distortion corresponds to the determination of a ratio between: - the maximum spacing of a pixel of the image from a straight line of the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the edge of the image, or at the corners of the edge of the image, and - the length of this edge.

[0064] Commonly, distortion is considered, in the world of photography, as: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • bothersome if it is greater than or equal to 1% or more.

[0065] A barrel distortion is characterized by a positive percentage, while a crescent distortion is characterized by a negative percentage.

[0066] According to a particular exemplary embodiment, a field of view of the first camera 11 covers at least half of a field of view of the second camera 12 and a field of view of the second camera 12 covers at least half of a field of view of the first camera 11. In other words, more than half of the pixels of an image acquired by the first camera 11 correspond to an object of the three-dimensional scene seen by the second camera 12, pixels of an image acquired by the second camera 12 also corresponding to this object of the three-dimensional scene. Similarly, more than half of the pixels of an image acquired by the second camera 12 correspond to an object of the three-dimensional scene seen by the first camera 11, pixels of an image acquired by the first camera 11 also corresponding to this object of the three-dimensional scene.

[0067] The images acquired by the first camera 11 and by the second camera 12 are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0068] A method for determining a depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, that is to say by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the vision system of the vehicle 10.

[0069] [Fig. 2] illustrates a flowchart of the different steps of a method 2 for determining a depth of a pixel of an image by depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, for example in the vehicle 10 of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention. The method 2 is for example implemented by a device of the vision system embedded in the vehicle 10 or by the device 5 of [Fig. 5].

[0070] In a step 21, data representative of an image acquired by the first camera 11 and of an image acquired by the second camera 12 are received.

[0071] In a step 22, depths associated with a set of pixels of one of the received images are determined by the depth prediction model from the two received images.

[0072] Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object of the three-dimensional scene with which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

[0073] If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.

[0074] [Fig. 3] illustrates a flowchart of the different steps of a method for learning the depth prediction model used in a method for determining a depth of a pixel of an image, for example in method 2 of [Fig. 2], according to a particular and non-limiting exemplary embodiment of the present invention.

[0075] The learning method 3 is for example implemented by the device on board the vehicle 10 implementing the method for determining a depth by a vision system on board a vehicle or by the device 5 of [Fig.5].

[0076] In a step 31, data representative of a first image 41 and a second image 42 are received, the first image 41 being acquired by the first camera 11 at an acquisition time instant and the second image 42 being acquired by the second camera 12 at the same acquisition time instant.

[0077] According to a particular exemplary embodiment, the first image and second image have the same definition, that is to say they comprise the same number of pixels, have the same number of pixels according to their height and the same number of pixels according to their width.

[0078] According to another particular exemplary embodiment, the first image and second image are not of the same definition. An additional step then consists of resizing or cropping them to obtain a first image and a second image of the same definition.

[0079] In a step 32, a first feature map associated with the first image 41 and a second feature map associated with the second image 42 are determined by a feature extractor.

[0080] Such a feature extractor is known to those skilled in the art and is for example presented in the document “Unifying Flow, Stereo and Depth Estimation” written by Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao and Andréas Geiger, published in July 2023.

[0081] Such characteristics are for example representative of information relating to a shape of an object in an image and / or a texture of a set of pixels of an image and / or a color of a set of pixels of an image.

[0082] In a step 33, directions associated with pixels of a first set of pixels of the first image, called first pixels, and directions associated with pixels of a first set of pixels of the second image, called second pixels, are determined by a direction prediction model from the first image and the second image respectively.

[0083] Such a direction prediction model is known to those skilled in the art; it is notably presented in the document “Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion” written by Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich and Adrien Gaidon, published in August 2020.

[0084] In a step 34, depths associated with the first pixels and the second pixels are determined by the depth prediction model from the first image and the second image respectively.

[0085] Such a depth prediction model, implemented by a convolutional neural network, is known to those skilled in the art and is for example presented in the document “Unifying Flow, Stereo and Depth Estimation”

[0086] In a step 35, a third image is generated from: • of the first image, • directions and depths associated with the first pixels, • extrinsic parameters of the stereoscopic vision system, and • of the direction prediction model.

[0087] Similarly, a fourth image is generated from: • of the second image, • directions and depths associated with the second pixels, • extrinsic parameters of the stereoscopic vision system, and • of the direction prediction model.

[0088] The generation of an image from an image acquired by a camera of the vision system consists of the reprojection of a pixel of the acquired image into the three-dimensional scene in the form of a point, then projecting this point into the image plane of a another camera in the vision system, so as to obtain an image corresponding to a view of the three-dimensional scene from the point of view of the other camera. The image plane of a camera corresponds to a plane defined in the camera's frame of reference, normal to the optical axis of the camera and located at the first focal length of the camera. Thus, the third image generated from the first image is comparable to the second image. Similarly, the fourth image generated from the second image is comparable to the first image. Since the cameras' fields of vision are not the same and objects may mask other objects in the scene, the generated images are not identical to acquired images. In addition, depth prediction, like the models used to generate the images, are not error-free. Thus, comparing a generated image to an acquired image makes it possible to evaluate the relevance of the different models used.

[0089] According to a particular exemplary embodiment, the third and fourth images are generated using the following function:

[0090] [Math.l] Ps = D{Pt) ) ] )

[0091] With: • Ps a pixel of a generated image corresponding to the third image, respectively to the fourth image, • 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector, • K a direction prediction model associated with the second camera 12, respectively with the first camera 11, • K' a direction prediction model associated with the first camera 11, respectively with the second camera 12, • T an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and • [)(p ) is a depth of one pixel Pt of the first image, respectively of the second image, determined by the convolutional neural network.

[0092] Note that the extrinsic matrix is ​​not the same for the generation of the two images; in fact, a first extrinsic matrix makes it possible to move from a reference frame associated with the first camera 11 to a reference frame associated with the second camera 12 during the generation of the third image, while a second extrinsic matrix makes it possible to move from a reference frame associated with the second camera 12 to a reference frame associated with the first camera 11 during the generation of the fourth image.

[0093] Projections and reprojections are inverse functions obtained from the direction prediction model and are a function of depth, such a projection model is notably presented in the document “Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion”.

[0094] In a step 36, a first error associated with each first pixel is determined by comparing pixels of the first and fourth images and a second error associated with each second pixel is determined by comparing pixels of the second and third images;

[0095] According to a first particular exemplary embodiment, the first and second errors are photometric errors (in English “photometric error”) as presented in the document “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019 and are determined by the following function:

[0096] [Math.2] - £p[ (1-a) ■ \I(p) -I(p)\+a- (l-±SSIM(j(p),I(p)))]

[0097] With: • L*(p) the first reconstruction error noted L^p), respectively the second reconstruction error noted L2(p),P being a pixel defined by its co ordered in an image, • I(p) a value of the pixel P not included in the invisibility mask in the first image, respectively second image, * a value of the pixel P not included in the invisibility mask in the fourth image, respectively third image, • SSIM a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0098] According to a second particular exemplary embodiment, the first and second errors comprise a determination of a reconstruction error of a generated image further determined by the following function:

[0099] [Math.3]

[0100] With: P )> °) a First error for a pixel p of the third image, respectively a second error for a pixel p of the fourth image, * p ) is a one-pixel depth Pt obtained from the first depth map, respectively obtained from the second depth map; • W is a parameter matrix; • 0 is the order of a smoothing gradient; • an L1 norm of second-order depth gradients is calculated with W =1, and0 =2; • x and y are the dimensions of the third image, respectively fourth image; • is an environment-dependent hyperparameter; and • [ ( nj is a value of the pixel Pt in the third image, respectively fourth image.

[0101] This second function is generally used to deal with discontinuity at the edge of objects (in English “edge aware smoothness”).

[0102] The reconstruction error is thus defined, for example, from the photometric errors and reconstruction errors previously defined.

[0103] In a step 37, an invisibility mask is determined from first coordinates of a second set of pixels of the first image associated with at least one object of the three-dimensional scene not visible in the second image and from second coordinates of a second set of pixels of the second image associated with at least one object of the three-dimensional scene not visible in the first image.

[0104] According to a particular exemplary embodiment, the invisibility mask is determined using an algorithm known to those skilled in the art, for example the torch.nn.functional.grid_sample() function, used in python coding, makes it possible to obtain the invisibility mask from an image acquired by a camera and depths and directions associated with pixels of the image. Such a function modifies a pixel according to its potential presence in an image acquired by another camera, associating it for example with a zero RGB value equivalent to a black pixel when it has no image in the other image, the other image corresponding to a previously generated image. Indeed, a pixel of the image acquired by a camera can be associated with an object located outside the field of vision of the other camera.

[0105] [Fig.4] schematically illustrates an invisibility mask determined from images received during the learning method, according to a particular and non-limiting exemplary embodiment of the present invention.

[0106] [Fig.4] thus represents the first image 41 and the second image 42. A set 410 of pixels of the first image 41 are associated with at least one object of the three-dimensional scene not visible by the second camera 12 while a set 420 of pixels of the second image 45 are associated with at least one object of the three-dimensional scene not visible in the first image 420.

[0107] The example illustrated in [Fig.4] corresponds to a first camera 11 positioned to the left and lower than the second camera 12, corresponding for example to a first camera 11 placed in a left rearview mirror of the vehicle 10 and to a second camera 12 placed at the top center of the windshield of the vehicle 10. The set 410 of pixels of the first image 41 thus comprises pixels located in a left part and in a lower part of the first image 41, while the set 420 of pixels of the second image 42 comprises pixels located in a right part and in a high part of the second image 42.

[0108] According to a particular embodiment, the visibility mask 43 corresponds to the pixels included both in the set 410 of pixels of the first image 41 and in the set 420 of pixels of the second image 42 when these are superimposed. Thus, the invisibility mask 43 comprises a first zone 431 and a second zone 432, corresponding in this example to the upper left and lower right corners of the images to which the invisibility mask is applied. In other words, the invisibility mask 43 corresponds to the intersection of the set 410 of pixels of the first image 41 and the set 420 of pixels of the second image 42.

[0109] According to this particular exemplary embodiment, the invisibility mask comprises coordinates included in the first coordinates and in the second coordinates. A zero value is then assigned to each first error and each second error associated with a pixel included in the invisibility mask.

[0110] In a step 38, the depth prediction model is learned by minimizing a loss error determined from the first and second errors, a first error and a second error associated with a pixel included in the invisibility mask being zero.

[0111] According to a particular exemplary embodiment, the loss error is determined by the following function:

[0112] [Math.4] L' = ^minfL^p), L2(p) )

[0113] With: • The loss error, • L\(p) the first reconstruction error for a pixel P of the first image, and • L2{ p) the second reconstruction error for a pixel P of the second image corresponding to the pixel P of the first image.

[0114] It should be noted that p ) = 0 when P is included in the mask of invisibility.

[0115] Training the depth prediction model consists of adjusting input parameters of the convolutional neural network in order to minimize the previously calculated loss error.

[0116] Thus, the depth prediction model used for the depth prediction of a pixel of an image acquired by the first camera 11 or by the second camera 12 is made reliable thanks to this learning process. In addition, the data enabling this learning are obtained from the stereoscopic vision system itself, they then correspond to data perfectly representative of the use of the vision system on board the vehicle 10.

[0117] [Fig. 5] schematically illustrates a device 5 configured for the determination of a depth by a vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 5 corresponds for example to a device on board the first vehicle 10, for example a computer associated with the stereoscopic vision system.

[0118] The device 5 is for example configured for the implementation of the operations described with regard to figures 1 and 4 and / or steps described with regard to figures 2 and 3. Examples of such a device 5 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 5, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 5 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0119] The device 5 comprises one (or more) processor(s) 50 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 5. The processor 50 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 5 further comprises at least one memory 51 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0120] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the memory 51.

[0121] According to various particular and non-limiting embodiments, the device 5 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0122] According to a particular and non-limiting exemplary embodiment, the device 5 comprises a block 52 of interface elements for communicating with external devices. The interface elements of block 52 include one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0123] According to another particular and non-limiting exemplary embodiment, the device 5 comprises a communication interface 53 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 530. The communication interface 53 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 530. The communication interface 53 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0124] According to a particular and non-limiting exemplary embodiment, the device 5 can provide output signals to one or more external devices, such as a display screen 540, touch-sensitive or not, one or more speakers 550 and / or other peripherals 560 via the output interfaces 54, 55, 56 respectively. According to a variant, one or other of the external devices is integrated into the device 5.

[0125] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for measuring a distance separating an object from a vehicle carrying a vision system acquiring highly distorted images, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0126] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 5 of [Fig.5].

Claims

Claims

1. Method for determining a depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising a first camera (11) and a second camera (12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method being implemented by at least one processor, and being characterized in that the depth prediction model is learned in a learning phase comprising the following steps: - reception (31) of data representative of a first image and a second image acquired by the first camera (11) and the second camera (12) respectively at the same acquisition time instant; - determining (32) a first feature map associated with the first image and a second feature map associated with the second image by a feature extractor; - determination (33) of directions associated with pixels of a first set of pixels of the first image, called first pixels, and of directions associated with pixels of a first set of pixels of the second image, called second pixels, by a direction prediction model from respectively the first image and the second image; - determination (34) of depths associated with the first pixels and the second pixels by said depth prediction model from the first image and the second image respectively; - generation (35) of a third image from the first image, the directions and depths associated with the first pixels, extrinsic parameters of the stereoscopic vision system and the direction prediction model and generation of a fourth image from the second image, the directions and depths associated with the second pixels, the extrinsic parameters of the stereoscopic vision system and the direction prediction model; - determination (36) of a first error associated with each first pixel by comparing pixels of the first and fourth images and of a second error associated with each second pixel by com- comparison of pixels of the second and third images; - determining (37) an invisibility mask from first coordinates of a second set of pixels of the first image associated with at least one object of the three-dimensional scene not visible in the second image (410) and from second coordinates of a second set of pixels of the second image associated with at least one object of the three-dimensional scene not visible in the first image (420), a zero value being assigned to each first error and each second error associated with a pixel included in the invisibility mask; and - learning (38) the depth prediction model by minimizing a loss error determined from the first and second errors.

2. Method according to claim 1, for which the first and second errors are photometric errors determined by the following function: ^(p) = Ep[(la) ' \l(p)-î(p)\+a-(ly>SIM[l(p^ With: • L*(p) the first reconstruction error noted L^p), respectively the second reconstruction error noted L2( p), P being a pixel defined by its coordinates in an image, • I(p) a value of the pixel P not included in the invisibility mask in the first image, respectively second image, * a value of the pixel P not included in the invisibility mask in the fourth image, respectively third image, • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

3. Method according to claim 1 or 2, for which the loss error is determined by the following function: L' = p), L2(p) ) With: • L' the loss error, • the first reconstruction error for a pixel P of the first image, and • L2(p) the second reconstruction error for a pixel P of the second image corresponding to pixel P of the first image.

4. Method according to one of claims 1 to 3, for which the third and fourth images are generated using the following function: With: • Ps a pixel of a generated image corresponding to the third image, respectively the fourth image, • 77 a function for going from homogeneous coordinates to pixel coordinates by removing a dimension of a vector, • K a direction prediction model associated with the second camera (12), respectively with the first camera (11), • K' a direction prediction model associated with the first camera (11), respectively with the second camera (12), • T an extrinsic matrix of the stereoscopic vision system, • 0 a projection function in the three-dimensional scene of a pixel as a function of its depth, and • D^p ) is a depth of a pixel Pt of the first image, respectively of the second image, determined by the convolutional neural network.

5. Method according to one of claims 1 to 4, for which the invisibility mask comprises coordinates included in the first coordinates and in the second coordinates.

6. Method according to one of claims 1 to 5, wherein at least one of the cameras (11, 12) of the stereoscopic vision system is a wide-angle camera.

7. A method according to one of claims 1 to 6, wherein a field of view of the first camera (11) covers at least half of a field of view of the second camera (12) and a field of view of the second camera (12) covers at least half of a field of view of the first camera (11).

8. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.

9. Device (5) for determining a depth by a vision system on board a vehicle (10), said device (5) comprising a memory (51) associated with at least one processor (50) configured for implementing the steps of the method according to any one of the claims 1 to 7.

10. Vehicle (10) comprising the device (5) according to claim 9.

Images