Method and device for generating training images of a depth prediction model for a vision system with heterogeneous cameras.

Abstract

A method or device for determining the depth of a pixel in an image acquired by a camera of a vision system, the depth being determined by a depth prediction model learned via a learning process (3) from pairs of training images acquired by the cameras of the vision system. Specifically, for each pair of training images, a set of augmented image pairs is generated (31), the augmented images then being filtered (32) using an edge extraction filter. Average images are then generated (33) from the filtered images, and values ​​are assigned (34) to the pixels based on their value compared to a threshold value. The depth prediction model is then learned (37) by minimizing an error determined by comparing the average images to images generated from these average images and the predicted depths for their pixels. Figure for the abstract: Figure 3

Description

Title of the invention: Method and device for generating training images of a depth prediction model for a vision system with heterogeneous cameras. technical field

[0001] The present invention relates to methods and devices for generating training images of a depth prediction model for a vision system embedded in a vehicle, for example in a motor vehicle, and more particularly for a vision system comprising several heterogeneous cameras.

[0002] The present invention also relates to a method for learning a depth prediction model for a vehicle-mounted vision system from training images acquired by the vehicle-mounted vision system. Technological background

[0003] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0004] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0005] The AD AS systems embedded in a vehicle are powered by data obtained one or more on-board sensors such as, for example, cameras. These cameras make it possible to detect and locate other road users or any obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle's speed; - to act on the braking system in case of risk of impact with an object.

[0006] The position of another user or an obstacle is, for example, determined by a vision system comprising a depth or distance prediction model. Existing depth prediction models are generally designed for vision systems comprising a single camera or multiple cameras. Homogeneous cameras generate images where the pixels have the same values ​​for the same color of object observed in the three-dimensional scene. However, if the cameras are not homogeneous, then the pixels in the images acquired by these cameras and corresponding to the same object in the three-dimensional scene will not have the same value, and their correspondence is not easy to establish. Thus, it is necessary to develop a depth prediction model capable of finding corresponding pixels even when the images are acquired by heterogeneous cameras in order to reliably predict depths or distances.

[0007] The quality of the training of the prediction model is very important; indeed, the depths or distances predicted by the prediction model represent, for example, the distances to other road users or obstacles present in the road environment of the vehicle equipped with the vision system and ADAS. The proper functioning of the driver assistance devices using this data therefore depends on the quality of the data emitted by the vision system. Summary of the present invention

[0008] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0009] Another object of the present invention is to learn a depth prediction model associated with a vision system comprising heterogeneous cameras.

[0010] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from a vision system on board the vehicle.

[0011] According to a first aspect, the present invention relates to a method for determining the depth of a pixel of an image acquired by a camera of a vision system embedded in a vehicle, the vision system comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from a different viewpoint, the depth being determined by a depth prediction model, the method being implemented by at least one processor, and being characterized in that the depth prediction model is learned via a learning process from a set of pairs of training images, each pair of training images comprising a first training image and a second training image acquired respectively by the first camera and the second camera at the same time of acquisition, the learning process comprising the following steps: - for each pair of training images in the set of training image pairs: • generation of a set of augmented image pairs, each augmented image pair comprising a first augmented image and a second augmented image generated respectively from the first training image and the second training image; • generation of a set of filtered image pairs, each filtered image pair comprising a first filtered image and a second filtered image generated respectively by applying an edge extraction filter to a first augmented or training image and to a second augmented or training image of an augmented image pair or of the training image pair; • generation of a pair of average images comprising a first average image and a second average image generated respectively from the first filtered images and the second filtered images; • for each pixel of the first average image and the second average image, a first value is assigned if the value of said pixel is less than a threshold value and a second value if the value of said pixel is greater than or equal to the threshold value; • generation of a third image from the first average image and depths associated with the pixels of the first average image and generation of a fourth image from the second average image and depths associated with the pixels of the second average image, the depths being predicted by the depth prediction model from the first and second average images; • determination of a particular error by comparing the fourth image to the first average image and by comparing the third image to the second average image; - learning the depth prediction model by minimizing a general error determined from the particular errors.

[0012] The averaged images thus processed allow us to consider only geometric data obtained via the extraction of contours from images acquired by cameras that may be heterogeneous. Thus, even if the cameras do not generate pixels of the same value for the same object point in the three-dimensional scene, the prediction model is able to predict a reliable depth.

[0013] Moreover, the depth prediction model is learned from data generated by the vision system itself, the images of the learning image pairs are then relevant because they are representative of real road environments encountered by the vehicle carrying the vision system and observed by the cameras of the vision system.

[0014] According to a variant of the method, a pixel value of a first average image or a pixel value of a second average image is equal to an average of corresponding pixel values ​​in respectively the first filtered images and the second filtered images.

[0015] According to another variant of the method, the edge extraction filter assigns a value equal to the first value to each pixel not belonging to an edge and a value equal to the second value to each pixel belonging to an edge.

[0016] According to yet another variant of the method, the first value is equal to 0 and the second value is equal to 255 when pixel values ​​of the first and second training images are coded on 8 bits.

[0017] According to a further variant of the method, the contour extraction filter is a Canny filter.

[0018] According to a further variant of the method, the edge extraction filter includes two adjustment parameters corresponding to detection thresholds for salient object edges and non-salient object edges, the parameters being defined by comparing average images with training images.

[0019] According to yet another variant, the method further includes a step of determining a statistical distribution of pixel values ​​of the first and second average images, the threshold value being defined according to the statistical distribution.

[0020] According to a second aspect, the present invention relates to a device for determining the depth of a pixel of an image acquired by a camera of a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for the implementation of the steps of the method according to the first aspect of the present invention.

[0021] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0022] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0023] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0024] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which a program is recorded computer including instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0025] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0026] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0027] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0028] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 4, in which:

[0029] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0030] [Fig.2] illustrates a flowchart of the different steps of a process for determining the depth of a pixel of an image by a depth prediction model associated with a vision system embedded in the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention;

[0031] [Fig. 3] illustrates a flowchart of the different stages of a method for learning the depth prediction model used in the method of [Fig. 2], according to a particular and non-limiting embodiment of the present invention; and

[0032] [Fig.4] schematically illustrates a device configured to determine the depth of a pixel of an image by a neural network associated with a vision system embedded in the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0033] A method and device for determining the depth of a pixel of an image by a neural network associated with a vision system embedded in a vehicle will now be described in the following with joint reference in figures 1 to 4. The same elements are identified with the same reference symbols throughout the description that follows.

[0034] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0035] According to a particular and non-limiting embodiment of the present invention, a method and a device implementing the method of determining the depth of a pixel of an image acquired by a camera of a vision system, the depth being determined by a depth prediction model learned via a learning process from pairs of training images acquired by the cameras of the vision system.

[0036] Indeed, for each pair of training images, a set of augmented image pairs is generated, the augmented images then being filtered using an edge extraction filter. Average images are then generated from the filtered images and values ​​are assigned to the pixels of these filtered images based on their value compared to a threshold value.

[0037] The depth prediction model is then learned by minimizing a determined error by comparing the average images to images generated from these average images and predicted depths for the pixels of the average images.

[0038] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0039] Such an environment 1 corresponds, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0040] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say, a vehicle operating according to a predetermined level of autonomy or under the total supervision of the driver.

[0041] The vehicle 10 advantageously comprises at least two on-board cameras, a first camera 11 and a second camera 12, configured to acquire images of a three-dimensional scene unfolding in the environment of the vehicle 10 from distinct observation positions. The first camera 11 and the second camera 12 form a stereoscopic vision system when used together, as illustrated in [Fig. 1]. The first camera 11 forms a monoscopic vision system when used alone, as does the second camera 12 forms another monoscopic vision system when used alone. The present invention, however, extends to any vision system comprising at least two cameras, for example 2, 3 or 5 cameras.

[0042] The intrinsic parameters of the first camera 11 characterize the transformation which associates, for an image point, hereafter called "point", its three-dimensional coordinates in the reference frame of the first camera 11 with the pixel coordinates in an image acquired by the first camera 11. These parameters do not change if the first camera 11 is moved. The intrinsic parameters of the first camera 11 include in particular a first focal length fl associated with the first camera 11.

[0043] The intrinsic parameters of the second camera 12 characterize, for their part, the transformation which associates, for an image point, its three-dimensional coordinates in the reference frame of the second camera 12 with the pixel coordinates in an image acquired by the second camera 12. These parameters do not change if the second camera 12 is moved. The intrinsic parameters of the second camera 12 include in particular a second focal length f2 associated with the second camera 12.

[0044] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account, and their correction is addressed during image acquisition or calibration.

[0045] These two cameras 11, 12 are arranged so that each acquires an image of a scene from a different viewpoint. The first viewpoint is, for example, located on or in the left-hand rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. The second viewpoint is, for example, located on or in the right-hand rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. If both cameras are located at the top of the windshield of the vehicle, they are then positioned at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, and the second camera 12 is located in the right-hand rearview mirror of the vehicle 10.

[0046] A first marker is associated with the first camera 11: - the direction of the x-axis is defined as horizontal and normal to the optical axis Cl of the first camera 11. The distance B separating the optical center of the first camera 11 from the projection of the optical center of the second camera 12 onto the horizontal plane passing through the optical center of the first camera 11 is called the reference base (in English "baseline"); - the direction of the y-axis is defined as vertical and normal to the optical axis Cl of the first camera 11; - The direction of the z-axis is defined as orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal coordinate system.

[0047] The optical axis Cl of the first camera 11 and the optical axis C2 of the second camera 12 are not necessarily parallel or even included in the same plane.

[0048] The extrinsic parameters related to the position of cameras 11, 12 are the following parameters: - three translations in the x, y, and z directions: Tx, Ty, and Tz, constituting the translation vector T; and - three rotations in the x, y and z directions: 0x, 0y and 0z.

[0049] An extrinsic matrix of the vision system then includes the extrinsic parameters previously defined.

[0050] The extrinsic parameters are determined, for example, during a calibration phase of the stereoscopic vision system comprising the first camera 11 and the second camera 12.

[0051] A key constraint of stereoscopic vision systems used in automobiles is, for example, the large distance between the two cameras. Indeed, to cover a measurement range of 200 meters, the reference base must be 60 cm for cameras commonly used in this field.

[0052] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 alone covering a first acquisition field 13, the second camera 12 alone covering a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0053] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0054] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system comprising the second camera 12.

[0055] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system comprising the first camera 11.

[0056] Finally, field 19 is not visible to any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0057] According to one particular embodiment, the field of view of the second camera 12 covers at least half of the field of view of the first camera 11.

[0058] It is evident that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently.

[0059] Each pixel of an acquired image represents an object in the three-dimensional scene present in the camera's field of view. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a point of light resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, which are captured by a photosensitive sensor of the camera after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image that faithfully represents the physical object observed by the camera. An image point described above is therefore a point on the surface of an object in the three-dimensional scene.

[0060] The images acquired by cameras 11, 12 represent views of the same scene taken from different viewpoints, the camera positions being distinct. This scene includes, for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0061] The first camera 11 and second camera 12 are, according to a particular embodiment, color cameras, that is to say, they acquire RGB (“Red Green Blue”) type images, representing the three-dimensional scene as it would be, for example, seen by the human eye. Thus, the pixels represent the scene in the visible spectrum. Such an RGB type image comprises a set of pixels, each pixel in this set of pixels being coded on three channels, each channel corresponding to one of the primary colors mentioned previously. However, the cameras do not necessarily assign the same colors to the same image points in the three-dimensional scene. Indeed, the parameters of the first and second cameras 11, 12, their optical systems, and their sensors may differ, in which case the rendering of the same color in the three-dimensional scene differs in pixels of the images acquired by each camera. The images acquired by the cameras exhibit, for example, a difference in white balance or ISO value.

[0062] According to a particular embodiment, the first camera 11 and / or the second camera 12 is of the "wide-angle" type, a wide-angle camera being, for example, equipped with a lens designed to acquire a representative image of a three-dimensional scene seen over a wider field of view than a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene unfolding in front of or around the wide-angle camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle α of the field of view of the wide-angle camera is, for example, equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, a field of view open at an angle of 45° or less.A wide-angle camera, for example, is a camera equipped with mirrors or a fisheye camera. Wide-angle lenses have a shorter focal length compared to standard lenses, making them suitable for capturing images of landscapes, architecture, road intersections, or any other subject requiring a wide perspective. Wide-angle cameras are used, for example, to capture immersive and dynamic images with an extended depth of field.

[0063] According to a particular embodiment, an image acquired by the first camera 11 and / or an image acquired by the second camera 12 includes a distortion equal to 0.5%, 0.8%, or greater than 1%. The measurement of such distortion corresponds to determining a ratio between: - the maximum spacing of a pixel in the image of a straight line in the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the image edge or at the corners of the image edge, and - the length of this edge.

[0064] In the world of photography, distortion is commonly considered to be: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • bothersome if it is greater than or equal to 1% or more.

[0065] Barrel distortion is characterized by a positive percentage, while crescent distortion is characterized by a negative percentage.

[0066] According to a particular embodiment, a field of view of the first camera 11 covers at least half of a field of view of the second camera 12, and a field of view of the second camera 12 covers at least half of a field of view of the first camera 11. In other words, more than half of the pixels of an image acquired by the first camera 11 correspond to an object in the three-dimensional scene seen by the second camera 12, and pixels of an image acquired by the second camera 12 also correspond to this object in the three-dimensional scene. Similarly, more than half of the pixels of an image acquired by the second camera 12 correspond to an object in the three-dimensional scene seen by the first camera 11, and pixels of an image acquired by the first camera 11 also correspond to this object in the three-dimensional scene.

[0067] The images acquired by the first camera 11 and by the second camera 12 are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0068] A method for determining depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, i.e. by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer or computers in charge of the vision system of the vehicle 10.

[0069] Figure 2 illustrates a flowchart of the different steps of a method 2 for determining the depth of a pixel in an image acquired by the first camera of a vision system mounted in a vehicle, for example by the first camera 11 of the vision system mounted in the vehicle 10 of Figure 1. The vision system mounted in the vehicle 10 thus comprises a first and a second camera arranged so as to each acquire an image of a three-dimensional scene, i.e. taking place in the vicinity of the vehicle 10. Such a three-dimensional scene includes, for example, an object located in the field of vision of the first and second cameras, i.e. in the third acquisition field 15 shown opposite [Fig. 1]. Method 2 is for example implemented by a device of the vision system on board the vehicle 10 or by device 4 of [Fig. 4].

[0070] According to a particular embodiment, process 2 comprises the following steps.

[0071] In a step 21, an image acquired by the first camera 11 and an image acquired by the second camera 12 are received. The reception of these images consists of receiving data representative of these images. These images are acquired respectively by the first camera 11 and by the second camera 12 at the same acquisition time. They therefore represent the same three-dimensional scene observed from two distinct viewpoints at the same time.

[0072] According to a particular embodiment, the received images are of the same definition, that is to say they have the same number of pixels, have the same number of pixels according to their height and the same number of pixels according to their width and a large part of each image represents the same part of the current three-dimensional scene.

[0073] According to another particular embodiment, the received images are not of the same resolution and / or only a small part of each of them represents the same part of the current three-dimensional scene. An additional step then consists of resizing or cropping them to obtain two images of the same resolution and whose pixels correspond mainly to the same part of the three-dimensional scene observed at the time of acquisition, for example more than 80% of the pixels of each image are representative of the same part of the current three-dimensional scene seen by the two cameras 11, 12.

[0074] If the cameras are calibrated, for example by following a calibration method known to those skilled in the art such as that presented in the document "Single View Point Omnidirectional Camera Calibration from Planar Grids", written by Christopher Mei and Patrick Rives and published in April 2007, then the intrinsic parameters of the first and second cameras 11, 12 are known. Edges of the common field of view can be identified in the images received by a simulation via a back-projection and projection program, which are described below.

[0075] The rear projection program performs the following operations: • The pixel coordinates are corrected to remove distortion defined by distortion parameters: the parameter of the model presented in the cited document, called the Mei model, and distortion coefficients kh k2, k3, pi and p2, the definition of which can be found in the free library OpenCV® for example. • The coordinates are projected onto the unit sphere according to the Mei model. • Projection lines are calculated, always according to the Mei model. • Projection lines are multiplied by a depth.

[0076] The projection program then proceeds by performing the following operations: • The coordinates of the points in space are transformed into the coordinate system of the other image. • The coordinates of the points in space are modified via the distortion function defined by the same parameters as those mentioned previously. • Multiplication with the inverse of the intrinsic matrix similar to the pinhole camera lens.

[0077] By associating a depth corresponding to the midpoint of a measurement range, for example 100 m (one hundred meters) for a measurement range from 10 m (ten meters) to 200 m (two hundred meters), with the pixels of the first current image, and by backprojecting and projecting one of the received images onto the other received image with this depth, the simulation makes it possible to determine the average edge of the shared field of view on the other received image. The received image is to be cropped with the same resolution as the other received and cropped image, and on the opposite side along a diagonal to the cropped side of the other received image.

[0078] If the first and second cameras are not calibrated, the projection and reprojection models are defined by a neural network such as the one described in the paper "Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion" by Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich, and Adrien Gaidon, published in August 2020. This particular embodiment is well-suited to images acquired by the vision system cameras and exhibiting significant distortion. It is not possible to determine the exact cropping before training. The common field of view in the received images can be found using a trial-and-error method, starting with cropping by observing the current images and training the projection and reprojection models to improve their effectiveness.If the pixels at the edge of the received images do not have a reasonable prediction, for example with a value that is too high or too low, or when a depth map does not match the texture of an original image, the size of the cropped image must be reduced. If some of the pixels, preferably half, have reasonable predictions while the others do not, the correct cropping is achieved.

[0079] In step 22, depths associated with a set of pixels in the received image are predicted from the two received images by the depth prediction model. Indeed, this depth prediction model is learned in a learning phase as described opposite [Fig. 3] so as to allow the depth prediction from a portion of images acquired by the cameras of the vehicle's on-board vision system 10.

[0080] Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object point of the three-dimensional scene to which the pixel corresponds, the determination of a depth of a pixel then corresponding to a measurement of a distance separating the vehicle 10 carrying the vision system from an object point associated with the object of the three-dimensional scene.

[0081] In an optional step 23, the distance separating the observed object from the vehicle 10 is determined and is, for example, equal to an average of depths associated with pixels corresponding to the object of the three-dimensional scene.

[0082] If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another road user, the ADAS is then able to determine this distance precisely and act accordingly. For example, if the ADAS is designed to activate the braking system of the vehicle 10 in the event of a risk of collision with another road user, and the distance between the vehicle 10 and that road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of the vehicle 10 to avoid a possible accident.

[0083] Figure 3 illustrates a flowchart of the different stages of a method for learning the depth prediction model used in the method of Figure 2, according to a particular and non-limiting embodiment of the present invention.

[0084] The learning process 3 is, for example, implemented by the device on board the vehicle 10 implementing the process of determining a distance separating an object from the vehicle 10 or by the device 4 of the [Fig.4].

[0085] The learning process 3 is implemented using a set of pairs of training images and includes the steps described below. Each pair of training images includes, in particular, a first training image acquired by the first camera 11 and a second training image acquired by the second camera 12, the first and second training images of a pair of training images being acquired at the same time instant of acquisition and therefore representative of the same three-dimensional scene observed by the two cameras at the same time instant.

[0086] Steps 31 to 36, detailed below, are followed for each pair of training images in the set of training image pairs. The results obtained from step 36 are then grouped for all image pairs. learning in such a way as to pool them during step 37 of learning the depth prediction model.

[0087] The pairs of training images represent the different environments in which the vehicle 10 operates or travels, for example, city centers, highways, suburbs, and rural areas amidst agricultural fields. The training images include, in particular, pixels corresponding to objects present in these environments representing different three-dimensional scenes.

[0088] The set of pairs of training images includes, for example, 1000 (one thousand) images of each type of environment.

[0089] In step 31, a set of augmented image pairs is generated for a pair of training images. Each augmented image pair comprises a first augmented image generated from the first training image and a second augmented image generated from the second training image using the same process. Each augmented image pair is generated using different algorithms to produce different augmented images. Note that using the same algorithm with different parameters, including parameters with randomly generated values, is considered to be using multiple algorithms.

[0090] Algorithms for generating augmented images are known to those skilled in the art. For example, some of the best-known ones, presented in certain libraries, include: • Grayscale®, which changes the values ​​of RGB (Red Green Blue) encoded pixels to generate a grayscale image by averaging the RGB channel values; and / or • ColorJitter®, which randomly changes the brightness, contrast, saturation, and hue of an image. Varying the brightness changes the gradient in areas without prominent objects much more noticeably than around the edges of prominent objects. This effect is even stronger with saturation, which increases the pixel value in certain areas to its maximum. Increasing the contrast enhances the visibility of key edges. Changing the hue alters the intensity level of each color channel to increase or decrease edge visibility; and / or • GaussianBlur®, which helps reduce the presence of edges by averaging the values ​​of adjacent pixels, thus reducing the gradient in areas without protruding objects much more than for pixels representing the edges of protruding objects; and / or • RandomInvert®, which gives the image another view with "complementary" colors: red is changed to cyan, green is changed to magenta, and blue is changed to yellow. This operation may produce a different result than edge detection; and / or • RandomPosterize®, which reduces the resolution of the color presentation: instead of 8 bits with a maximum value of 255, 4 bits are used to change the image's appearance. The effect is similar to saturation; and / or • RandomSolarize® is a partial application algorithm of RandomInvert®; only a portion of the pixels are given the complementary color; and / or • RandomAdjusteSharpness® is an algorithm similar to GaussionBlur® that enhances the appearance of edges; and / or • RandomEqualize® adjusts the statistical distribution of values ​​across the RGB channels to achieve a uniform distribution of values ​​across the three RGB channels. The appearance of the colors is changed.

[0091] The objective of these transformations is to create different image representations. Since the contours of salient objects have wider gradients, edge detection in these pixels is more robust, i.e., repeatable across different views. Edge detection in areas without salient objects is not repeatable because weak gradients are more affected by image transformations.

[0092] Since some of the above transformations have random parameters, repeating the same type of transformation generates different augmented images. It is therefore necessary to repeat the transformations several times, for example 5 to 15 times.

[0093] The selected image augmentations are intended to change the colors of the image pixels while keeping the main contours, where faint contours may disappear in the transformed images.

[0094] In step 32, a set of filtered image pairs is generated from each augmented image pair and from the first and second training images. Each filtered image pair then comprises a first filtered image generated by applying an edge extraction filter to a first augmented image from an augmented image pair or to the first training image, and a second filtered image generated by applying an edge extraction filter to the second augmented image from the augmented image pair or to the second training image. Thus, the filtered images of a filtered image pair are images derived from training images of the same training image pair to which the following have been applied: • the same augmentation algorithm, except for the filtered image pair generated directly from the training image pair, then • the same edge extraction filter.

[0095] The edge extraction filter is, according to a particular embodiment, a Canny filter. The filtered images then have pixels having the following values: • a first value when the pixel does not belong to a detected edge, or • a second value when the pixel belongs to a detected edge.

[0096] The Canny detector has two parameters: a lower gradient threshold, below which no contours are detected, and an upper threshold, above which contours are detected. Between these two thresholds, contours detected by the lower threshold are retained if they are connected with contours detected by the upper threshold; otherwise, they are discarded. The upper threshold allows the detection of contours of salient objects that represent geometric features in the image, such as bicycles, motorcycles, cars, and pedestrians, and environmental features such as sidewalks, buildings, traffic signs, and road markings. The lower threshold prevents contour detection in areas without salient objects, such as the sky, ground with low texture, or trees with many leaves, despite the slight color variation.Edge detection in these areas is unreliable: an edge may be detectable in one image but not in another due to different viewing perspectives. Therefore, the edge extraction filter includes two adjustment parameters corresponding to detection thresholds for salient and non-salient object edges. These parameters can then be defined by comparing filtered images to the training images from which they are derived, or even defined or adjusted after the following step 33 by comparing averaged images to training images.

[0097] According to a particular embodiment, for images whose pixels are coded on three RGB channels, the first value is zero, that is to say that the value of each channel is equal to 0 (zero), and the first value is equal to 255 (two hundred fifty-five), that is to say that the value of each channel is equal to 255.

[0098] In a step 33, a pair of average images is generated. The pair of average images comprises a first average image generated from the first filtered images and a second average image generated from the second filtered images.

[0099] A pixel value of a first average image or of a pixel of a second average image is, for example, equal to an average of corresponding pixel values ​​in respectively the first filtered images and the second filtered images.

[0100] According to the embodiment example where the first value is zero and the second value is equal to 255, by averaging the contour images of all the transformed images, the contours in areas without protruding objects have a value much lower than 255.

[0101] In an optional step, a probability distribution of the pixels in the first or second averaged image is determined, for example, over a range of values ​​from 1 to 255 with an interval of 8. The probability of an interval is the number of pixels in that interval accumulated among all averaged images divided by the number of all pixels with a non-zero value among all averaged images. Pixels with a value of zero are excluded because they provide no useful information for edge detection and they also represent most of the pixels in the averaged edged images, introducing confusion if they are included in the probability distribution curve.

[0102] The probability distribution is bimodal: pixels representing the contours of salient objects have a value close to 255, while other detected contours have a much lower value. This curve allows for further adjustment of the two parameters of the Canny filter. If the peak of the pixels representing the contours of salient objects is not very steep, the upper threshold must be increased to reduce the number of contours representing salient objects. If the peak representing the contours detected in areas without salient objects is not steep enough, the lower threshold must be increased to reduce the number of contours detected in these areas.

[0103] This second adjustment of the Canny filter parameters does not replace a visual comparison between the filtered and training images during step 32 because it is necessary to ensure a meaningful appearance of the contours of salient objects. Parameters representing excessively high thresholds generate steeper peaks, i.e., accentuate the bimodal nature of the statistical distribution, but the remaining contours may be insufficient to represent the objects in a three-dimensional scene.

[0104] In a step 34, for each pixel of the first average image and the second average image, a first value is assigned to the pixel if the pixel value is less than a threshold value and a second value is assigned to the pixel if the pixel value is greater than or equal to the threshold value.

[0105] The first and second values ​​are, for example, 0 and 255 respectively, as shown in previous specific embodiments. The threshold value is determined based on the statistical distribution described above. A value of zero is then assigned to all pixels with a value less than the threshold value in the first or second averaged image to suppress edge detection, while a value of 255 is assigned to all pixels with a value greater than the threshold value to maintain edge detection. This method increases the reliability of edge detection for salient objects. The output of this step 34 is the first and second averaged images include pixels representing detected contours whose value is equal to the second value, here 255, while the other pixels have a value of zero.

[0106] The threshold value is determined, for example, with regard to the statistical distribution; it is, for example, equal to a value separating the bimodal peaks of the curve representing the statistical distribution. This threshold value then belongs to an interval situated between the two peaks in the probability distribution curve and is, for example, the average value of this interval. This threshold value is, for example, equal to 128.

[0107] Step 34 thus makes it possible to remove any contour described as "noisy" in the first and second average images.

[0108] In a step 35, a third image is generated from the first averaged image and depths associated with the pixels of the first averaged image, and a fourth image is generated from the second averaged image and depths associated with the pixels of the second averaged image. The depths are predicted, in particular, by the depth prediction model from the first and second averaged images.

[0109] Image reconstruction is performed using the equation below:

[0110] [Math.l] p2 = ir(K'[T4>{p^',D(p^]) And [Math.l] [YES] With: • P [the coordinates of a pixel in the first averaged image, • P2 the coordinates of a pixel in the second averaged image, • A function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K a projection model associated with the first camera 11, • Kx a reprojection model associated with the first camera 11, • K' a projection model associated with the second camera 12, • a reprojection model associated with the second camera 12, • T and p1 matrices including extrinsic parameters, • 0 a projection function in the three-dimensional scene of a pixel Py as a function of its coordinates in the first average image, and the depth j associated with it, and • 0' a projection function in the three-dimensional scene of a pixel P2 as a function of its coordinates in the second average image, and of the depth «JI Which is associated with it.

[0112] Depending on the type of cameras, K and K' represent intrinsic matrices of the first and second cameras when these cameras have pinhole lenses.

[0113] Generating an image from an averaged image derived from an image acquired by a camera of the vision system consists of reprojecting a pixel of the averaged image into the three-dimensional scene as a point, and then projecting this point into the image plane of another camera of the vision system, so as to obtain an image corresponding to a view of the three-dimensional scene from the viewpoint of the other camera. The image plane of a camera corresponds to a plane defined in the camera's frame of reference, normal to the camera's optical axis and located at the camera's first focal length. Thus, the third image generated from the first averaged image is comparable to the second averaged image. Similarly, the fourth image generated from the second averaged image is comparable to the first averaged image. Since objects can obscure other objects in the scene, the generated images are not identical to acquired images.Furthermore, depth prediction, like the models used to generate or filter images, is not error-free. Therefore, comparing a generated image to an average image allows us to assess the accuracy of the different models used.

[0114] In a step 36, a particular error is determined by comparing the fourth image to the first average image and by comparing the third image to the second average image.

[0115] The particular error is for example determined from a main error associated with each pixel of the first average image, called the first pixel, by comparing the first pixels to the pixels of the fourth generated image and a secondary error associated with each pixel of the second average image, called the second pixel, by comparing the second pixels to the pixels of the third image.

[0116] According to a first particular embodiment, the primary and secondary errors are photometric errors as presented in the document “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019 and are determined by the following function: [Math.2]

[0117] With: • L*(p) the principal error denoted L^p}, respectively the secondary error denoted £2(jP), being a pixel defined by its coordinates in an image, • l(p) a value of the pixelp in the first average image, respectively second average image, * / (p) a pixelp value in the fourth image, respectively third image, • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of road environment in which the vehicle travels 10.

[0118] According to a second particular embodiment, the primary and secondary errors include a determination of a reconstruction error of a generated image determined, furthermore, by the following function:

[0119] [Math.3] w, o) = w(p,)

[0120] With: * ^smooth ( ( Pf ) ' ) a major error for a pixel pt of the third image, respectively a secondary error for a pixel pt of the fourth image, • D(pt^ is a depth associated with a pixel pt, • VF is a parameter matrix, • ° is the order of a smoothing gradient, • an L1 norm of the second-order depth gradients is calculated with VF = 1, et° = 2, • x and y are the dimensions of the third image, and the fourth image respectively. • P is a hyperparameter dependent on the road environment in which vehicle 10 is traveling, and • A( Pt} is a value of the pixel pt in the third image, respectively fourth image.

[0121] This second function is generally used to deal with discontinuity at the boundary of objects.

[0122] The particular error is thus defined, for example, from the photometric errors and reconstruction errors previously defined.

[0123] According to a particular embodiment, the particular error is determined by the following function: [Math.4] £ = Epmin(L1(p),£2(p))

[0124] With: • The specific error, • L^p) the principal error for a pixel p of the first averaged image, and • L2{p) the secondary error for a pixelp of the second averaged image corresponding to the pixel p of the first averaged image.

[0125] Additionally, the particular error is determined from a consistency error determined by a loss function representative of the consistency of depth maps. A first depth map includes the pixel coordinates and an associated depth in the first averaged image, and a second depth map includes the pixel coordinates and an associated depth in the second averaged image. The following are then generated: • a third depth map derived from the first depth map, and • a fourth depth map derived from the second depth map. We then obtain:

[0126] [Math.5] L c = L p {D t (p) -D t (p)) )

[0127] With: • The consistency error, • Dt(p) the depth of a pixelp in a reconstructed depth map, the third depth map, respectively the fourth depth map, and • Dt(p) the depth of the pixelp in a depth map, the second depth map, respectively the first depth map.

[0128] The third and fourth depth maps are generated similarly to the third and fourth generated images, with the pixel values ​​here being replaced by depths. Instead of using the image in the interpolation function, for example via the use of the torch.nn.functional.grid_sample function in Python®, a depth map is the input data.

[0129] The difference for each pixel can be the absolute value, or the squared value to obtain a positive value. The agglomeration function of the pixel differences of the two depth maps can be a sum or an average. This loss function for consistency is to be calculated for the two average images, that is, for the first and second depth maps. The agglomeration of the two loss functions to define the particular error can also be the sum or the average.

[0130] In a step 37, the depth prediction model is learned by minimizing a general error determined from the particular errors.

[0131] The general error is, for example, the sum of the absolute values ​​of the particular errors determined for each pair of training images.

[0132] The learning of the depth prediction model then consists of minimizing this general error and adjusting the input parameters of the convolutional neural network used by the depth prediction model.

[0133] Thus, the depth prediction model used for pixel depth prediction is made more reliable by this learning process, even when the first and second cameras are not identical or homogeneous. Furthermore, the data used for this learning are obtained from the stereoscopic vision system itself, and therefore correspond to data that perfectly represent the use of the vision system onboard the vehicle 10.

[0134] Figure 4 schematically illustrates a device 4 configured for determining a depth of one pixel of an image acquired by a camera of a vision system embedded in a vehicle 10, for determining the distance separating an object from the vehicle 10 and / or for training a depth prediction model associated with a vision system embedded in the vehicle 10, according to a particular and non-limiting embodiment of the present invention. The device 4 corresponds, for example, to a device embedded in the first vehicle 10, for example, a computer associated with the stereoscopic vision system embedded in the vehicle 10.

[0135] Device 4 is, for example, configured to carry out the steps described opposite Figures 2 and / or 3. Examples of such a device 4 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of device 4, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 4 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0136] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0137] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 41.

[0138] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0139] According to a particular and non-limiting embodiment, the device 4 includes a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 include one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HD MI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0140] According to another particular and non-limiting embodiment, the device 4 includes a communication interface 43 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 430. The communication interface 43 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0141] According to a particular and non-limiting embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch or not, one or more speakers 450 and / or other peripherals 460 (projection system) via output interfaces 44, 45, 46 respectively. In one variant, one or more of the external devices is integrated into device 4.

[0142] Of course, the present invention is not limited to the embodiments described above but extends to a method for measuring the distance between an object and a vehicle equipped with a vision system, which would include secondary steps without falling outside the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0143] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 4 of [Fig.4].

Claims

1. Demands Method for determining the depth of a pixel of an image acquired by a camera of a vision system embedded in a vehicle (10), the vision system comprising a first camera (11) and a second camera (12) arranged so as to each acquire an image of a three-dimensional scene from a different viewpoint, said depth being determined by a depth prediction model, said method being implemented by at least one processor, and being characterized in that the depth prediction model is learned via a learning process (3) from a set of pairs of training images, each pair of training images comprising a first training image and a second training image acquired respectively by the first camera (11) and the second camera (12) at the same acquisition time instant, the learning process (3) comprising the following steps: - for each pair of training images in said set of pairs of training images: • generation (31) of a set of augmented image pairs, each augmented image pair comprising a first augmented image and a second augmented image generated respectively from the first training image and the second training image; • generation (32) of a set of filtered image pairs, each filtered image pair comprising a first filtered image and a second filtered image generated respectively by applying an edge extraction filter to a first augmented or training image and to a second augmented or training image of an augmented image pair or of the training image pair; • generation (33) of a pair of average images comprising a first average image and a second average image generated respectively from the first filtered images and the second filtered images; • for each pixel of the first averaged image and the second averaged image, assign (34) a first value if the value said pixel is less than a threshold value and a second value if the value of said pixel is greater than or equal to the threshold value; • generation (35) of a third image from the first average image and depths associated with the pixels of the first average image and generation of a fourth image from the second average image and depths associated with the pixels of the second average image, the depths being predicted by the depth prediction model from the first and second average images; • determination (36) of a particular error by comparison of the fourth image to the first average image and by comparison of the third image to the second average image; - learning (37) of the depth prediction model by minimizing a general error determined from the particular errors.

2. A method according to claim 1, wherein a pixel value of a first average image or a pixel value of a second average image is equal to an average of corresponding pixel values ​​in the first filtered images and the second filtered images respectively.

3. A method according to claim 1 or 2, wherein the edge extraction filter assigns a value equal to said first value to each pixel not belonging to an edge and a value equal to said second value to each pixel belonging to an edge.

4. A method according to any one of claims 1 to 3, wherein said first value is equal to 0 and said second value is equal to 255 when pixel values ​​of the first and second training images are coded on 8 bits.

5. A method according to any one of claims 1 to 4, wherein the contour extraction filter is a Canny filter.

6. A method according to claim 5, wherein the edge extraction filter comprises two adjustment parameters corresponding to detection thresholds for salient object edges and non-salient object edges, said parameters being defined by comparing average images with training images.

7. A method according to any one of claims 1 to 6, further comprising a step of determining a statistical distribution of values pixels of the first and second average images, said threshold value being defined according to said statistical distribution.

8. A computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by a processor.

9. Device (4) for determining the depth of a pixel of an image acquired by a camera of a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for carrying out the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images