Method and device for determining the depth of a pixel in an image using a depth prediction model learned from heterogeneous images

Abstract

A method or device for determining the depth of a pixel using a depth prediction model associated with a vision system comprising a color camera and an infrared camera. The model is learned in a learning phase that includes the detection (32) of edges in the first and second images acquired by the cameras to obtain single-channel filtered first and second images, the determination (33) of a depth map associated with the second image from the filtered images, the generation (34) of a third image from the first image by adding black pixels, then the detection of edges in the third image, and the determination (35) of a depth map from the filtered third and second images. The depth prediction model is learned (36) by minimizing an error determined by comparing the depth maps. Figure 3 (for the abstract)

Description

Title of the invention: Method and device for determining the depth of a pixel in an image using a depth prediction model learned from heterogeneous images. Technical field

[0001] The present invention relates to methods and devices for determining depth using a vision system mounted in a vehicle, for example, in a motor vehicle. The present invention also relates to a method and device for measuring the distance between an object and a vehicle equipped with a vision system. Technological background

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] The AD AS systems embedded in a vehicle are powered by data obtained one or more onboard sensors, such as cameras. These cameras make it possible to detect and locate other road users or potential obstacles around a vehicle in order to, for example: • to adapt the vehicle's lighting according to the presence of other road users; • to automatically regulate the vehicle's speed; • to act on the braking system in case of risk of impact with an object.

[0005] In order to obtain a broad view of the vehicle's surroundings, that is, a three-dimensional scene unfolding around the vehicle, a wide-angle camera, i.e., a camera with a wide field of view, is recommended. Indeed, the use of a wide-angle camera offers numerous advantages compared to "standard" cameras: • a wider field of view allowing for the capture of a larger portion of the scene, which is particularly important in the context of driving, where it is essential to monitor the environment both to the sides and in front of the vehicle, for example, • Greater efficiency in perceiving complex environments, such as intersections, sharp turns, parking spaces, etc., by minimizing blind spots and providing a more complete view of driving situations, and • Increased safety by more easily detecting obstacles, other vehicles, pedestrians and cyclists in areas adjacent to the vehicle.

[0006] Although wide-angle cameras offer many advantages, they can also introduce distortions in the captured images, and these distortions can lead to certain problems such as: • the distortion of straight lines, for example barrel or pincushion distortion, causing a curvature of straight lines in an image acquired by the wide-angle camera, making it difficult to estimate the actual distances between objects, particularly towards the edges of the image, • Stretching or compressing objects, especially towards the edges of the image, alters the apparent size of objects, which can be problematic when assessing the distance or actual size of objects, • the alteration of object proportions, making them larger or smaller than their actual size and more complex to identify, and • the difficulty of rectifying or correcting the distortion in post-processing can be complex and may lead to a loss of information.

[0007] Thus, processing an image acquired by a wide-angle camera requires special handling, particularly due to the significant distortion present in the image. Determining the distance between the vehicle carrying the camera and an object in the scene is therefore not feasible using the methods commonly employed for standard cameras used in certain vision systems.

[0008] Predicting the distance between an object and a vehicle, that is, the depth associated with a pixel in an image acquired by a camera, the pixel representing the object, requires high precision to feed ADAS. Predicting the depth associated with a pixel relies in particular on a depth prediction model, the depth prediction model being adapted to the vehicle's onboard vision system. Such a prediction model is commonly implemented by a neural network, which is trained in a learning phase to predict accurate depths with respect to the onboard vision system and the type of environment in which the vehicle is operating. A learning phase therefore requires data, for example, Annotated data can be obtained from libraries or annotated via other, more precise embedded systems such as LiDAR®. However, this type of learning requires a massive accumulation of data or the presence of this other embedded system, which the vehicle does not always have. It is then possible to perform a training phase for the depth prediction model using data acquired by the onboard vision system, but existing solutions are not suitable for the configuration of every onboard vision system in a vehicle.

[0009] In order to use such a vision system in a dimly lit environment, at least one of the cameras in the vision system is, for example, infrared, meaning that it has a sensor capable of capturing light waves in the infrared range, for example with a wavelength between 0.7 pm and 100 pm. Such a camera acquires, for example, single-channel images, meaning images with a single channel, whereas cameras commonly used for vision systems are RGB (Red Green Blue), meaning they acquire multi-channel images comprising, for example, three channels, representing values ​​associated with light waves captured in the visible range, for example with a wavelength between 380 nm and 780 nm.The images acquired by these two types of camera are therefore not directly comparable because they are not homogeneous and have sensors that do not capture the same waves. Summary of the present invention

[0010] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0011] Another object of the present invention is to improve the quality of data from the processing of an image acquired by a vision system, in particular by a depth prediction model implemented by a neural network associated with a stereoscopic vision system including an infrared camera.

[0012] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from an infrared camera.

[0013] According to a first aspect, the present invention relates to a method for determining the depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from different viewpoints, the first camera being configured for color image acquisition and the second camera being configured for infrared image acquisition, the process being implemented by at least one processor, and being characterized in that the depth prediction model is learned in a learning phase comprising the following steps: - reception of a first image acquired by the first camera and a second image acquired by the second camera at the same time instant of acquisition; - obtaining a first, respectively second, single-channel filtered images by applying edge detectors to the first, respectively second, images; - determination of a first depth map including depths associated with a set of pixels from the second image, the depths being predicted with the depth prediction model from the first and second filtered images; - generation of a third image from the first image, the third image comprising a set of black pixels obtained by modifying the values ​​of a set of pixels from the first image and obtaining a third single-channel filtered image by applying an edge detector to the generated third image; - determination of a second depth map comprising depths associated with a set of pixels from the second image, the depths being predicted with the depth prediction model from the third and second filtered images; - learning the depth prediction model by minimizing a loss error determined by comparing the first and second depth maps.

[0014] Edge detectors applied to the images acquired by the vision system cameras, which are heterogeneous, make them homogenized and thus comparable. The depth prediction model is therefore applicable to the filtered images and allows for the accurate prediction of the depth of a pixel in an acquired image, particularly by the infrared camera of the stereoscopic vision system; that is, the distance separating the vehicle carrying the infrared camera from a physical object in the three-dimensional scene associated with that pixel. This learning is performed using data acquired by the onboard vision system and therefore does not require data annotated by another onboard system or the storage of a training image library.Furthermore, the training data is representative of the data received when the stereoscopic vision system is in operation or in production; indeed, the training data is representative of real environments in which the vehicle carrying the vision system moves, and is therefore particularly relevant.

[0015] According to an alternative method, the loss error is determined by the following function: With : • The loss error, • D(p) a depth of a pixel p obtained from the first depth map for a pixel p, and • D'(p) a depth of one pixel pt obtained from the second depth map for pixel p.

[0016] According to another variant, one of the edge detectors is a Canny filter.

[0017] According to yet another variant, the process further comprises an upstream learning phase comprising the following steps: - reception of a fourth image and a fifth image acquired by the first camera and the second camera respectively at the same time instant of acquisition; - obtaining a fourth, respectively fifth, single-channel filtered images by applying edge detectors to the fourth, respectively fifth, images; - determination of a third depth map including depths associated with a set of pixels from the fourth image and a fourth depth map including depths associated with a set of pixels from the fifth image, the depths being predicted with the depth prediction model from the filtered fourth and fifth images; - generation of fifth and sixth depth maps from the third and fourth depth maps and extrinsic parameters of the stereoscopic vision system; - generation of sixth and seventh images from the filtered fourth and fifth images and from pixel arrival coordinates of the sixth and fifth depth maps; and - learning the depth prediction model by minimizing a second loss error including: • a reconstruction error determined by comparing the fourth filtered image to the seventh image and comparing the fifth filtered image to the sixth image, and • a consistency error determined by comparing the third and sixth depth charts and by comparing the fourth and fifth depth charts.

[0018]

[0019]

[0020]

[0021] According to a further variant of the process, the fifth and sixth depth maps are generated using the following function: / >( / -,))]) With : • Ps the coordinates of a pixel on the fifth depth map, respectively on the sixth depth map, • 77 a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K' is a reverse direction prediction model associated with the second camera, respectively with the first camera, • K is a direction prediction model associated with the first camera, and with the second camera respectively. • T is an extrinsic matrix of the stereoscopic vision system, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the fifth image, respectively fourth image, and of the depth ) which is associated with it in the fourth depth map, respectively third depth map. According to another variant of the process, the direction prediction model associated with a camera is an intrinsic matrix of the camera. According to yet another variant of the process, the direction prediction model is implemented by a neural network. According to a second aspect, the present invention relates to a device for determining depth by means of a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0022] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0023] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0024] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0025] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0026] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0027] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0028] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0029] Other features and advantages of the present invention will become apparent from the description of the specific and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 5, in which:

[0030] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0031] [Fig.2] illustrates a flowchart of the different stages of a process of determination of the depth of a pixel of an image by a depth prediction model associated with a vision system embedded in the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention;

[0032] [Fig.3] illustrates a flowchart of the different stages of a learning process of the depth prediction model used in the process of [Fig.2], according to a particular and non-limiting embodiment of the present invention;

[0033] [Fig.4] schematically illustrates a device configured to determine a depth of a pixel of an image by a neural network associated with a vision system embedded in the vehicle of [Fig. 1], according to a particular and non-limiting embodiment of the present invention; and

[0034] [Fig.5] illustrates a flowchart of the different stages of a learning process upstream of the depth prediction model used in the process of [Fig.2], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0035] A method and device for determining the depth of a pixel of an image by a neural network associated with a vision system embedded in a vehicle will now be described in what follows with joint reference to Figures 1 to 5. The same elements are identified with the same reference signs throughout the description that follows.

[0036] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0037] According to a particular and non-limiting embodiment of the present invention, a method for determining the depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system comprising several cameras including a color camera and an infrared camera, these cameras generating images in different and not directly comparable formats.

[0038] Indeed, the depth prediction model is learned in a learning phase comprising edge detection in a first and second image acquired by respectively the first color camera and the second infrared camera to obtain a first and second filtered image each having a channel, the determination of a first depth map associated with the second image from the first and second filtered images, the generation of a third image from the first image by adding black pixels then edge detection in the third image and the determination of a depth map from the third and second filtered images.

[0039] The depth prediction model is then learned by minimizing an error determined by comparing depth maps.

[0040] For the whole of the description, image reception or image generation means the reception of data representative of these images or the generation of data representative of these images.

[0041] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0042] Such an environment 1 corresponds, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0043] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, with electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a Land vehicle such as a car, truck, bus, motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say a vehicle operating according to a determined level of autonomy or under the total supervision of the driver.

[0044] The vehicle 10 advantageously comprises at least two on-board cameras, a first camera 11 and a second camera 12, configured to acquire images of a three-dimensional scene unfolding in the environment of the vehicle 10 from distinct observation positions. The first camera 11 and the second camera 12 form a stereoscopic vision system when used together, as illustrated in [Fig. 1]. The first camera 11 forms a monoscopic vision system when used alone, and similarly, the second camera 12 forms another monoscopic vision system when used alone. The present invention, however, extends to any vision system comprising at least two cameras, for example, 2, 3, or 5 cameras.

[0045] The intrinsic parameters of the first camera 11 characterize the transformation which associates, for an image point, hereafter called "point", its three-dimensional coordinates in the reference frame of the first camera 11 with the pixel coordinates in an image acquired by the first camera 11. These parameters do not change if the first camera 11 is moved. The intrinsic parameters of the first camera 11 include in particular a first focal length fl associated with the first camera 11.

[0046] The intrinsic parameters of the second camera 12 characterize, for their part, the transformation which associates, for an image point, its three-dimensional coordinates in the reference frame of the second camera 12 with the pixel coordinates in an image acquired by the second camera 12. These parameters do not change if the second camera 12 is moved. The intrinsic parameters of the second camera 12 include in particular a second focal length f2 associated with the second camera 12.

[0047] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account, and their correction is addressed during image acquisition or calibration.

[0048] These two cameras 11, 12 are arranged so that each acquires an image of a scene from a different viewpoint. The first viewpoint is, for example, located on or in the left-hand rearview mirror of vehicle 10 or at the top of the windshield of vehicle 10. The second viewpoint is, for example, located on or in the right-hand rearview mirror of vehicle 10 or at the top of the windshield of vehicle 10. If both cameras are located at the top of the windshield of the vehicle, they are then positioned at a certain distance. In this example, the first camera 11 is located at the top of the windshield of vehicle 10, and the second camera 12 is located in the right-hand rearview mirror of vehicle 10.

[0049] A first marker is associated with the first camera 11: - the direction of the x-axis is defined as horizontal and normal to the optical axis Cl of the first camera 11. The distance B separating the optical center of the first camera 11 from the projection of the optical center of the second camera 12 onto the horizontal plane passing through the optical center of the first camera 11 is called the reference base (in English "baseline"); - the direction of the y-axis is defined as vertical and normal to the optical axis Cl of the first camera 11; - The direction of the z-axis is defined as orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal coordinate system.

[0050] The optical axis Cl of the first camera 11 and the optical axis C2 of the second camera 12 are not necessarily parallel or even included in the same plane.

[0051] The extrinsic parameters related to the position of cameras 11, 12 are the following parameters: - three translations in the x, y, and z directions: Tx, Ty, and Tz, constituting the translation vector T; and - three rotations in the x, y and z directions: 0x, 0y and 0z.

[0052] An extrinsic matrix of the vision system then includes the extrinsic parameters previously defined.

[0053] The extrinsic parameters are determined, for example, during a calibration phase of the stereoscopic vision system comprising the first camera 11 and the second camera 12.

[0054] A key constraint of stereoscopic vision systems used in automobiles is, for example, the large distance between the two cameras. Indeed, to cover a measurement range of 200 meters, the reference base must be 60 cm for cameras commonly used in this field.

[0055] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 alone covering a first acquisition field 13, the second camera 12 alone covering a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

[0056] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0057] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system comprising the second camera 12.

[0058] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system comprising the first camera 11.

[0059] Finally, field 19 is not visible to any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0060] According to one particular embodiment, the field of view of the second camera 12 covers at least half of the field of view of the first camera 11.

[0061] It is evident that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently.

[0062] The first camera 11 is a color camera, meaning that it acquires RGB (Red Green Blue) images, representing the scene as it would be seen, for example, by the human eye. Thus, the pixels represent the scene in the visible spectrum. Such an RGB image comprises a set of pixels, each pixel in this set being coded on three channels, each channel corresponding to one of the primary colors mentioned above.

[0063] The second camera 12 is an infrared camera, meaning that it acquires images representing the scene from waves emitted by objects in the scene in the infrared range, i.e., waves whose wavelength is, for example, between 700 nm (seven hundred nanometers) and 100 pm (one hundred micrometers). Such an image comprises a set of pixels, each pixel in this set of pixels being encoded on a single channel.

[0064] Such an infrared camera has the advantage of being usable both day and night, that is to say, whether the observed three-dimensional scene is illuminated or not. A living being, for example, emits infrared waves through the radiation of its warm body. It is then visible to the infrared camera, also called a thermal camera, even when it is in a shaded area.

[0065] Each pixel of an acquired image represents an object in the three-dimensional scene present in the camera's field of view. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a point of light resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, which are captured by a photosensitive sensor of the camera after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image that faithfully represents the physical object observed by the camera. An image point described above is therefore a point on the surface of an object in the three-dimensional scene.

[0066] The images acquired by cameras 11, 12 represent views of the same scene taken from different viewpoints, the camera positions being distinct. This scene includes, for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0067] According to a particular embodiment, the first camera 11 and / or the second camera 12 is of the "wide-angle" type, a wide-angle camera being, for example, equipped with a lens designed to acquire a representative image of a three-dimensional scene seen over a wider field of view than a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene unfolding in front of or around the wide-angle camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle α of the field of view of the wide-angle camera is, for example, equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, a field of view open at an angle of 45° or less.Such a wide-angle camera is, for example, a camera equipped with mirrors or a "fisheye" camera. Wide-angle lenses have a shorter focal length compared to standard lenses, making them suitable for image acquisition. Images of landscapes, architecture, road intersections, or any other subject requiring a wide perspective. Wide-angle cameras, for example, are used to capture immersive and dynamic images with an extended depth of field.

[0068] According to a particular embodiment, an image acquired by the first camera 11 and / or an image acquired by the second camera 12 includes a distortion equal to 0.5%, 0.8%, or greater than 1%. The measurement of such distortion corresponds to determining a ratio between: - the maximum spacing of a pixel in the image of a straight line in the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the image edge or at the corners of the image edge, and - the length of this edge.

[0069] In the world of photography, distortion is commonly considered to be: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • bothersome if it is greater than or equal to 1% or more.

[0070] Barrel distortion is characterized by a positive percentage, while crescent distortion is characterized by a negative percentage.

[0071] According to a particular embodiment, a field of view of the first camera 11 covers at least half of a field of view of the second camera 12, and a field of view of the second camera 12 covers at least half of a field of view of the first camera 11. In other words, more than half of the pixels of an image acquired by the first camera 11 correspond to an object in the three-dimensional scene seen by the second camera 12, and pixels of an image acquired by the second camera 12 also correspond to this object in the three-dimensional scene. Similarly, more than half of the pixels of an image acquired by the second camera 12 correspond to an object in the three-dimensional scene seen by the first camera 11, and pixels of an image acquired by the first camera 11 also correspond to this object in the three-dimensional scene.

[0072] The images acquired by the first camera 11 and by the second camera 12 are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0073] A method for determining depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, i.e. by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer or computers in charge of the vision system of the vehicle 10.

[0074] Figure 2 illustrates a flowchart of the different steps of a process 2 of Determining the depth of a pixel in an image using a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, for example in vehicle 10 of [Fig. 1], according to a particular and non-limiting embodiment of the present invention. Method 2 is implemented, for example, by a device of the vision system embedded in vehicle 10 or by device 4 of [Fig. 4].

[0075] In a step 21, an image acquired by the first camera 11 and an image acquired by the second camera 12 are received.

[0076] In step 22, contours are detected in each of the received images by applying a feature filter to obtain two filtered images, each having a single channel. Details of the feature filter applied to the images acquired by the cameras of the stereoscopic vision system are presented below in detection step 32, described in relation to learning method 3.

[0077] In a step 23, depths associated with a set of pixels of one of the received images are determined by the depth prediction model from the two filtered images.

[0078] Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object in the three-dimensional scene to which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

[0079] If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another road user, the ADAS is then able to determine this distance precisely. For example, if the ADAS's function is to activate the braking system of the vehicle 10 in the event of a risk of collision with another road user, and the distance between the vehicle 10 and that same road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of the vehicle 10 to avoid a possible accident.

[0080] According to a particular embodiment, the learning process 3 further comprises an upstream learning phase. This upstream learning phase is included in an upstream learning process 5, this upstream learning process 5 being described with regard to [Fig.5].

[0081] The upstream learning process 5 is implemented prior to the learning process 3. Indeed, this upstream learning process 5 makes it possible to improve the reliability of depth prediction by the depth prediction model under normal operating conditions without favoring one camera over the other. The learning process 3 then makes it possible to develop the stereoscopic vision system's ability to predict depths from images acquired by the infrared camera, including when images acquired by the color camera are unusable, for example, when they have too many and / or too large areas of shadow or blackening.

[0082] In a step 51, a fourth image and a fifth image acquired by respectively the first camera 11 and the second camera 12 at the same time instant of acquisition are received.

[0083] According to a particular embodiment, the fourth and fifth images are of the same definition, that is to say they have the same number of pixels, have the same number of pixels according to their height and the same number of pixels according to their width.

[0084] According to another particular embodiment, the fourth and fifth images are not of the same resolution. An additional step then consists of resizing or cropping them to obtain a fourth and fifth image of the same resolution.

[0085] In a step 52, a fourth, respectively fifth, single-channel filtered images are obtained by applying edge detectors to the fourth, respectively fifth, images.

[0086] The fourth filtered image has a single channel and is obtained by applying a first edge detector to the fourth image. The fifth filtered image also has a single channel and is obtained by applying a second edge detector to the fifth image.

[0087] Such edge detectors include, for example, a Canny filter. Indeed, a Canny filter is commonly used in image processing to detect edges and is applicable to both an RGB image, like the first image, and an image with a single channel, like the second image. Only the input parameters of the edge detector differ, as the images acquired by the first and second cameras are heterogeneous.

[0088] Each of the fourth and fifth filtered images then includes white pixels representing the identified geometric features, here contours or edges, a white pixel having a value equal to 1, while the rest of the filtered images include black pixels, a black pixel having a value of zero.

[0089] In a step 53, a third depth map comprising depths associated with a set of pixels from the fourth image and a fourth depth map comprising depths associated with a set of pixels from the fifth image are determined, the depths being predicted with the depth prediction model from the filtered fourth and fifth images.

[0090] The depth prediction model, implemented by a convolutional neural network, is known to those skilled in the art and is presented for example in the document "UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos" by Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang and Wei Xu published in June 2019, adapted to a stereoscopic vision system including cameras whose optical axes are contained in the same plane, or in the document "Unifying Flow, Stereo and Depth Estimation" written by Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao and Andreas Geiger, published in July 2023, adapted for any stereoscopic vision system including those including cameras whose optical axes are not contained in the same plane.

[0091] Thus, a predicted depth is associated with each pixel of a set of pixels of the fourth image, the third depth map comprising the coordinates of each pixel of the set of pixels of the fourth image and an associated depth recorded for example in a lookup table, in a memory accessible to the processor implementing this upstream learning process 5. Similarly, a predicted depth is associated with each pixel of a set of pixels of the fifth image, the fourth depth map comprising the coordinates of each pixel of the set of pixels of the fifth image and an associated depth recorded for example in the lookup table previously described.

[0092] In a step 54, fifth and sixth depth maps are determined from the third and fourth depth maps and extrinsic parameters of the stereoscopic vision system.

[0093] The generation of the fifth depth map from the third depth map consists of: • the determination of spatial coordinates in the three-dimensional scene, in a reference frame associated with the first camera 11, of a point associated with a first pixel of the third depth map, from the coordinates of the first pixel in the fourth image and the depth associated with this first pixel, its coordinates and depth being recorded in the third depth map. It should be noted that the coordinates of the first pixel in the fourth image comprise two components while the spatial coordinates of the point associated with the first pixel comprise three components; • the determination of spatial coordinates in the three-dimensional scene of the point associated with the first pixel in a reference frame associated with the second camera 12, in other words a change of reference frame from that associated with the first camera 11 to that associated with the second camera 12, using the extrinsic parameters of the stereoscopic vision system; and • The projection of the point associated with the first pixel onto the image plane of the second camera 12, this image plane corresponding to that of the fifth image, allows us to determine the arrival coordinates of the first pixel of the fourth image in an image as the second camera would have acquired it. Note that the image plane of a camera corresponds to a plane defined in the camera's frame of reference, normal to the camera's optical axis and located at the camera's first focal length. The fifth depth map then includes the arrival coordinates of each first pixel and the depth associated with each first pixel.

[0094] Similarly, the generation of the sixth depth map from the fourth depth map consists of: • the determination of spatial coordinates in the three-dimensional scene, in a reference frame associated with the second camera 12, of a point associated with a second pixel of the fourth depth map, from the coordinates of the second pixel in the fifth image and the depth associated with this second pixel, its coordinates and depth being recorded in the fourth depth map. It should be noted that the coordinates of the second pixel in the fifth image comprise two components while the spatial coordinates of the point associated with the second pixel comprise three components; • the determination of spatial coordinates in the three-dimensional scene of the point associated with the second pixel in a reference frame associated with the first camera 11, in other words a change of reference frame from that associated with the second camera 12 to that associated with the first camera 11, using the extrinsic parameters of the stereoscopic vision system; and • the projection of the point associated with the second pixel in the image plane of the first camera 11, this image plane corresponding to that of the fourth image, allowing to determine arrival coordinates of the second pixel of the fifth image in an image such as the first camera would have acquired it. The sixth depth map then includes the arrival coordinates of every second pixel and the depth associated with each second pixel.

[0095]

[0096]

[0097] According to a particular implementation example, the fifth and sixth depth maps are generated using the following function: [Math.l] ) ] ) With : • Ps the arrival coordinates of a pixel on the fifth depth map, respectively from the sixth depth map, •77 a function to convert from homogeneous coordinates to pixel coordinates in removing one dimension from a vector, • K' is a reverse direction prediction model associated with the second camera (12), respectively to the first camera (11), • K is a direction prediction model associated with the first camera (11), respectively to the second camera (12), • T is an extrinsic matrix of the stereoscopic vision system, and • 0 a projection function in the three-dimensional scene of a pixel Pt in function of its coordinates in the fifth image, respectively fourth image, and of the depth ) which is associated with it in the fourth card of

[0098]

[0099]

[0100] depths, respectively third depth map. According to one variant, the direction prediction model associated with a camera is an intrinsic matrix of the camera. Such a variant is particularly applicable in the case of pinhole cameras generating images with little distortion. According to a second variant, the direction prediction model is implemented by a neural network. This model is better suited to wide-angle cameras that generate highly distorted images. Such a direction prediction model is known to those skilled in the art; it is notably presented in the paper "Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion" by Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich, and Adrien Gaidon, published in August 2020. The projections and reprojections are then inverse functions obtained from the direction prediction model and are a function of depth and pixel coordinates. Note that the extrinsic matrix is ​​not the same for generating the two images. Indeed, a first extrinsic matrix allows the transition from a reference frame associated with the first camera 11 to a reference frame associated with the second camera 12 during the generation of the sixth depth map, while a second extrinsic matrix allows the transition from a reference frame associated with the second camera 12 to a reference frame associated with the first camera 11 during the generation of the fifth depth map.

[0101] Projections and reprojections are inverse functions obtained from the direction prediction model and are a function of a depth; such a projection model is notably presented in the document "Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion".

[0102] In a step 55, sixth and seventh images are generated.

[0103] The sixth image is generated from the filtered fourth image and from the arrival coordinates of a pixel of the sixth depth map, a pixel of the sixth image having the value of a pixel of the filtered fourth image and the coordinates in the sixth image being the arrival coordinates associated with the pixel of the fourth image.

[0104] The seventh image is generated from the filtered fifth image and from the arrival coordinates of a pixel of the fifth depth map, a pixel of the seventh image having the value of a pixel of the filtered fifth image and the coordinates in the seventh image being the arrival coordinates associated with the pixel of the fifth image.

[0105] In a step 56, the depth prediction model is learned by minimizing a second loss error comprising: • a reconstruction error determined by comparing the fourth filtered image to the seventh image and comparing the fifth filtered image to the sixth image, and • a consistency error determined by comparing the third and sixth depth charts and by comparing the fourth and fifth depth charts.

[0106] The reconstruction error includes, for example, a first component determined by comparing pixels of the fourth and seventh images and by comparing pixels of the fifth and sixth images.

[0107] According to a first particular embodiment, the first component is representative of photometric errors as presented in the document "Digging Into Self-Supervised Monocular Depth Estimation" by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019 and are determined by the following function:

[0108] [Math.2] L^p) - EJ (1-a) • ] / ( p)-Hp) \+a\\-{SSIM(l(pY ï{p) ) ) ]

[0109] With: • L*(p) the first photometric error denoted LA p) respectively the second photometric error denoted L2(p), P being a pixel defined by its coordinates in an image, • l(p) a value of pixel p in the fourth image, respectively fifth image, * l(p) a value of pixel p in the seventh image, respectively sixth image, • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment.

[0110] According to a second particular embodiment, the reconstruction error includes a second component determined, furthermore, by the following function: [YES] [Math.3]

[0112] With: j a second component for a pixel p of the seventh image, respectively of the sixth image, • D(p{^ is a depth of one pixel pt obtained from the fifth depth map, respectively obtained from the sixth depth map; • VF is a parameter matrix; • ° is the order of a smoothing gradient; • an L1 norm of the second order depth gradients is calculated with W = 1, and ° = 2; • x and V are the dimensions of the images; • P is an environment-dependent hyperparameter; and * P is a pixel value pt in the seventh image, respectively sixth image.

[0113] This second function is generally used to deal with the discontinuity at the edge of objects (in English "edge aware smoothness").

[0114] The second loss error also includes a consistency error and is determined by the following function: [Math.4] <£>.,( / >)-Z>6 <p) )2+                  ) 2 )

[0115] With: • The consistency error, * D^p) a depth of a pixel P obtained from the third depth map for a pixel p, • D^p) a depth of a pixel P obtained from the sixth depth map for a pixel P, * D4(p) a depth of one pixel obtained from the fourth depth map for a pixelp, and * D^p) a depth of one pixel p obtained from the fifth depth map for one pixel p. It should be noted that the pixels compared are those with similar or equal coordinates in an acquired image and in a generated image.

[0116] Learning the depth prediction model consists of adjusting the input parameters of the convolutional neural network associated with the depth prediction model in order to minimize the second loss error, the second loss error comprising the reconstruction error and the consistency error. The second loss error is, for example, the sum of the reconstruction error and the consistency error.

[0117] Thus, the depth prediction model used for the depth prediction of a pixel of an image acquired by the first camera 11 or by the second camera 12 is made more reliable by this learning process.

[0118] Figure 3 illustrates a flowchart of the different stages of a method for learning the depth prediction model used in a method for determining the depth of a pixel of an image, for example in method 2 of Figure 2, according to a particular and non-limiting embodiment of the present invention.

[0119] The learning process 3 is for example implemented by the device on board the vehicle 10 implementing the depth determination process by vehicle-mounted vision system or by the device 4 of the [Fig.4],

[0120] In a step 31, a first image and a second image are received, the first image being acquired by the first camera 11 at an acquisition time instant and the second image being acquired by the second camera at the same acquisition time instant.

[0121] According to a particular embodiment, the first image and second image are of the same definition, that is to say they have the same number of pixels, have the same number of pixels according to their height and the same number of pixels according to their width.

[0122] According to another particular embodiment, the first and second images are not of the same resolution. An additional step then consists of resizing or cropping them to obtain a first and second image of the same resolution.

[0123] In a step 32, a first filtered image having a single channel is obtained by applying a first edge detector to the first image, and a second filtered image having a single channel is obtained by applying a second edge detector to the second image. Such edge detectors are, for example, a Canny filter. Only the input parameters of the edge detector differ, the images acquired by the first 11 and the second 12 cameras being heterogeneous.

[0124] Each of the first and second filtered images then includes white pixels representing the identified geometric features, here contours or edges, a white pixel having a value equal to 1, while the rest of the filtered images include black pixels, a black pixel having a value of zero.

[0125] In a step 33, a first depth map comprising depths associated with a set of pixels of the second image is determined, the depths being predicted with the depth prediction model from the first and second filtered images.

[0126] The depth prediction model, implemented by a convolutional neural network, is for example one of those presented in step 53 previously described.

[0127] Thus, a predicted depth is associated with each pixel of a set of pixels of the second image, the first depth map comprising the coordinates of each pixel of the set of pixels of the second image and an associated depth recorded for example in a lookup table, in a memory accessible to the processor implementing this learning process 3.

[0128] In a step 34, a third image is generated from the first image, the third image comprising a set of black pixels obtained by modifying the values ​​of a set of pixels from the first image and a third single-channel filtered image is obtained by applying an edge detector to the third generated image.

[0129] As a reminder, as with filtered images, a black pixel is a pixel with a value of zero.

[0130] The portion of the third image covered by the set of black pixels varies randomly at each iteration of the steps of the learning process 3, the area covered by the set of black pixels representing, for example, 10% to 100% of the third image. The set of pixels covers, for example, 10% of the area of ​​the third image at a first iteration, growing at each iteration until it covers the entire third image. The area of ​​the third image covered by the set of pixels has, for example, a rectangular or circular shape and is positioned randomly within the third image.

[0131] Adding black pixels to the third image simulates an image acquired by the first camera in a dark environment. Indeed, the black pixels represent a three-dimensional scene, part of which is unlit; the unlit part of the three-dimensional scene corresponds to the area covered by the set of black pixels.

[0132] In a step 35, a second depth map comprising depths associated with a set of pixels from the second image is determined, the depths being predicted with the depth prediction model from the filtered third and second images. This step is thus similar to step 33.

[0133] In a step 36, the depth prediction model is learned by minimizing a loss error determined by comparing the first and second depth maps.

[0134] The training of the depth prediction model consists of adjusting the input parameters of the convolutional neural network associated with the depth prediction model in order to minimize the previously calculated loss error.

[0135] According to a particular embodiment, the first error is determined by the following function: [Math.5] L=ï,f(D(p)-D'(p) )2)

[0136] With: • The loss error, • D(p) a depth of one pixel p of the second image obtained from the first depth map, and • D'(p) a depth of pixel p of the second image obtained from the second depth map for pixel p.

[0137] Progressively increasing the size or area of ​​the region covered by the set of black pixels allows the depth prediction model to learn to accurately predict the depth of a pixel corresponding to an object in the dimly lit three-dimensional scene. Indeed, the learning of the depth prediction model under dim lighting conditions in the three-dimensional scene is thus supervised by the depth predictions corresponding to the same scene under good lighting. Therefore, when only the second camera 12 is able to perceive objects in a three-dimensional scene, the predicted depths associated with pixels corresponding to these objects are more accurate. The learned depth prediction model is then able to predict accurate depths under real-world conditions when the scene is dimly lit, for example, when the vehicle 10 is moving at night.

[0138] The learning process 3, as well as the upstream learning process 5, use data acquired by the on-board vision system and therefore do not require data annotated by another on-board system or the storage of a training image library. Furthermore, the training data is representative of the data received when the system is in operation or in production; indeed, the training data is representative of real-world environments in which the vehicle carrying the vision system moves, making this training data particularly relevant.

[0139] Edge detectors applied to the images acquired by the vision system cameras, which are heterogeneous, make them homogenized and thus comparable. The depth prediction model is therefore applicable to the filtered images and makes it possible to accurately predict the depth of a pixel in an acquired image, particularly by the infrared camera of the stereoscopic vision system, that is to say, the distance separating the vehicle carrying the infrared camera from a physical object in the three-dimensional scene associated with that pixel.

[0140] Figure 4 schematically illustrates a device 4 configured for determining depth by a vision system mounted in a vehicle 10, according to a particular and non-limiting embodiment of the present invention. The device 4 corresponds, for example, to a device mounted in the first vehicle 10, for example a computer associated with the stereoscopic vision system.

[0141] Device 4 is, for example, configured to carry out the operations described opposite Figures 1 and 4 and / or the steps described opposite Figures 2 and 3. Examples of such a device 4 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of device 4, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 4 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0142] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41, corresponding, for example, to volatile and / or non-volatile memory, and / or includes a memory storage device which may include memory volatile and / or non-volatile, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0143] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 41.

[0144] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0145] According to a particular and non-limiting embodiment, the device 4 includes a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 include one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HD MI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0146] According to another particular and non-limiting embodiment, the device 4 includes a communication interface 43 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 430. The communication interface 43 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0147] According to a particular and non-limiting embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch or not, one or more speakers 450 and / or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or more of the external devices is integrated into the device 4.

[0148] Of course, the present invention is not limited to the embodiments described above but extends to a method for measuring the distance between an object and a vehicle equipped with a vision system that acquires images, which would include secondary steps without falling outside the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0149] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 4 of [Fig.4].

Claims

1. Demands Method for determining the depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising a first camera (11) and a second camera (12) arranged so as to each acquire an image of a three-dimensional scene from different viewpoints, the first camera being configured for color image acquisition and the second camera (12) being configured for infrared image acquisition, said method being implemented by at least one processor, and being characterized in that the depth prediction model is learned in a learning phase comprising the following steps: - reception (31) of a first image acquired by the first camera (11) and a second image acquired by the second camera (12) at the same time instant of acquisition; - obtaining (32) a first, respectively second, single-channel filtered images by applying edge detectors to the first, respectively second, images; - determination (33) of a first depth map comprising depths associated with a set of pixels from the second image, the depths being predicted with the depth prediction model from the first and second filtered images; - generation (34) of a third image from the first image, the third image comprising a set of black pixels obtained by modifying the values ​​of a set of pixels from the first image and obtaining a third single-channel filtered image by applying an edge detector to the third generated image; - determination (35) of a second depth map comprising depths associated with a set of pixels from the second image, the depths being predicted with the depth prediction model from the third and second filtered images; - learning (36) of the depth prediction model by minimizing a loss error determined by comparing the first and second depth maps.

2. A method according to claim 1, wherein the loss error is determined by the following function: t=EP(O(p)-D(p))2) With: • L the loss error, • I)(p) a depth of a pixelp obtained from the first depth map for a pixel p, and • D'(p) a depth of a pixel p obtained from the second depth map for the pixel p.

3. A method according to claim 1 or 2, wherein one of said edge detectors is a Canny filter.

4. A method according to any one of claims 1 to 3, further comprising an upstream learning phase including the following steps: - receiving (51) a fourth image and a fifth image acquired by the first camera (11) and the second camera (12) respectively at the same acquisition time instant; - obtaining (52) a fourth, and respectively a fifth, single-channel filtered images by applying edge detectors to the fourth, and respectively a fifth, images; - determining (53) a third depth map comprising depths associated with a set of pixels from the fourth image and a fourth depth map comprising depths associated with a set of pixels from the fifth image, the depths being predicted with the depth prediction model from the fourth and fifth filtered images;- determination (54) of the fifth and sixth depth maps from the third and fourth depth maps and extrinsic parameters of the stereoscopic vision system; - generation (55) of the sixth and seventh images from the filtered fourth and fifth images and from pixel arrival coordinates of the sixth and fifth depth maps; and; - learning (56) of the depth prediction model by minimizing a second loss error comprising: • a reconstruction error determined by comparing the fourth filtered image to the seventh image and comparing the fifth filtered image to the sixth image, and • a consistency error determined by comparing the third and sixth depth maps and by comparing the fourth and fifth depth maps.

5. A method according to claim 4, wherein the fifth and sixth depth maps are generated using the following function: ^(^(pjA-.Dfp,))]) With: • Ps the coordinates of a pixel of the fifth depth map, respectively of the sixth depth map, •77 a function to go from homogeneous coordinates to pixel coordinates by removing one dimension of a vector, • K' an inverse direction prediction model associated with the second camera (12), respectively with the first camera (11), • K a direction prediction model associated with the first camera (11), respectively with the second camera (12), • T an extrinsic matrix of the stereoscopic vision system, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the fifth image, respectively fourth image, and of the depth ) which is associated with it in the fourth depth map, respectively third depth map.

6. A method according to claim 5, wherein the direction prediction model associated with a camera is an intrinsic matrix of the camera.

7. A method according to claim 5, wherein the direction prediction model is implemented by a neural network.

8. A computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by a processor.

9. 30 Device (4) for determining depth by means of a vision system mounted in a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for carrying out the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images