Determining depth information of a scene from a pattern projection

Abstract

The invention relates to a system for determining depth information of a scene (10) facing a motor vehicle (100), the system comprising:- at least one light module (101.1; 101.2) arranged to project towards the scene a light beam (120.1; 120.2) in a first wavelength range not comprising any visible wavelength, the light module comprising a pixelated light source and being able to control said pixelated light source to project the light beam according to a light pattern;- at least one camera (102) able to acquire at least one image in the first wavelength range and representative of a projection of the light pattern in the scene facing the vehicle;- an image processing module (104) configured to determine depth information of the scene from said at least one image. FIG. 1a

Description

Title of the invention: Determining depth information of a scene from a pattern projection

[0001] The present invention relates to the field of image processing, in particular, for determining depth information in a scene represented in an image. More specifically, the invention relates to a system and a method for determining depth information of a scene facing a vehicle, such as a motor vehicle for example.

[0002] Estimating the distance of an object in a scene facing a vehicle, such as a motor vehicle or a motorized two-wheeled vehicle, is essential to many driver assistance functions, also called AD AS, for “Advanced Driver Assistance Systems” in English. Such functions make it possible to assist the driver of a vehicle in steering the vehicle, or even to completely control certain driving parameters without requiring driver intervention in the case of autonomous vehicles.

[0003] In order to estimate the distance of objects in the scene facing the vehicle, it is known to use a vehicle sensor of the Lidar type in particular. Such a sensor allows a precise evaluation of the object distance in the scene.

[0004] However, it is preferable, or even necessary in the case of autonomous vehicles, to ensure redundancy in estimating the distance of objects, since this information is used by AD AS functions ensuring control of the vehicle. Such redundancy thus makes it possible to ensure robustness in estimating the distance of an object, and makes it possible to ensure better safety in the execution of the driving assistance function.

[0005] Since a lidar sensor is expensive, it is preferable not to ensure such redundancy by equipping the vehicle with a second lidar sensor.

[0006] It is thus desirable to have data representative of the distance of objects in a scene facing the vehicle, preferably without adding a new sensor in the vehicle.

[0007] Furthermore, the desired solution must make it possible to obtain data representative of the distance of objects, including in night scenes or in poor ambient light conditions. Furthermore, it is preferable that such a solution does not disturb the driver when driving the vehicle, when the vehicle is not an autonomous vehicle, and does not disturb other users of the road on which the vehicle is traveling.

[0008] The present invention improves the situation.

[0009] A first aspect of the invention relates to a system for determining depth information of a scene facing a motor vehicle, the system comprising: - at least one light module arranged to project towards the scene a light beam in a first wavelength range not comprising any visible wavelength, the light module comprising a pixelated light source and being capable of controlling said pixelated light source to project the light beam according to a light pattern; - at least one camera capable of acquiring at least one image in the first wavelength domain and representative of a projection of the light pattern in the scene facing the vehicle; - an image processing module configured to determine depth information of the scene from said at least one image.

[0010] Thus, it is made possible to determine depth information of a scene, which can be used by a vehicle function such as a driver assistance function. Such a determination is made possible from a single camera, which is natively present in most vehicles now available. Projecting a light pattern into the scene makes it possible to determine the depth information without having to have a two-camera system based on the principles of stereovision. Furthermore, since the light pattern is produced by a light beam in a wavelength range that is not visible, the determination of the depth information is done in a manner that is imperceptible to the driver of the vehicle and to other road users.The light pattern may be selected from a set of predetermined light patterns associated with controls stored in the vehicle, the selection being implemented based on contextual information related to driving, or may be a single predetermined light pattern. Alternatively, the light pattern is determined directly based on information descriptive of the vehicle environment and / or the driving situation.

[0011] According to embodiments, the system may comprise a right light module and a left light module capable of projecting a right light beam and a left light beam in the first wavelength range according to the light pattern.

[0012] Thus, the same light pattern is projected by the right and left light modules, which improves the contrast associated with the projection of the light pattern, and thus improves the determination of depth information by the image processing module.

[0013] Alternatively, the system may comprise a straight light module capable of projecting a straight light beam according to a first light pattern in the first domain of wavelengths and a left light module capable of projecting a left light beam according to a second light pattern in the first wavelength domain, said at least one camera may be capable of obtaining said at least one image in the first wavelength domain and representative of the projection of the first light pattern and of the second light pattern in the scene facing the vehicle.

[0014] Thus, the robustness associated with the determination of depth information by the image processing module is improved.

[0015] According to embodiments, the at least one light module may also be capable of projecting another light beam in the visible range or in a second wavelength range separate from the first wavelength range and not comprising any visible wavelength.

[0016] Thus, the light module can be pooled to carry out projections of light beams in two distinct wavelength domains, which reduces the size in a constrained space such as a motor vehicle headlight.

[0017] In addition, the at least one light module may be capable of projecting the other light beam in the visible range to perform a lighting, signaling or driving assistance function.

[0018] Thus, the projection of the light pattern in the first wavelength range can be implemented by a light module such as a lighting or signaling module already present in all motor vehicles. Alternatively, the at least one light module may be capable of projecting the other light beam in the visible range or in the second wavelength range, according to the light pattern or according to another light pattern. The camera may further be capable of acquiring at least one other image in the visible range or in the second wavelength range, and representative of the projection of the light pattern or the other light pattern in the scene. The image processing module may be configured to determine the depth information from the at least one image in the first visible range and from the at least one other image.

[0019] Thus, the robustness associated with the determination of depth information is improved.

[0020] According to embodiments, the pixelated light source of the at least one light module may be a matrix of light elements individually controllable by a control unit of the at least one light module, the matrix comprising a first set of light elements capable of emitting light rays in the first wavelength range and a second set of light elements capable of emitting light rays in the visible range or in the second wavelength range.

[0021] Thus, a single matrix source with individually controllable light elements can be used to form two light beams in two distinct wavelength ranges, thereby reducing the costs associated with implementing the invention and reducing the space requirement in the vehicle headlight. Furthermore, such a solution makes it possible to form the beams in the two distinct wavelength ranges simultaneously.

[0022] Alternatively, the pixelated light source of the at least one light module may comprise a matrix of micro-mirrors, each micro-mirror being individually controllable by a control unit of the at least one light module, the pixelated light source further comprising a first light source capable of emitting light rays in the first wavelength range towards the matrix of micro-mirrors and a second light source capable of emitting light rays in the visible range or in the second wavelength range towards the matrix of micro-mirrors.

[0023] Thus, a single pixelated light source can be used to form two light beams in two distinct wavelength ranges, thereby reducing the costs associated with implementing the invention and reducing the space requirement in the vehicle headlight. The projection of the light beams in the two wavelength ranges can be alternated, with a given frequency. When the second light source emits in the visible range, the given frequency can be greater than 700 Hz so that the alternation is not visible to the human eye.

[0024] As a further variant, the pixelated light source may be laser-scanned and may comprise a mirror arranged to scan a set of predetermined positions of the scene, a first laser light source capable of emitting light rays in the first wavelength range towards the mirror, and a second laser light source capable of emitting light rays in the visible range or in the second wavelength range towards the mirror.

[0025] Thus, a single pixelated light source can be used to form two light beams in two distinct wavelength ranges, thereby reducing the costs associated with implementing the invention and reducing the space requirement in the vehicle headlight. When the second light source emits in the visible range, the given frequency can be greater than 700 Hz so that the alternation is not visible to the human eye.

[0026] According to embodiments, the light pattern may comprise dark areas and illuminated areas in the first wavelength range.

[0027] Such a light pattern presenting a plurality of zones highly contrasted with each other, allows the image processing module to easily identify the deformation of the light pattern by the scene, and therefore to accurately determine the depth information of the scene.

[0028] Additionally, the dark and light areas may have identical shapes and sizes and may form a checkerboard pattern.

[0029] Such a light pattern allows for precise determination of depth information of the scene, while being associated with simple control of the pixelated light source.

[0030] According to embodiments, the image processing module can implement an artificial neural network capable of receiving said at least one image as input, and of predicting the depth information of the scene as output.

[0031] Thus, the image processing module can be based on a model derived from machine learning, which allows great robustness and precision in determining the depth information of the scene.

[0032] In addition, the artificial neural network can be a U-Net network.

[0033] Such an artificial neural network structure is particularly suitable for predicting depth information and is capable of taking one or more images as input.

[0034] A second aspect of the invention relates to a method for determining depth information of a scene facing a vehicle comprising the following steps: - projection towards the scene, by at least one light module of the vehicle, of a light beam in a first wavelength range, according to a light pattern, the first wavelength range not including any visible wavelength; - acquisition, by at least one camera, of at least one image in the first wavelength domain and representative of a projection of the light pattern in the scene facing the vehicle; - determination, by an image processing module, of depth information of the scene from said at least one image.

[0035] Other characteristics and advantages of the invention will appear on examining the detailed description below, and the appended drawings in which:

[0036] [Fig.la] illustrates a system for determining depth information of a scene facing a vehicle, according to embodiments of the invention;

[0037] [Fig.lb] illustrates a light module of a system for determining depth information of a scene facing a vehicle, according to embodiments of the invention;

[0038] [Fig.2a] illustrates a light pattern projected by a lighting module of a system for determining depth information of a scene facing a vehicle, according to embodiments of the invention;

[0039] [Fig.2b] illustrates an image acquired by a camera following the projection of a pattern luminous in a scene facing a vehicle, according to embodiments of the invention;

[0040] [Fig.3] illustrates the structure of an image processing module according to modes of realization of the invention;

[0041] [Fig.4a] illustrates a pixelated light source of a light module of a system determining depth information of a scene facing a vehicle, according to a first embodiment;

[0042] [Fig.4b] illustrates a pixelated light source of a light module of a system determining depth information of a scene facing a vehicle, according to a second embodiment;

[0043] [Fig.5] is a diagram illustrating the steps of a determination method depth information of a scene facing a vehicle, according to embodiments of the invention.

[0044] The description focuses on the characteristics that distinguish the system and method from those known in the state of the art.

[0045] [Fig. 1a] illustrates a system for determining depth information of a scene 10 facing a vehicle 100, according to embodiments of the invention.

[0046] The vehicle 100 comprises a right front headlight 105.1 and a left front headlight 105.2. The right front headlight 105.1 comprises at least one first right light module 101.1 and the left front headlight comprises at least one light module 101.2.

[0047] According to the invention, at least one of the light modules 101.1 and 101.2 is capable of projecting at least one light beam producing a light pattern in the scene 10, in a first wavelength range distinct from the visible range. The two light modules 101.1 and 101.2 may be capable of projecting light beams producing the same light pattern, by superimposing the respective beams, or producing two distinct light patterns, for example a first light pattern and a second light pattern, in a synchronized manner (simultaneously) or during separate time intervals.

[0048] A right light beam 120.1 is projected by the right light module 101.1, the right light beam 120.1 being able to comprise several light beams in several distinct wavelength domains, including the first wavelength domain, as will be better understood upon reading the following. Similarly, a left light beam 120.2 is projected by the left light module 101.2, the left light beam 120.2 being able to comprise several light beams in several distinct wavelength domains, including the first wavelength domain, as will be better understood upon reading the following.

[0049] According to the invention, the first wavelength domain is outside the visible domain, and therefore does not include any wavelength in the visible domain. No restriction is attached to the first wavelength domain according to the invention, which may be an infrared or ultraviolet domain for example. For example, the first wavelength domain may be included in a near infrared domain, also called NIR, for “Near InfraRed” in English, in a shortwave infrared domain, also called SWIR, for “Short Wave InfraRed”, in a mediumwave infrared domain, also called MWIR, for “Medium Wave InfraRed”, or in a longwave infrared domain, also called LWIR, for “Long Wave InfraRed” in English. Note that the MWIR and LWIR domains are also called the thermal domain.

[0050] In the following, for illustrative purposes, and unless otherwise stated, it is considered that the first wavelength range is in the infrared range.

[0051] In certain embodiments, and as explained in the following, the light module 101.1 and / or 101.2 may further be capable of projecting a light beam in the visible range, in addition to the light beam in the first wavelength range, to produce another light pattern according to the invention, or to perform a lighting or signaling function. For example, the light module 101.1 and / or 101.2 may project, alternately according to a given frequency, or simultaneously, the light beam in the first wavelength range and the light beam in the visible range.

[0052] Furthermore, in certain embodiments, and as explained in the following, the light module 101.1 and / or 101.2 may further be capable of projecting a light beam in a second wavelength range distinct from the visible range, the second wavelength range being different, preferably disjoint from the first wavelength range. For example, the first wavelength range may be in the infrared and the second wavelength range may be in the ultraviolet. The projection of the light beam in the second wavelength range may be according to the same light pattern or according to another light pattern, within the framework of the determination of depth information of the scene facing the vehicle, as described in the following, or may be dedicated to a function of detecting objects in the scene.

[0053] According to other embodiments, the light module 101.1 and / or 101.2 may be capable of projecting during the same time interval (simultaneously or by alternating the beams at a frequency corresponding to a period less than the time interval): - a light beam according to a light pattern in the first wavelength range; and - a light beam in the visible range either according to a light pattern (the same as in the first range or another light pattern) for determining depth information or performing a lighting or signaling function; and - a light beam in the second wavelength range either according to a light pattern (the same as in the first range or another light pattern) for determining depth information, or for performing an object detection function.

[0054] Preferably according to the invention, the light beam in the visible range performs a lighting, signaling or driving assistance function (to assist in the detection of objects for example), rather than the display of a light pattern for the determination of depth information. Thus, it is avoided to disturb the driver while driving the vehicle when determining depth information.

[0055] According to the invention, the light beams projected by the light module 101.1 and / or 101.2 are pixelated, which allows the light pattern to be produced with a resolution depending on the number of pixels permitted by a pixelated light source of the light module 101.1 and / or 101.2.

[0056] [Fig.lb] illustrates the structure of a light module 101 with a pixelated light source 111, for example a matrix light source, according to embodiments of the invention. The light module 101 may be the front right module 101.1 and / or the front left module 101.2 previously described.

[0057] The light module 101 comprises: - a control unit 110 of the pixelated light source 111; - the pixelated light source 111; - an optic for projecting the light from the pixelated light source 111 to produce a pixelated light beam projected in front of the vehicle towards the scene 10. No restriction is attached to the projection optic, which can comprise any set of optical elements.

[0058] No restriction is attached to the number of pixels of the pixelated light source 111. Preferably, the pixelated light source 111 is a high-definition light source, i.e. allowing the projection of a light beam comprising more than one hundred pixels, preferably more than 1000 pixels. The pixelated light source 111 may further allow the projection of a light beam comprising more than 10,000 pixels according to embodiments of the invention.

[0059] Furthermore, no restrictions are attached to the technology associated with the pixelated light source 111, which may be: - according to a first example, a matrix of individually controllable light elements, of which at least a first set of light elements is capable of emitting in the first wavelength range. As detailed below, according to embodiments of the invention, the same matrix of light elements may comprise a first set of light elements capable of emitting in the first wavelength range and a second set of light elements capable of emitting in the visible range. Alternatively, the second set of light elements comprises light elements capable of emitting in the second wavelength range; - according to a second example, a first light source capable of emitting in the first wavelength range and a micro-mirror matrix, also called DMD for Digital Micromirror Devices, individually activatable to reflect the light from the first light source towards the projection optics 112. In addition, the pixelated light source 111 may comprise a second light source capable of emitting in the visible range or in the second wavelength range, as described later. In this case, the micro-mirror matrix is ​​alternately controlled, at a given frequency, to produce a first beam when the first light source is activated, and to produce a second beam when the second light source is activated.Additionally, the pixelated light source 111 may further comprise a second light source capable of emitting in the visible range and a third light source capable of emitting in the second wavelength range. In this case, the micro-mirror array is alternately controlled at a given frequency, to produce a first beam when the first light source is activated, to produce a second beam when the second light source is activated, and to produce a third beam when the third light source is activated; or. - according to a third example, a first laser light source capable of emitting in the first wavelength range, and a controllable mirror for scanning a predetermined set of positions. Such technology is called laser scanning. The control element 110 is capable of synchronously controlling the movement of the mirror and the activation of the first laser light source. Furthermore, the pixelated light source 111 may comprise a second laser light source capable of emitting in the visible range or in the second wavelength range, as described later. In addition, the pixelated light source 111 may further comprise a second laser light source capable of emitting in the visible range and a third laser light source capable of emitting in the second wavelength range.In this case, the mirror is alternately controlled at a given frequency, to achieve a first . beam when the first light source is controlled, to produce a second beam when the second light source is controlled, and to produce a third beam when the third light source is controlled.

[0060] The three examples listed above have the advantage of allowing the production of a light pattern in a pixelated beam with high resolution. In the first example listed above, the light elements may be electroluminescent elements, individually controlled by a voltage applied to the terminals of each light element by the control unit 110. Each electroluminescent light element may be mounted on a substrate of its own. Alternatively, the electroluminescent light elements may be on the same substrate, in which case the pixelated light source is said to be monolithic.

[0061] A so-called "monolithic" source may have a particularly high density of light elements, which makes it particularly interesting for a plurality of applications. A monolithic source involves a plurality of electroluminescent semiconductor elements of submillimeter dimensions, epitaxially grown directly on a common substrate, the substrate generally being formed of silicon. Unlike sources comprising conventional LED matrices, in which each elementary light element is an electronic component produced individually and mounted on a substrate such as a printed circuit, PCB, a monolithic source is to be considered as a single electronic component, during the production of which several areas of electroluminescent semiconductor junctions are generated on a common substrate, in the form of a matrix.This production technique allows for the production of electroluminescent areas, each acting as an elementary luminous element, very close to each other. The gaps between the luminous elements can have submillimeter dimensions. An advantage of this production technique is the high level of pixel density that can result on a single substrate.

[0062] The first example has the advantage, when the pixelated light source comprises a first set of light elements and a second light element, of projecting two light beams simultaneously in the first wavelength range, and in the second wavelength range or in the visible range. Indeed, the two sets of light elements can be controlled separately by the control unit 110. According to embodiments of the first example, the pixelated light source comprises the first set of light elements, a second set of light elements capable of emitting light in the visible range and a third set of light elements capable of emitting light in the second wavelength range. Three beams can thus be projected simultaneously in three wavelength ranges. distinct waveforms. The resolution of each light beam depends on the number of light elements in the corresponding light element set.

[0063] In the second example listed above, the control unit 110 controls the micromirrors of the matrix to produce the light beam comprising the light pattern in the first wavelength range, when the first light source is active. The resolution of the light pattern then depends on the number of micromirrors of the matrix. Optionally, when the pixelated light source 111 comprises at least one second light source and when the second light source is active, the control unit 110 controls the micromirrors of the matrix to produce the light beam in the visible range (according to the same pattern, according to another pattern or to perform a lighting, signaling or driving assistance function) or to produce the light beam in the second wavelength range (according to the same pattern, according to another pattern or for the object detection function).

[0064] In the third example listed above, the control unit 110 controls the mirror and the first laser light source to scan the predetermined set of positions and produce the light beam comprising the light pattern in the first wavelength range. The resolution of the light pattern then depends on the number of positions scanned by the mirror. Optionally, when the pixelated light source 111 comprises at least one second laser light source, the control unit 110 controls the mirror and the second laser light source to produce the light beam in the visible range (according to the same pattern, according to another pattern or to perform a lighting, signaling or driving assistance function) or to produce the light beam in the second wavelength range (according to the same pattern, according to another pattern or for the object detection function).The same applies when the pixelated light source 111 comprises a third laser light source capable of illuminating in the second wavelength range.

[0065] Referring again to [Fig.1a], the vehicle 100 further comprises at least one camera 102 comprising a sensor capable of acquiring an image, or a series of images, of the scene 10 in the first wavelength domain. When the first wavelength domain is an infrared domain, the camera 102 may be a camera comprising at least one infrared sensor, for example a thermal camera.

[0066] According to advantageous embodiments, the camera 102 can comprise several sensors, including a first sensor capable of acquiring a first image, or a first series of images, in the first wavelength domain, and a second sensor capable of acquiring a second image, or a second series of images, in the second wavelength domain or in the visible domain. Alternatively, the vehicle 100 comprises several cameras 102: a first camera 102 capable of acquiring a first image, or a first series of images, in the first wavelength domain, and a second camera 102 capable of acquiring a second image, or a second series of images, in the second wavelength domain or in the visible domain. As a further variant, the vehicle comprises three cameras, with one camera for each of the domains among the first domain, the second domain and the visible domain.

[0067] According to the invention, when several patterns (the same or different patterns) are projected in several different wavelength domains, the at least one camera 102 is capable of acquiring several images of the same scene (simultaneously), one image corresponding to each wavelength domain in which a light pattern is projected.

[0068] When the same light module is capable of projecting light patterns (the same or different light patterns) in several wavelength domains with the same matrix light source, and when the matrix light source is according to the second example or the third example, the alternation frequency between the first beam and the second beam is preferably greater than 100 Hz, for example equal to 700 Hz, so as to correspond to a period much shorter than the exposure time of the at least one camera 102, which allows simultaneous capture of several images in the several wavelength domains.

[0069] The vehicle 100 further comprises an image processing module 104 according to the invention, configured to determine depth information of the scene 10 from the images acquired by the camera 102.

[0070] According to the invention, the projection of at least one light pattern in the first wavelength range in the scene 10 makes it possible to determine the depth information in an image representative of the scene and acquired in the first wavelength range. The projection of the light pattern makes it possible to overcome the need to have a system with two cameras capable of acquiring images in the same wavelength range, according to the principle of stereovision, to measure a disparity of each object or each pixel of the scene 10 and to deduce depth information from the image therefrom.

[0071] Indeed, according to a known method of stereovision, two cameras, whose relative positions are predefined and known, can each acquire an image of the same scene in the same wavelength domain. The offset between pixels corresponding to the same object is called disparity, and allows, geometrically, and from the known spacing between the two cameras, to determine the distance of the object. It is thus made possible to determine a depth map for a pair of images captured by the two cameras, the depth map indicating the distance of each of the pixels of the captured scene. The projection of a light pattern onto the scene makes it possible to overcome the need for a two-camera system, the deformations of the light pattern by the scene making it possible to access the depth information of each of the pixels of an image acquired by a single camera, as will be better understood in light of the description of Figures 2a and 2b below.

[0072] The system further comprises a control module 103 capable of sending commands to the light module 101.1 and to the left light module 101.2: - to control the projection of a beam according to the light pattern, in particular in the first wavelength range, but also, according to embodiments in the visible range and / or in the second wavelength range; - to control the projection of a beam in the visible range performing a lighting or signaling function; and / or - to control the projection of a light beam in the second wavelength range for an object detection function.

[0073] [Fig.2a] shows an example of a light beam 120 according to a light pattern, in the first wavelength range, projected by a light module 101 as previously described.

[0074] In particular, [Fig.2a] corresponds to the usual representation of a projected light beam, which corresponds to a projection onto a screen equipped with an orthonormal reference frame and positioned 25 meters from the projector. Thus, [Fig.2a] does not correspond to the projection of the light pattern onto a real scene, which will be described with reference to [Fig.2b].

[0075] No restriction is attached to the manner in which the light pattern is determined by the control module 103. The light pattern can be: - predetermined, i.e. it is fixed. In the case where several light patterns are projected by one or more light modules in one or more wavelength ranges, each light pattern may be predetermined; - selected from a set of predetermined light patterns, the selection being made based on information about the vehicle's environment and / or describing the driving situation, for example, the light pattern is selected from the set based on a current speed of the vehicle; - determined by the control module 103 based on information on the environment of the vehicle and / or describing the driving situation. For example, the light pattern can be determined based on objects in the scene identified from images acquired by the at least one camera 102.

[0076] According to the invention, the light pattern is a discontinuous pattern, composed for example of dark zones and lit zones, the deformations induced by the scene 10 in the light pattern, in particular at the discontinuities which are the boundaries between dark zones and lit zones, allowing the determination of depth information.

[0077] In the example of [Fig.2a], the light pattern is a checkerboard pattern, presenting a regular alternation of dark zones 201 and illuminated zones 202, with a strong contrast between these zones.

[0078] Each zone 201 or 202 corresponds to a set of at least one pixel, and preferably to a plurality of pixels, for example several tens or hundreds of pixels.

[0079] Thus, a dark area is produced by deactivating the pixels of the area, while an illuminated area is obtained by activating at least a portion of the pixels of the illuminated area. The activation or deactivation of the pixels is implemented by the control unit 110 which is capable of controlling the pixelated light source 111. In the first example of a pixelated light source described above, the control unit 110 produces the light pattern by applying a voltage to the light elements corresponding to the pixels of the illuminated areas 202, and does not apply any voltage to the light elements corresponding to the pixels of the dark areas 201.

[0080] A high-definition pixelated light source 111 thus allows the projection of a light pattern with a large number of dark and illuminated areas, which then allows high precision in determining the depth map.

[0081] The checkerboard shown in [Fig.2a] is given for illustrative purposes. It comprises 9 columns and 8 rows of zones, each zone comprising a plurality of pixels. No restriction is attached to the shape of the zones, nor to the distribution of the lit zones and dark zones. However, a regular checkerboard comprising a large number of zones, for example more than 100 zones, allows in practice an accurate determination of a depth map by the image processing module 104.

[0082] [Fig.2b] shows an image 210 acquired by the camera 102 following the projection of a light beam according to a light pattern in the scene 10 facing the vehicle 100, according to embodiments of the invention.

[0083] The scene shown in [Fig.2b] is deliberately simplified, with few objects, in order to simplify the understanding of the invention.

[0084] The scene in which the light pattern is projected includes another vehicle, as well as a vertical wall located behind the other vehicle. The scene may thus correspond to the interior of a parking lot, a situation given for illustrative purposes only. The invention may advantageously be implemented in an outdoor driving scene, in particular during a night scene.

[0085] As previously explained, the light pattern projected into the scene may be formed in the first wavelength range, and the image 210 captured by the camera 102 is thus an image in the first wavelength range. When the camera 102, or another of the cameras 102, is furthermore capable of capturing an image in the visible range, and a light pattern is projected by a light beam in the visible range by the light module, the image 210 may be a visible image. Similarly, when the camera 102, or another of the cameras 102, is furthermore capable of capturing an image in the second wavelength range, and a light pattern is projected by a light beam in the second wavelength range, the image 210 may be an image in the second wavelength range.

[0086] Thus, the at least one camera 102 can acquire: - in one embodiment a single image 210 of the scene in the first wavelength domain during the projection of a light beam in the first wavelength domain according to a light pattern; - in a variant, a single image 210 of the scene in the first wavelength domain during the projection of a first light beam in the first wavelength domain according to a first light pattern and during the simultaneous projection of a second light beam in the first wavelength domain according to a second light pattern (identical or different from the first pattern) by the left light module 101.2; - in a variant, the image 210 of the previous variant, as well as a second image 210 of the scene in the visible domain or in the second domain, during the projection of a first light beam in the visible domain or in the second domain, according to a third light pattern (identical or different from the first and second patterns), and during the projection of a second light beam in the visible domain or in the second domain, according to a fourth light pattern (identical or different from the first, second and third patterns);- in a variant, an image 210 of the scene in the first wavelength domain, during the projection of a light beam in the first wavelength domain according to a light pattern, and another image 210 of the scene in the visible domain or in the second wavelength domain, during the projection of a light beam in the visible domain or in the second wavelength domain, according to the same light pattern or according to another light pattern.;

[0087] In the above variants in which several images are acquired, the images are acquired simultaneously (i.e. the exposure time periods of the camera 102 for each of its sensors, or of the cameras 102, overlap or are identical).

[0088] Other variants are possible according to the invention.

[0089] Thus, according to the embodiments, the camera 102 can acquire as many images as there are wavelength domains in which light patterns are projected. In the examples described above, one or two wavelength domains are used for the projection of light patterns, but the invention also applies to strictly more than two wavelength domains in which light patterns are projected.

[0090] In the following, for the sake of simplification, it is considered that the image 210 is in the first wavelength domain and follows the projection of the light pattern in the first wavelength domain by the two light modules 101.1 and 101.2.

[0091] It can be observed in [Fig.2b]: - that the dark and illuminated areas 211 projected onto a substantially horizontal plane such as the ground have a stretched shape and are thus elongated; - that the dark and illuminated areas 212 projected onto a substantially vertical and nearby plane, such as the rear of the other vehicle, retain a square format and have a first given size; - that the dark and lit areas 213 projected onto a substantially vertical plane further away than the areas 212, such as the wall behind the other vehicle, also retain a square format but have a second size greater than the first size; - that the dark and lit areas 214 projected onto irregular objects, with curves, such as the top of the other vehicle, are strongly distorted.

[0092] Thus, the projection of a light pattern into a scene provides, by analyzing the deformation of the projected pattern and after processing the image, disparity information of the scene which makes it possible to estimate depth information of the scene. Thus, a light pattern comprising a regular repetition of contrasting sub-patterns, such as a repetition of square dark and lit areas to form a checkerboard, is particularly suitable for estimating depth information of the scene.

[0093] Such depth information can be estimated by the image processing module 104, from the image acquired by the camera 102, or from the several images acquired by the camera 102 or by the cameras 102.

[0094] [Fig.3] shows a structure of the image processing module 104 according to embodiments of the invention.

[0095] The image processing module 104 comprises an input interface 303 for receiving the image, or images, captured by the at least one camera 102 and representative of the projection of the light pattern, or light patterns, projected into the scene 10 facing the vehicle 100.

[0096] The image processing module 104 further comprises a calculation unit 301, such as a processor, configured to determine depth information of the scene 10, for example in the form of a depth map, from the at least one image received on the input interface 303. The processor 301 may for example be a graphics processor, also called GPU for “Graphical Processing Unit” in English.

[0097] The processor 301 is configured to communicate unidirectionally or bidirectionally, via one or more buses or via a direct wired connection, with a memory 302 such as a “Random Access Memory” type memory, RAM, or a “Read Only Memory” type memory, ROM, or any other type of memory (Flash, EEPROM, etc.). Alternatively, the memory 302 comprises several memories of the aforementioned types. The memory can, for example, temporarily store the image or images received on the input interface 303.

[0098] The memory 302 may further permanently store an algorithm executing an image processing function applicable to the received image(s) and providing as output scene depth information such as a depth map.

[0099] The memory 302 can for example store instructions allowing the execution of a model defined by a set of parameters and capable of generating a depth map of a scene from at least one image representative of the projection of a light pattern in the scene.

[0100] No restriction is attached to the model, which can be constructed from a set of rules for determining depth information from: -a priori knowledge of the pattern or patterns when they are predetermined or from a predetermined set, or - from information describing the light pattern at the input of the model, when the light pattern is not among a predefined set of light patterns.

[0101] According to embodiments of the invention, the model is trained during a preliminary phase by machine learning, the machine learning allowing the training of the parameters defining the model, the model having a predefined structure. The model may for example have one of the following structures: - a convolutional neural network, such as a U-Net type network for example; - an artificial neural network of the auto-encoder or variational auto-encoder type, also called VAE in English; - any other model structure capable of receiving as input one or more images representing the projection of a light pattern in the scene, and optionally information describing the light pattern, and of predicting depth information of the scene, for example in the form of a depth map of the scene.

[0102] No restrictions are attached to the machine learning applied to the model to optimize its parameters. Machine learning can for example be of the supervised type, from a training database comprising associations between: - an image of a projection of a light pattern in a scene, in the first wavelength domain, or a set of several images of projections of several light patterns in a scene in the first wavelength domain, or a set of several images of projections of a light pattern in several wavelength domains; - a depth map of this same scene, indicating the “ground truth”, from which a loss function can be applied to evaluate the difference between the model prediction and the ground truth, which allows the optimization of the parameters during supervised learning.

[0103] The associations stored in the training database may be, for example: - obtained by accumulating images and associated depth maps, in various real driving situations, for example for various scenes (driving in the city, on the motorway, in the countryside) and in various real climatic conditions; Or - obtained by simulating images and associated depth maps, in various simulated driving situations, for example for various simulated scenes (in town, on the motorway, in the countryside) and in various simulated climatic conditions.

[0104] When the model is capable of receiving as input information describing the light pattern (in the case of a light pattern determined by the control module 103 as a function of environmental or driving conditions), the scenes also vary as a function of the light patterns projected there.

[0105] No restriction is attached to the loss function used to train the model, which can be any known loss function. The model thus trained is specific to the wavelength domain in which the light pattern is projected (or the wavelength domains), as well as to the relative positionings of the camera 102 and the light module projecting the light pattern or patterns (or light modules projecting the light pattern or patterns).

[0106] When the light pattern is predetermined, or when a set of light patterns is predetermined, the model thus trained may be specific to this light pattern or to this set of light patterns.

[0107] Alternatively, when descriptive information of the light pattern can be received as input to the model, the model is generic and can be applied to a set of light patterns which is not predefined.

[0108] The image processing module 104 may apply pre-processing to each received image before applying the depth information prediction model. The pre-processing may consist of cropping the received image to retain only the part of the received image corresponding to the projection of the light pattern in the scene 10. Other pre-processing may be applied according to the invention.

[0109] The image processing module 104 further comprises an output interface 104 capable of transmitting, to a separate module, the depth information of the scene, such as the depth map. For example, the separate module to which the depth information is transmitted may be an AD AS module capable of implementing at least one driving assistance function based on the depth information. As mentioned previously, the AD AS module may receive information on the distance of objects (therefore on the depth of the scene), in a redundant manner, from at least one other sensor of the vehicle, for example from a lidar, which makes it possible to improve the safety associated with the driving assistance function.

[0110] [Fig.4a] illustrates a pixelated light source 111 according to the first example mentioned above, capable of forming two light beams in two distinct wavelength ranges.

[0111] The pixelated light source 111 according to the first example comprises: - a first set of individually controllable light elements 401, capable of emitting light in the first wavelength range; - a second set of individually controllable light elements 402, capable of emitting light in the visible range or in the second wavelength range.

[0112] In the example of [Fig.4a], the second set 402 of light elements is capable of emitting light in the visible range, and notably comprises subsets of blue, red and green light elements, which makes it possible to project a colored light beam into the scene 10 facing the vehicle, to display a light pattern or to perform a lighting or signaling function, as previously described. Alternatively, each element of the second set 402 may be capable of emitting white light.

[0113] Alternatively, as explained previously, the second assembly 402 may comprise light elements capable of emitting light in the second wavelength range.

[0114] Thus, the light elements of the first set 401 can be controlled to form a first light beam according to the light pattern described above, with a resolution depending on the number of light elements of the first set 401.

[0115] Simultaneously, the light elements of the second set 402 can be controlled to form a second light beam according to the light pattern, according to another light pattern, or performing the lighting or signaling function.

[0116] [Fig.4b] illustrates a pixelated light source 111 according to the second example, comprising two light sources 411.1 and 411.2 and a matrix of DMD micro-mirrors.

[0117] The first light source 411.1 is capable of emitting light rays in the first wavelength range. The second light source 411.2 is capable of emitting light rays in the visible range or in the second wavelength range. According to a variant not shown, the second light source is capable of emitting light rays in the visible range, and the pixelated light source 111 further comprises a third light source capable of emitting light rays in the second wavelength range.

[0118] The light sources 411.1 and 411.2 can be arranged on the same support 412, as shown in [Fig.4b], or on two separate supports.

[0119] A single micro-mirror 410 is shown in [Fig.4b] to facilitate understanding of the figure. However, in practice, the matrix comprises a large number of micro-mirrors, in particular more than 100, or even more than 1000 or more than 10,000 micro-mirrors individually controllable by the control unit 110 described above.

[0120] Each micro-mirror 410 can be controlled so as to switch between at least a first position 420.1 and a second position 420.2. In the first position 420.1, the micro-mirror is arranged to reflect a light ray from a source, for example a light ray 413 emitted by the first light source 411.1, towards the projection system 112 so as to contribute to the formation of a light beam outside the vehicle 100. Thus, by controlling the positions of the micro-mirrors of the matrix by the control unit 110, the light module projects a first light beam according to the light pattern, in the first wavelength range, when the first light source 411.1 is activated.In the same way, by controlling the positions of the micro-mirrors of the matrix by the control unit 110, the light module projects a second light beam according to the same pattern, according to another pattern, or to carry out a given function, for example lighting or signaling (for the visible domain) or to allow the detection of objects in the scene (for the second domain).

[0121] The control unit 110 can thus control the same matrix of micro-mirrors, alternately at a given frequency to alternately project the first beam and the second beam. The switching frequency between the two beams can be greater than 700 Hz, so that the flashing of the second beam is not perceptible by the human eye and therefore does not disturb the driver and other road users, and is not perceptible by the camera 102 or cameras 102, having an exposure time greater than the period associated with the switching frequency.

[0122] [Fig.5] is a diagram illustrating the steps of a method for determining depth information of a scene facing a vehicle, according to embodiments of the invention.

[0123] In a step 501, at least one of the light modules 101.1 and 101.2 projects a light beam according to a light pattern in the first wavelength range in the scene 10.

[0124] According to embodiments, step 501 can be repeated to: - project the same pattern or a different light pattern in the visible range or in the second wavelength range by the same light module, or by the two light modules 101.1 and 101.2; and / or - project the same pattern or another light pattern, by the other light module 101.1 or 101.2.

[0125] Thus, step 501 can be implemented several times, to project one or more light patterns into the scene 10, in one or more wavelength domains, according to the embodiments of the invention.

[0126] Examples of embodiments are listed below for illustrative purposes: - only one of the light modules 101.1 or 101.2 projects a single light pattern in the first wavelength range at a single step 501; - the light module 101.1 and the light module 101.2 both project a single light pattern in the first wavelength range in two simultaneous steps 501; - the right light module 101.1 projects a light pattern in the first wavelength range at a step 501 and the left light module 101.2 projects another light pattern in the first wavelength range at another step 501 implemented simultaneously; - the light modules 101.1 and 101.2 both project a light pattern in the first wavelength range at a step 501, and the same light pattern or a different light pattern in the visible range or in the second wavelength range at another step 501 carried out simultaneously; - the right light module 101.1 projects a first light pattern in the first wavelength range at a first step 501, the right light module 101.1 projects a second light pattern in the visible range or in the second wavelength range at a second step 501 (at the same time as the first step 501), the left light module 101.2 projects a third light pattern in the first wavelength range at a third step 501 (at the same time as the first and second steps 501), and the left light module projects a fourth light pattern in the visible range or in the second wavelength range at a fourth step 501 (at the same time as the first, second and third light patterns).

[0127] Other combinations are possible according to the invention.

[0128] At a step 502, the camera 102 captures an image representative of the scene 10 in which the light pattern in the first wavelength domain is projected during step 501.

[0129] When step 501 is implemented several times in several wavelength domains, several images are acquired simultaneously (during exposure time periods which overlap or are identical) by the camera 102 or by several cameras 102, during step 502.

[0130] The at least one camera 102 thus captures one or more images of the scene 10 in which a light pattern is projected (or several light patterns are projected).

[0131] From the image, or images acquired by the at least one camera 102, the image processing module 104 determines depth information of the scene 10, for example in the form of a depth map, at a step 403.

[0132] At a step 404, the image processing module 104 can transmit the depth information to another module, for example to an AD AS module, as previously explained.

[0133] The present invention is not limited to the embodiments described above as examples; it extends to other variants.

Claims

Claims

1. System for determining depth information of a scene (10) facing a motor vehicle (100), the system comprising: - at least one light module (101.1; 101.2) arranged to project towards the scene a light beam (120; 120.1; 120.2) in a first wavelength range not comprising any visible wavelength, the light module comprising a pixelated light source (111) and being capable of controlling said pixelated light source to project the light beam according to a light pattern (201; 202); - at least one camera (102) capable of acquiring at least one image (210) in the first wavelength range and representative of a projection of the light pattern in the scene facing the vehicle; - an image processing module (104) configured to determine depth information of the scene from said at least one image.

2. System according to claim 1, comprising a right light module (101.1) and a left light module (101.2) capable of projecting a right light beam (120.1) and a left light beam (120.2) in the first wavelength range according to the light pattern (201; 202).

3. System according to claim 1, comprising a right light module (101.1) capable of projecting a right light beam (120.1) according to a first light pattern in the first wavelength range and a left light module (101.2) capable of projecting a left light beam (120.2) according to a second light pattern in the first wavelength range, in which the at least one camera (102) is capable of obtaining the at least one image in the first wavelength range and representative of the projection of the first light pattern and of the projection of the second light pattern in the scene facing the vehicle.

4. System according to one of the preceding claims, in which the at least one light module (101.1; 101.2) is further capable of projecting another light beam in the visible range or in a second wavelength range separate from the first wavelength range and not including any visible wavelengths.

5. System according to claim 4, in which the at least one light module (101.1; 101.2) is capable of projecting the other light beam in the visible range for performing a lighting, signaling or driving assistance function.

6. System according to claim 4, wherein the at least one light module (101.1; 101.2) is capable of projecting the other light beam in the visible range or in the second wavelength range, according to the light pattern (201; 202) or according to another light pattern, wherein the at least one camera (102) is further capable of acquiring at least one other image in the visible range or in the second wavelength range, and representative of the projection of the light pattern or the other light pattern in the scene; wherein the image processing module (104) is configured to determine the depth information from the at least one image (210) in the first visible range and from the at least one other image.

7. System according to one of claims 4 to 6, wherein the pixelated light source (111) of the at least one light module (101.1; 101.2) is a matrix of light elements individually controllable by a control unit (110) of the at least one light module, the matrix comprising a first set (401) of light elements capable of emitting light rays in the first wavelength range and a second set of light elements (402) capable of emitting light rays in the visible range or in the second wavelength range.

8. System according to one of claims 4 to 6, wherein the pixelated light source (111) of the at least one light module (101.1; 101.2) comprises a matrix of micro-mirrors (410), each micro-mirror being individually controllable by a control unit (110) of the at least one light module, the pixelated light source further comprising a first light source (411.1) capable of emitting light rays in the first wavelength range towards the matrix of micro-mirrors and a second light source (411.2) capable of emitting light rays in the visible range or in the second wavelength range towards the micro-mirror array.

9. System according to one of claims 4 to 6, in which the pixelated light source (111) is laser scanning and comprises a mirror arranged to scan a set of predetermined positions of the scene, a first laser light source capable of emitting light rays in the first wavelength range towards the mirror, and a second laser light source capable of emitting light rays in the visible range or in the second wavelength range towards the mirror.

10. System according to one of the preceding claims, in which the light pattern (201; 202) comprises dark areas (201) and illuminated areas (202) in the first wavelength range.

11. System according to one of the preceding claims, in which the dark areas (201) and the illuminated areas (202) have identical shapes and sizes and form a checkerboard.

12. System according to one of the preceding claims, in which the image processing module (104) implements an artificial neural network capable of receiving said at least one image (210) as input, and of predicting the depth information of the scene (10) as output.

13. The system of claim 12, wherein the artificial neural network is a U-Net.

14. Method for determining depth information of a scene (10) facing a vehicle (100) comprising the following steps: - projection (501) towards the scene, by at least one light module (101.1; 101.2) of the vehicle, of a light beam (120; 120.1; 120.2) in a first wavelength domain, according to a light pattern (201; 202), the first wavelength domain not comprising any visible wavelength; - acquisition (502), by at least one camera (102), of at least one image (210) in the first wavelength domain and representative of a projection of the light pattern in the scene facing the vehicle; - determination (503), by an image processing module (104), of depth information of the scene from said at least one image.

Images