Improving semantic perception in an image by controlling the lighting of a vehicle

Abstract

The invention relates to a method comprising, upon obtaining (500: 501) a first image representative of a scene facing the vehicle, determining (502) a light intensity map from a light intensity map generation model and the first image obtained. The light intensity map indicates light intensity values ​​for controlling the light elements of a matrix source in the vehicle's lighting module. The determined light intensity map is transmitted (503) to the lighting module for projection of a pixelated lighting beam onto the scene facing the vehicle. A second image representative of the scene facing the vehicle is obtained following the projection of the pixelated lighting beam, and image processing is applied to the second image obtained. This image processing is a semantic image perception process. FIG. 5

Description

Title of the invention: Improvement of semantic perception in an image by controlling the lighting of a vehicle

[0001] The present invention relates to the field of controlling a lighting module in a motor vehicle. More specifically, the invention concerns a method and a control module for a motor vehicle lighting module, to improve a semantic perception function, for example, object detection or semantic segmentation of an image captured in the motor vehicle.

[0002] Most motor vehicles are now equipped with a driver assistance module, also called AD AS, for “Advanced Driver-Assistance Systems”, capable of implementing at least one driver assistance function, allowing the driver to be assisted in driving or to control, in an automated manner, without contribution from the driver, certain driving parameters of the vehicle.

[0003] Such functions are based on data captured by sensors on the vehicle, such as a lidar, a radar, one or more cameras, etc.

[0004] Several driver assistance functions implemented by ADAS modules use images captured by a camera capable of obtaining representative images of a scene facing the vehicle. However, some ADAS functions require that these images be processed before being used. Such processing may include, in a known manner: - Semantic segmentation aims to segment each captured image into several pixel regions, each region being labeled with a class from a set of predefined classes. In automotive applications, the following classes might be used: car, pedestrian, sign, road, etc. A segmentation score can be associated with the image segmentation, or with each segment determined within the image, the score representing the degree of certainty associated with the segmentation; and / or - Object detection aims to detect one or more objects in the scene represented by each image, to identify the category of each detected object from among several predefined categories, and to determine the position of each detected object (or a part of the image in which the object is located). Each detected object is further associated with a detection score representing the certainty associated with the detection.

[0005] However, such processing is less effective when ambient light is low, or even very low, as is the case when the camera captures images representative of a night scene. In such situations, the segmentation score or the detection score is much lower than in driving situations. during the day, this can prevent the implementation of certain driver assistance functions, or even cause safety problems.

[0006] There is therefore a need to improve the accuracy associated with semantic segmentation or object detection in a scene facing a motor vehicle, when the ambient light is low.

[0007] The present invention improves the situation.

[0008] A first aspect of the invention relates to a method for controlling a vehicle lighting module, the method comprising, during a typical phase, the following steps: - obtaining a first representative image of a scene facing the vehicle; - determining a light intensity map, from a light intensity map generation model and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first representative image of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control light elements of a matrix source of the vehicle's lighting module; - transmission of the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle; - obtaining a second image representative of the scene facing the vehicle, following the projection of the pixelated lighting beam; - application of image processing to the second image obtained, the image processing being a semantic perception processing. Thus, the invention makes it possible to project a beam of light to obtain a second image representative of a scene. This second image has a greater capacity than the first image, obtained before the lighting was controlled, to be processed by semantic perception, for example, by semantic segmentation or object detection, particularly in night scenes. As a result, semantic perception processing is improved. Indeed, the light intensity map generation model can be designed, for example, trained by machine learning, particularly supervised learning, specifically to improve the performance of the image processing implemented in the vehicle. That is, to ensure that a certain level of performance of the semantic perception processing (for example, an object detection score or a semantic segmentation score) is higher for the second image than for the first.Moreover, a high degree of responsiveness is allowed for lighting control (represented by the time between obtaining the first image and the second image), since the model of . The generation of the light intensity map is capable of directly producing an output command that can be used to control the matrix source of the lighting module.

[0009] According to embodiments, the second image processed by the image processing can be transmitted to a vehicle driving assistance module, capable of implementing at least one driving assistance function based on said second processed image.

[0010] Thus, improving image processing performance enables an improvement in the driver assistance function based on such image processing. This enhances vehicle safety and / or driving comfort.

[0011] According to embodiments, the light intensity map generation model can have one of the following structures: - a convolutional neural network; - an artificial neural network of the auto-encoder or variational auto-encoder type; - a self-aware or transformative model; or - a network generating a system of antagonistic generative networks.

[0012] Such model structures are particularly well-suited for receiving an image as input and generating an output command in the form of a light intensity map. Furthermore, such structures can be trained using machine learning.

[0013] According to embodiments, the method may further include a training phase of the light intensity map generation model, comprising a modification of at least one parameter of the light intensity map generation model as a function of a loss evaluated from the output of a training image processing algorithm applied by a training image processing module, the training image processing being a semantic perception processing.

[0014] Thus, the light intensity map generation model is specifically trained to produce a light intensity map for controlling lighting that improves the performance of an image processing algorithm, which is a semantic perception processor. The training image processing is at least of the same type of semantic perception processor as the image processing of the current phase: for example, if the image processing of the current phase is object detection, the training image processing is also object detection.

[0015] In addition, the image processing algorithm for training the training phase can be identical to the image processing applied during the current phase.

[0016] Thus, the image generation model is specifically trained to improve the performance of the image processing used during the current phase of the process, which makes it possible to optimize the performance associated with semantic perception.

[0017] In addition or as an alternative, the training phase may include the following steps: - obtaining an association of training data, the association comprising a representative training image of a scene and reference data; - application of the light intensity map generation model to the training image, to obtain a training light intensity map; - obtaining a synthetic image representative of the scene of the training image into which is projected a beam of lighting obtained according to the training light intensity map; - application of the training image processing algorithm to the synthetic image, to obtain a processed training image; - evaluation of a loss by comparing the processed training image with the reference data; - modification of at least one parameter of the light intensity map generation model based on the evaluated loss.

[0018] Such steps in the training phase allow for supervised learning of the light intensity map generation model when these steps are repeated until a predefined convergence criterion is reached. The light intensity map generation model thus obtained is optimal for improving image processing performance, the loss function being dependent on the performance of the training image processing algorithm.

[0019] In addition, the training phase may further include a modification of at least one parameter of the training image processing algorithm, and the image processing applied during the current phase corresponds to the training image processing algorithm at the end of the training phase.

[0020] Thus, the training phase allows for the joint optimization of the training image processing algorithm and the light intensity map generation model, which improves the ability of the light intensity map generation model to optimize lighting for better image processing performance during the current phase.

[0021] According to embodiments, the matrix source of the lighting module may comprise electroluminescent semiconductor elements of submillimeter dimensions, epitaxially mounted directly on a common substrate.

[0022] Such a lighting module allows the projection of a high-resolution pixelated lighting beam, which improves the map generation model's capability light intensity to optimize lighting to improve image processing performance.

[0023] A second aspect of the invention relates to an assembly comprising: - a camera arranged to obtain a first image representative of a scene facing the vehicle; - at least one lighting module comprising a matrix source comprising a plurality of individually controllable light elements; - a control device configured to determine a light intensity map, from a light intensity map generation model and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control the light elements of the matrix source of the vehicle lighting module. The control device is capable of transmitting the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle and, upon obtaining a second image representative of the scene facing the vehicle by the camera, following the projection of the pixelated lighting beam, an image processing module is configured to apply image processing to the second image obtained, the image processing being a semantic perception processing.

[0024] A third aspect of the invention relates to a vehicle comprising an assembly according to the second aspect of the invention and further comprising a driving assistance module, in which the driving assistance module is configured to implement at least one driving assistance function based on the second image processed by the processing module.

[0025] Other features and advantages of the invention will become apparent from an examination of the detailed description below, and the accompanying drawings in which:

[0026] [Fig-1] illustrates a vehicle according to embodiments of the invention;

[0027] [Fig.2] illustrates a vehicle lighting module according to embodiments of the invention;

[0028] [Fig.3] illustrates an image processed by an object detection module of a vehicle according to embodiments of the invention;

[0029] [Fig.4] illustrates a drive system for a light intensity map generation module to control a lighting module, according to embodiments of the invention;

[0030] [Fig.5] illustrates the steps of a common phase of a method for controlling a vehicle lighting module, according to embodiments of the invention;

[0031] [Fig.6] illustrates the steps of a training phase of a generation model of a light intensity map, of a method for controlling a vehicle lighting module, according to embodiments of the invention;

[0032] [Fig.7] illustrates a structure of a control device for a lighting module vehicle, according to embodiments of the invention.

[0033] The description focuses on the characteristics that distinguish the module and the method from those known in the state of the art.

[0034] Fig. 1 illustrates a vehicle 100 according to embodiments of the invention.

[0035] The vehicle 100 according to the invention comprises at least one lighting module 120 including a matrix light source, capable of forming a pixelated beam of light projected outwards from the vehicle so as to illuminate a scene facing the vehicle, particularly at night or when ambient light is low.

[0036] The lighting module 120 is integrated into a lighting device 110 such as a front headlight of the vehicle.

[0037] In some embodiments, the lighting device 110 may include at least one other lighting module, in addition to the lighting module 120, to perform a different or complementary lighting function to the lighting function implemented by the lighting module 120. However, the lighting module 120 may alternatively be the only module of the lighting device 110 performing a lighting function. In some embodiments, the lighting device 110 may further include a signaling module capable of implementing at least one vehicle signaling function.

[0038] In the example shown, the lighting device 110 in which the light module 120 is integrated can be the right front headlight or the left front headlight. Alternatively, a first light module 120 is arranged in the right front headlight and a second light module 120 is arranged in the left front headlight of the vehicle 100.

[0039] The vehicle 100 according to the invention further comprises a control device 130 capable of sending lighting commands to the light module 120, according to the images acquired by a camera 140, to which a model 131 of generation of a lighting command is applied.

[0040] According to the invention, the lighting control is a light intensity map indicating a light intensity for each light element of the matrix source of the lighting module 120, which is described below with reference to [Fig.2],

[0041] For this purpose, the control device 130 is capable of executing a light intensity map generation model 131, described in detail below.

[0042] The control of the lighting module 120 is thus said to be adaptive, in that it is determined from the external scene facing the vehicle, represented by the images acquired by the camera 140.

[0043] Thus, the vehicle 100 according to the invention further comprises the camera 140, or a plurality of cameras 140, capable of acquiring an image, or a series of images, of the scene into which the lighting module 120 is capable of projecting the pixelated light beam. Thus, when the lighting module 120 is mounted in the front headlight of the vehicle 100, the lighting module 120 projects the pixelated light beam into a scene facing the vehicle: in this case, the camera 140 is arranged in the vehicle 100 so as to acquire an image, or a series of images, representative of the scene facing the vehicle. For example, as illustrated in [Fig. 1], the camera 140 can be arranged in a central position at the top of the windshield of the vehicle 100.

[0044] More generally, the camera 140 is capable of acquiring an image of a scene in which the lighting module 120 is capable of projecting the pixelated light beam.

[0045] The vehicle 100 includes an image processing module 150, capable of applying at least one image processing operation to the images captured by the camera 140, before transmitting the processed images to a driver assistance module 160, referred to as the ADAS module 160 in the following. The ADAS module 160 is capable of implementing at least one vehicle driver assistance function.

[0046] According to the invention, the ADAS module 160 uses the images processed by the image processing module 150 to implement at least one driver assistance function. No restrictions are attached to this at least one driver assistance function, which may be a pedestrian avoidance function, an emergency braking assistance function, an adaptive cruise control function (ACC), etc. According to the invention, the image processing module 150 is a semantic perception processor. Thus, the image processing module 150 may be: - an object detection module 150, capable of detecting one or more objects in an image or a series of images, of associating a category with each detected object from a predefined set of categories, of determining the position of each detected object in the image, and of determining a detection score for each detected object; or - a semantic segmentation module 150, capable of segmenting each captured image into several pixel regions, each region being labeled with a class from a set of predefined classes (car, pedestrian, sign, road, etc.). The semantic segmentation module can determine a segmentation score associated with the image segmentation, or with each segment determined in the image; or - a module capable of performing another semantic perception task, for example panoptic segmentation.

[0047] The object detection module 150 can implement an object detection algorithm, such as an available object detection algorithm not covered by the present invention. For example, a well-known YOLO (You Only Look Once) type algorithm can be implemented by the object detection module 150. Alternatively, the object detection algorithm implemented by the object detection module 150 is an object detection model derived from a training phase of the process according to the invention, described below with reference to [Fig. 6].

[0048] The semantic segmentation module 150 can implement a semantic segmentation algorithm, such as a known semantic segmentation algorithm not covered by the present invention. Alternatively, the semantic segmentation algorithm implemented by the semantic segmentation module 150 is a segmentation model derived from a training phase of the process according to the invention, described below with reference to [Fig. 6].

[0049] The control device 130 can be a centralized vehicle control device, called “Body Controller” in English, capable of implementing a plurality of vehicle functions, including the function of controlling the lighting module 120. Alternatively, the control device 130 is dedicated to generating light intensity maps to control the lighting module 120.

[0050] Fig. 2 illustrates the structure of a lighting module 120 with a matrix light source 200, according to embodiments of the invention.

[0051] The lighting module 120 comprises: - a control unit 201, also called a pilot unit or “driver” in English, of the matrix light source 200; - the matrix light source 200; - a projection optic 202 for the light from the matrix light source 200 to project a pixelated beam of light outwards from the vehicle, into a scene facing the vehicle 100, towards which the previously described camera 140 is directed. No restrictions are attached to the projection optic, which may comprise any set of optical elements.

[0052] The matrix light source 200 comprises a plurality of light elements 210 that can be individually controlled by the control unit 201. The control unit 201 can thus individually control the light elements 210, and can thus control a pattern projected in the pixelated light beam by activating some light elements 210 and deactivating other light elements 210, and by controlling the light intensity of each light element 210 according to the light intensity map received.

[0053] Note that the light intensity map can directly include the light intensity values ​​for all the light elements 210 of the matrix source 200, but more generally includes information enabling the control unit 201 of the lighting module 120 to determine the light intensities of the light elements 210 of the matrix source 200.

[0054] There is no restriction on the number of light elements 210 in the matrix light source 200. Preferably, the matrix light source 200 is a high-definition light source, that is, one capable of projecting a beam of light comprising more than one hundred pixels, preferably more than 1000 pixels. The matrix light source 200 can project a beam of light comprising more than 10,000 pixels according to embodiments of the invention.

[0055] The luminous elements 210 can be electroluminescent.

[0056] For example, each light element 210 can be a light-emitting diode of the LED type, for “Light Emitting Diode” in English, the plurality of LEDs forming a matrix network of LEDs.

[0057] According to some embodiments, the matrix light source 200 can be monolithic. A so-called "monolithic" source can have a particularly high density of light elements 210, making it especially attractive for a wide range of applications. A monolithic source involves a plurality of submillimeter-sized electroluminescent semiconductor elements 210, epitaxially bonded directly onto a common substrate, the substrate generally being made of silicon.Unlike conventional LED arrays, where each individual light-emitting element is a uniquely produced electronic component mounted on a substrate such as a printed circuit board (PCB), a monolithic LED source is considered a single electronic component. During its production, multiple areas of light-emitting semiconductor junctions are generated on a common substrate, forming an array. This manufacturing technique allows for the creation of closely spaced areas of light-emitting elements, each acting as a single light-emitting element. The gaps between these elements can be submillimeter in size. One advantage of this manufacturing technique is the high pixel density that can be achieved on a single substrate.

[0058] The individual light elements can be individually controlled by the control unit 201, which is capable of receiving signals from the control device 201. command, in the form of a light intensity map. Thus, the light intensity map is interpreted by the control unit 201 to control the light intensity of each light element 210. For example, the light intensity map includes a light intensity value for each light element 210 (when the light intensity map and the matrix source have the same resolution) or to control the light intensity of a set of light elements 210 (when the matrix source 200 has a higher resolution than the light intensity map).

[0059] Individual control of each light element 210 may include control of the power supply provided to the individual light element 210 by pulse-width modulation, or PWM. The control unit 201 may, for example, be an ASIC (Application-Specific Integrated Circuit), which has the advantage of being very compact and can be integrated into the monolithic light source.

[0060] Alternatively, the matrix light source 210 comprises a light source and a micromirror matrix, each micromirror thus forming a light element 210 by reflecting the light rays from the light source. The micromirror matrix 210 is also called DMD for Digital Micromirror Devices, and the micromirrors 210 can be individually activated to reflect the light from the light source towards the projection optics 202, thus forming a pixelated beam of light, the pattern of which is controllable by controlling the micromirrors 210.

[0061] Figure 3 illustrates an example of an image 300 after processing by the object detection module 150, according to embodiments of the invention. The image 300 was previously captured by the camera 140, as explained above.

[0062] Image 300 represents a scene facing the vehicle, which may in particular be a scene in a night driving situation.

[0063] In the example shown in [Fig.3], the image can detect a first object 301 in a first part 301 of the image 300, a second object 302 in a second part 302 of the image 300 and a third object 303 in a third part 313 of the image 300.

[0064] A road 304 on which vehicle 100 is traveling can also be detected as an object in image 300.

[0065] The outlines of the attributes are sharp in [Fig. 3], for illustrative purposes. It will be understood, however, that the outline of objects and the parts of the image in which they are contained may be less clearly defined, particularly in night driving situations. Furthermore, in practice, the 300 image comprises grayscale pixels. or in color, and not sharp outlines formed by black lines as shown, for simplification purposes, in [Fig.3].

[0066] As described previously, the object detection module 150 can further assign a category to each identified object, for example, the category “car” to the first and second objects 301 and 302 and the category “pedestrian” to the third object 303. Parts 311 to 313 allow the position of the objects in the image 300 to be determined.

[0067] Inaccuracies may occur in the detection of an object, in the determination of its category and / or its position. As described previously, the degree of certainty with which an object, its category and its position are determined is expressed by a detection score determined by the object detection module 150. Note that an object may be associated with a high detection score while another object is associated with a lower confidence score, in the same image 300, as the brightness is not homogeneous in the scene.

[0068] As described below, the invention enables the lighting module to control the illumination so as to improve the performance of the processing module 150, and thus of semantic perception processing, in particular object detection or object segmentation. Such performance improvement may, in particular, be an improvement in the detection scores associated with object detection in the image 300, or an improvement in the segmentation score or scores, and / or a reduction in errors (relative to ground truth) during the detection, categorization, and / or localization of objects in the scene. In particular, the lighting control according to the invention may enable the detection of objects in a night scene that would not have been detected without such lighting control.

[0069] Figure 3 illustrates the result of image processing applied by the object detection module 150. For illustrative purposes, the processing module 150 is considered to be an object detection module: thus, the lighting control according to the invention improves the performance associated with object detection. However, as explained previously, the processing module 150 can more generally be a semantic perception module, for example, a semantic segmentation module, in which case the lighting control according to the invention improves the performance associated with semantic perception in general, for example, the performance associated with semantic segmentation.

[0070] Fig. 4 illustrates a drive system for the light intensity map generation model 131 implemented in the control device 130 of the vehicle 100 according to embodiments of the invention.

[0071] Such a drive system 400 is external to the vehicle 100 and is capable of implementing the drive phase of the process according to the invention, which is described later with reference to [Fig. 6]. The drive phase precedes a current phase during which the control device 130 controls the lighting of the lighting module 120 during a driving situation of the vehicle 100. The drive phase may, in particular, be part of the design and manufacturing process of the control device 130 (or even of the processing module 150 in the second embodiment described below), before its integration into the vehicle 100.

[0072] The training system 400 includes a training database 401, capable of storing training data associations, each training data association comprising: - a training image representative of a night driving scene; - ground truth comprising reference data corresponding to object detection (or semantic segmentation when the lighting control according to the invention aims to improve the performance of semantic segmentation) in the scene represented by the training image. The ground truth includes reference data describing all objects actually present in the scene, their respective actual categories, and their precise locations in the training image. In the case of semantic segmentation, the reference data includes the actual segments of the image and the category of each segment.

[0073] Preferably, the associations stored in the training database are varied, that is to say, they were obtained in driving situations varying according to several criteria: number of objects included in the scene, number of categories of objects in the scene, distribution of objects in the scene, level of brightness of the scene, weather conditions, etc.

[0074] In addition, the 401 training database includes more than one hundred training data associations, and preferably several thousand or even several tens of thousands of training data associations.

[0075] No restrictions are attached to the manner in which the 401 training database was constructed. The 401 training database can be obtained from images captured by a fleet of vehicles in real-world night driving situations (night driving situations that are preferably varied according to several criteria as described above), with a human operator or additional sensors indicating the ground truth for each captured image. Alternatively, the training data associations can be synthetic.

[0076] The drive system 400 further includes the light intensity map generation model 131 to be driven. The light intensity map generation model luminous 131 is structurally capable of generating a light intensity map of a given resolution, determined from the matrix source 200 with which the control device 130 is associated in the vehicle 100 during the current phase, from at least one image representative of a scene, an image which can be a training image during the training phase, and which is an image captured in real time by the camera 140 during the current phase.

[0077] According to some embodiments, the light intensity map generation model 131 is capable of receiving as input, in addition to the representative scene image, at least one other piece of information. In this case, each image stored in the training database associates a training image with reference data and at least one other piece of training information. The other information may be: - a depth map representing the same scene as the input image. In this case, each association in the training database combines a training image, reference data, and a training depth map (representing the same scene as the training image); and / or - an initial light intensity map (which has been previously determined), corresponding to the lighting of the scene in the input image. In this case, each association in the training database combines a training image, reference data, and an initial training light intensity map (corresponding to the lighting in the training image). In this case, the model aims to improve the light intensity map compared to the initial light intensity map, so that it enables better semantic perception performance.

[0078] In what follows, it is assumed for the sake of simplification that the light intensity map generation model takes as input only an image of a scene, which can be a color image, for example an RGB image for “Red Green Blue” in English.

[0079] For this purpose, the light intensity map generation model 131 may have one of the following structures: - a convolutional neural network, such as a U-Net type network for example; - an artificial neural network of the auto-encoder or variational auto-encoder type, also called VAE in English; - a generating network of a system of generative adversarial networks, also called GAN, for “Generative Adversarial Networks” in English; - any other generative model structure capable of receiving as input an image representing a scene, and of producing as output a light intensity map in a format capable of controlling the lighting module 120 previously described.

[0080] The drive system 400 further includes a synthesis module 402, capable of producing a synthetic image representative of a scene lit by a lighting module controlled by the light intensity map produced at the output of the light intensity map generation model 131. Thus, the synthesis module 402 makes it possible to simulate the capture by a camera of the scene of the training image, into which would be projected a beam of lighting corresponding to the light intensity map at the output of the light intensity map generation model 131.

[0081] The training system 400 further includes a training image processing module 403, which is a semantic perception processing module, for example, a training object detection module or a training semantic segmentation processing module. In accordance with the example described, the training image processing module 403 is considered to be a training object detection processing module in the following.

[0082] The drive image processing module 403 is preferably identical to the image processing module 150 integrated into the vehicle 100 during the current phase of the process according to the invention, and is at least of the same type of semantic perception processing (for example, object detection or semantic segmentation). The light intensity map generation module 131 is thus specifically driven to improve the performance of the image processing module 150 with which it is mounted in the vehicle 100 during the current phase.

[0083] According to a first embodiment, the training image processing module 403 is an already optimized module, that is to say that the parameters of its image processing algorithm do not vary during the training phase.

[0084] According to a second embodiment, the training image processing module 403 is trained at the same time as the light intensity map generation module 131, and the training image processing module 403 from the training phase can be implemented in the vehicle 100 for the current phase as a processing module 150.

[0085] The training image processing module 403, considered as a training object detection module 403 in this illustrative example, receives the synthetic image from the synthesis module 402, and detects one or more objects in the synthetic image, as well as a category, a location and a detection score for each detected object.

[0086] The training system 400 further includes a loss assessment module 403, which is suitable for: - to assess a loss by comparing the ground truth reference data from training database 401 and the outputs of the module detection of training object (objects detected, categories, locations, and detection scores), and / or based on the output detection score(s); - to modify one or more parameters of the light intensity map generation model 131 according to the evaluated loss, and according to a predefined training strategy, and optionally (according to the second embodiment) to modify one or more parameters of the image processing algorithm of the training image processing model 403.

[0087] No restrictions are attached to the loss function used, nor to its application to the comparison between the ground truth reference data and the outputs of the training image processing module 403.

[0088] Such a training system 400 is thus capable of implementing supervised learning of the light intensity map generation model 131, and optionally of the training image processing algorithm of the training image processing module 403 (in the second embodiment).

[0089] Alternatively, the light intensity map generation model 131 can be derived from machine learning other than supervised learning. For example, the light intensity map generation model 131 can be optimized by reinforcement learning, with a reward determined based on the performance of a training processing module implementing a semantic perception training algorithm. Thus, in general, the light intensity map generation model 131 can be trained by machine learning to improve the performance of semantic perception processing applied to an image captured after the lighting module has been controlled based on the light intensity map derived from the light intensity map generation model 131.

[0090] Fig. 5 illustrates the steps of a common phase of a method for controlling the lighting module 120 of the vehicle 100, according to embodiments of the invention.

[0091] The current phase of the process according to the invention can be implemented by the components of the vehicle 100 described above with reference to [Fig.1].

[0092] The method for controlling the lighting module 120 is implemented when the lighting module 120 is activated, therefore in driving situations where the brightness is low, in particular below a predefined brightness threshold, for example at night.

[0093] At a step 500, the camera 140 captures a first representative image of a scene facing the vehicle 100. Alternatively, the camera 140 captures several first images and obtains a first stabilized image.

[0094] Optionally, the first image thus captured is transmitted to the image processing module 150 at a step 505, which applies the processing algorithm of the image it implements, and thus obtains a first processed image at the end of step 505. The first processed image is: - a first image in which objects have been detected, categorized and located, along with a detection score for each of them; or - a first segmented image, including the location and classification of each segment, and one or more segmentation scores.

[0095] The first image thus processed is transmitted by the image processing module 150 to the AD AS module 160 at a step 506, for implementation of at least one AD AS function from the processed image.

[0096] It should be noted that during a first iteration, the lighting of the lighting module is not yet controlled, and thus, the image processing module 150 processes the image with a first level of performance, which can be represented by the object detection scores, the segmentation score(s), and the errors or absence of errors in detection or segmentation. The first level of performance is not determined during the current phase but is mentioned in this description to explain the technical effect associated with implementing the lighting control according to the invention, as described below.

[0097] At a step 501 following step 500, the first image captured by the camera 140 is received by the control device 130.

[0098] At a step 502, the device 130 applies the light intensity map generation model to the first image received at step 501, and obtains a light intensity map as output.

[0099] At a step 503, the light intensity map obtained in step 502 is transmitted to the lighting module 120 for execution of the light intensity map.

[0100] Upon receiving the light intensity map, the previously described control unit 201 drives the matrix source 200 according to the light intensity map, to project a beam of light in front of the vehicle, according to the light intensities of the pixels of the light intensity map.

[0101] Following the projection of the lighting beam according to the light intensity map determined by the control device 120, step 500 is repeated to capture a second image representative of the scene facing the vehicle 100.

[0102] The second image representing the scene facing the vehicle 100 is transmitted from the camera 140 to the processing module 150, and the processing module 150 applies the object detection or semantic segmentation algorithm to the second image, at a second iteration of step 505.

[0103] The second processed image is: - a second image in which objects have been detected, categorized, and located, along with a detection score for each object; and / or - a second segmented image, including the location and classification of each segment, and one or more segmentation scores.

[0104] The second image thus processed is transmitted by the processing module 150 to the AD AS module at a step 506 for implementation of at least one AD AS function depending on the second processed image.

[0105] The second iteration following the capture of the second image follows step 504, which involves projecting a light beam determined by the control device according to the invention. During this second iteration, the image processing module 150 processes the image with a second performance level that is superior to the first performance level, since the light intensity map generation model 131 has been previously trained to improve the processing performance applied by the processing module 150. Like the first performance level, the second performance level is not determined during the current phase but is mentioned in this description to explain the improvement in image processing performance through the control of the lighting module.

[0106] Note that steps 505 and 506 may not be implemented during the first iteration, in which case only the second image processed with the second performance level is transmitted to the AD AS module during step 506. Thus, only an image processed with a high performance level is transmitted to the AD AS module.

[0107] According to some embodiments, steps 501 to 504 are also applied to the second image captured during the second iteration of step 500. Thus, the pixelated lighting beam is controlled again, allowing the evolution of the scene facing the vehicle to be tracked. In these embodiments, a third image is captured at a third iteration of step 500, and steps 505 and 506 are applied to it on the one hand (to update the semantic perception) and steps 501 to 504 are applied to it on the other hand (to continue adapting the lighting to the evolution of the scene facing the vehicle). The steps of the current phase are thus repeated continuously, for successive iterations, during the vehicle's driving situation.

[0108] According to alternative embodiments, the lighting beam is adapted only once based on the first captured image, and the processing of step 505 is applied only to the second image. Such embodiments can be provided, in particular, when the current phase is triggered by an event. No restrictions are attached to the event, which may be an event associated with a predetermined condition. The event may, for example, be that data captured by a sensor on the vehicle, separate from the camera 140, satisfies a predefined activation condition. For example, upon detection of an object on the road by another sensor on the vehicle, such as a lidar, the current phase of the method according to the invention This can be triggered to verify the detection of the same object by applying the processing of step 505 to the second image. This ensures redundancy in the data captured by another sensor on the vehicle.

[0109] The improvement of image processing of semantic perception, for example of semantic segmentation or object detection, thus allows a better reliability of the processed images transmitted to the AD AS 506 module, and thus makes it possible to improve the accuracy and reliability of at least one AD AS function implemented by the AD AS module as a function of the images processed by the processing module 150.

[0110] Fig. 6 illustrates the steps of a training phase of a light intensity map generation model, of a method for controlling a vehicle lighting module, according to embodiments of the invention.

[0111] As previously explained, the training phase of the light intensity map generation model 131 can be implemented in the drive system 400 described with reference to [Fig.4].

[0112] At a step 600, the training database 401 obtains a training data association as previously described, comprising a training image of a scene facing a vehicle, and reference data representative of the ground truth. According to embodiments described previously, other training information is further obtained. In what follows, it is assumed for the sake of simplicity that only the training image is submitted as input to the light intensity map generation module 131.

[0113] At a step 601, the obtained training image is submitted as input to the light intensity map generation model 131 (without the ground truth reference data), and the light intensity map generation model 131 generates a light intensity map based on the obtained training image.

[0114] The light intensity map is transmitted from the light intensity map generation model 131 to the synthesis module 402, which also receives the training image from the training database 400. At a step 602, the synthesis module 402 determines a synthetic image representative of the scene of the training image, illuminated by a pixelated lighting beam obtained from the light intensity map produced at the output of the light intensity map generation model 131.

[0115] The synthetic image is transmitted to the training image processing module 403, which applies processing to the synthetic image at a step 603, the processing being semantic perception processing, for example semantic segmentation or object detection.

[0116] The image processed in step 603 is transmitted from the training image processing module 403 to the loss evaluation module 404. At step 604, the module The loss assessment 404 evaluates a loss based on the ground truth reference data received from the training database 401, and based on the processed image received from the training image processing module.

[0117] At a step 605, the loss evaluation module 404 can further determine whether a predefined convergence criterion is met or not, based in particular on the loss evaluated at step 605, and optionally on losses evaluated during previous iterations of steps 600 to 604.

[0118] If the convergence criterion is not met, the loss evaluation module 404 modifies at least one parameter of the light intensity map generation model at a step 606, depending on the evaluated loss, according to a predefined training strategy.

[0119] Furthermore, in the second embodiment described above, the loss evaluation module 404 can modify at least one parameter of the image processing algorithm implemented by the training image processing module 403, at a step 607, as a function of the evaluated loss, according to a predefined training strategy.

[0120] Following step 606, or following step 607 in the second embodiment, the process returns to step 600 to repeat steps 600 to 605 on the basis of a new association of training data from training database 401.

[0121] The light intensity map generation model 131 can thus be trained to convergence. Alternatively, according to the second embodiment, the light intensity map generation model 131 and the image processing algorithm implemented by the training image processing module 403 are trained to convergence.

[0122] At step 608, when the loss evaluation module 404 determines that the convergence criterion is met, the training phase is completed and the light intensity map generation model can be implemented in the control device 130 for implementation of step 502 of the current phase of the process according to the invention. In the second embodiment, the image processing algorithm thus trained can be implemented in the processing module 150 of the vehicle.

[0123] Figure [Fig.7] illustrates the structure of a control device 130 according to embodiments of the invention.

[0124] The control device 130 includes a processor 701 configured to communicate unidirectionally or bidirectionally, via one or more buses or via a direct wired connection, with a memory 702 such as a "Random Access Memory", RAM, or a "Read Only Memory", ROM, or any other type of memory (Flash, EEPROM, etc.). Alternatively, the 702 memory includes several memories of the aforementioned types.

[0125] The memory 702 is capable of storing, permanently or temporarily, at least some of the data used and / or resulting from the implementation of the steps of the current phase of the process according to the invention illustrated with reference to [Fig. 5]. In particular, the memory 702 can store the light intensity map generation model 131, and can temporarily store the first image received at each step 501 of the current phase of the process according to the invention.

[0126] The processor 701 is capable of executing instructions, stored in memory 702, for the implementation of step 502 described above. Alternatively, the processor 701 can be replaced by a microcontroller designed and configured to perform step 502 described above.

[0127] The control device 130 includes a first interface 703 capable of receiving, in the step 501 described above, a first image captured by the camera 140.

[0128] The control device 130 includes a second interface 704 capable of transmitting the light intensity map to the lighting module 120 during step 503 described previously.

[0129] The present invention is not limited to the embodiments described above by way of example; it extends to other variants.

Claims

Demands

1. A method for controlling a vehicle (100) lighting module (120), the method comprising, during a typical phase, the following steps: - obtaining (500: 501) a first representative image of a scene facing the vehicle; - determining (502) a light intensity map, from a light intensity map generation model and the first image obtained, the light intensity map generation model (131) being configured to receive as input at least the first representative image of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​for controlling light elements (210) of a matrix source (200) of the vehicle's lighting module; - transmitting (503) the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle;- obtaining a second image representative of the scene facing the vehicle, following the projection of the pixelated lighting beam; - application (505) of an image processing to the second image obtained, the image processing being a semantic perception processing of the image.

2. A method according to claim 1, wherein the second image processed by the image processing is transmitted (506) to a driver assistance module (160) of the vehicle, capable of implementing at least one driver assistance function based on said second processed image.

3. A method according to claim 1 or 2, wherein the light intensity map generation model (131) has one of the following structures: - a convolutional neural network; - an artificial neural network of the autoencoder or variational autoencoder type; - a self-attentive or transformer model; or - a network generating a system of antagonistic generative networks.

4. A method according to any one of the preceding claims, further comprising a training phase of the light intensity map generation model, comprising a modification (606) of at least one parameter of the light intensity map generation model as a function of an evaluated loss (604) from an output of a training image processing algorithm applied by a training image processing module (403), the training image processing being a semantic perception processing.

5. A method according to claim 4, wherein the image processing algorithm for training the training phase is identical to the image processing applied during the current phase.

6. A method according to claim 4 or 5, wherein the training phase comprises the following steps: - obtaining (600) a training data association, the association comprising a training image representative of a scene and reference data; - applying (601) the light intensity map generation model to the training image, to obtain a training light intensity map; - obtaining (602) a synthetic image representative of the scene of the training image into which a lighting beam obtained as a function of the training light intensity map is projected; - applying (603) the training image processing algorithm to the synthetic image, to obtain a processed training image; - evaluating (604) a loss by comparing the processed training image with the reference data;- modification (606) of at least one parameter of the intensity light map generation model (131) based on the evaluated loss.;

7. A method according to claim 6, wherein the training phase further comprises a modification (607) of at least one parameter of the training image processing algorithm, and wherein the image processing applied during the current phase corresponds to the training image processing algorithm at the end of the training phase.

8. A method according to any one of the preceding claims, wherein the matrix source (200) of the lighting module (120) comprises electroluminescent semiconductor elements (210) of submillimeter dimensions, epitaxially directly onto a common substrate.

9. A vehicle assembly (100) comprising: - a camera (140) arranged to obtain a first image representative of a scene facing the vehicle; - a lighting module (120) comprising a matrix source (200) comprising a plurality of individually controllable light elements (210); - a control device (130) configured to determine a light intensity map, from a light intensity map generation model (131) and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control the light elements of the matrix source of the vehicle's lighting module;the control device being capable of transmitting the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle; wherein, upon obtaining a second image representative of the scene facing the vehicle by the camera, following the projection of the pixelated lighting beam, an image processing module (150) is configured to apply image processing to the second image obtained, the image processing being a semantic perception processing of the image.;

10. Vehicle (100) comprising an assembly according to claim 9, further comprising a driver assistance module (160), wherein the driver assistance module is configured to implement at least one driver assistance function based on the second image processed by the processing module.