Simulation of the projection of a light pattern in a scene

Abstract

The invention relates to a method for simulating the projection of a light pattern onto a scene, the method comprising the following steps: - obtaining (401; 402) at least one image representative of a scene; - obtaining (404) a light pattern for a motor vehicle lighting module comprising a matrix light source; - generating (405) a simulation of the projection of the light pattern onto the scene represented by the at least one image, by applying a generative model to the at least one image and to the light pattern obtained. FIG. 4

Description

Title of the invention: Simulation of the projection of a light pattern in a scene

[0001] The present invention relates to the field of simulating the projection of a light pattern from a motor vehicle lighting function in an environment external to the vehicle. More specifically, the invention relates to a system and a method for simulating the projection of a pixelated light pattern in an environment external to a vehicle.

[0002] Motor vehicles are now equipped with lighting modules comprising a matrix light source with a plurality of individually activatable light elements.

[0003] The number of light elements composing the matrix source of a lighting module increases, and can now reach several thousand or even several tens of thousands of individually activatable light elements. Such lighting modules are thus called high-definition, or HD, in that they allow the projection of a pixelated light beam creating a high-definition light pattern in a scene facing the motor vehicle.

[0004] This is particularly the case for technologies based on electroluminescent elements, individually controlled by a voltage applied to the terminals of each luminous element by a control unit. Such electroluminescent luminous elements may be on the same substrate, in which case the matrix light source is said to be monolithic. A so-called "monolithic" source may have a particularly high density of luminous elements.

[0005] High-definition lighting modules make it possible to develop new lighting functions, in particular in an adaptive lighting approach in which the pattern projected in the pixelated light beam is determined according to elements detected in the scene facing the vehicle, by one or more sensors of the vehicle.

[0006] Developing a lighting function capable of determining a high-definition light pattern to be projected, whether based on machine learning or a human operator, requires the ability to determine the result of projecting light patterns in various scenes that may be encountered by a vehicle in which the lighting function is to be implemented. Depending on the result, the lighting function can be validated or improved. Determining the projection result of the developed lighting function can be: - based on a projection by a high-definition module into a real environment, of the high-definition light pattern from the lighting function. However, such an approach is complex to implement and requires constantly interrupting the development process to test the lighting function, which is incompatible with machine learning of the lighting function, and makes development by a human operator very long and tedious; - based on a simulation of the projection by a high-definition module in a real environment. However, current projection simulation methods are lengthy, and obtaining an accurate result can take several minutes, making this solution incompatible with development processes, whether by machine learning or by a human developer; - based on a simulation in a synthetic (or simulated) environment which allows for rapid simulation but does not give a realistic result.

[0007] There is therefore a need to facilitate the development of lighting functions for a high-definition lighting module. In particular, it is desirable to allow for the real-time acquisition of a precise result from the projection of a light pattern generated by a lighting function, for example during the development of the lighting function.

[0008] The present invention improves the situation.

[0009] A first aspect of the invention relates to a method for simulating the projection of a luminous pattern in a scene, the method comprising the following steps: - obtaining at least one image representative of a scene; - obtaining a light pattern for a motor vehicle lighting module including a matrix light source; - generation of a simulation of projection of the light pattern in the scene represented by at least one image, by applying a generative model to the at least one image and to the light pattern obtained.

[0010] Thus, the invention provides for the use of a generative model taking as input a representative image of a scene and a light pattern to be projected onto that scene in order to simulate the result of projecting the light pattern onto the scene. This makes it possible to quickly obtain a simulated projection result.

[0011] According to embodiments, the light pattern can be obtained from a lighting function module capable of implementing a lighting function for a motor vehicle, and the method can further include an update of the lighting function according to the simulation of projection of the light pattern in the scene.

[0012] Thus, it becomes possible to use real-time simulation in a lighting function development process, whether by machine learning or by a human operator. The development of the lighting function is thus facilitated and accelerated.

[0013] In addition, the light pattern can be obtained by the lighting function module as a function of at least one representative image of the scene.

[0014] Thus, it is made possible to develop an adaptive lighting function.

[0015] According to a first embodiment, the method may further include an evaluation of a loss based on a comparison between an expected result and the projection simulation and the lighting function may be updated from the evaluated loss.

[0016] Projection simulation can thus be advantageously exploited during machine learning of the lighting function.

[0017] Alternatively, according to a second embodiment, the method may further include a presentation of the projection simulation to a human operator, and the lighting function may be updated following said presentation.

[0018] Projection simulation can thus be advantageously exploited during development by a human operator of the lighting function.

[0019] According to embodiments, the at least one image obtained may include a first color image representative of the scene.

[0020] Thus, the generative model can be capable of modifying the first color image received to integrate the light pattern when obtaining the projection simulation.

[0021] In addition, the at least one image obtained may further include a second image, said second image being a depth image representative of the same scene as the first image.

[0022] Such an embodiment makes it possible to improve the accuracy associated with the simulation of projection of the light pattern.

[0023] According to some embodiments, the generative model can be: - a convolutional neural network; - an artificial neural network of the autoencoder or variational autoencoder type; or - a network generating a system of antagonistic generative networks; - a self-aware or transformative model; - a diffusion model.

[0024] Such generative model structures are particularly suited to generating a projection simulation in the form of an image.

[0025] In addition, the method may further include a learning phase, the learning phase comprising the following steps: - obtaining at least one representative training image of a scene from a training database; - obtaining a training light pattern associated with at least one training image in the training database; - generation of a simulation of the training light pattern projection in the scene represented by at least one training image, by applying the generative model to the training light pattern and to at least one training image; - evaluation of a loss between the training projection simulation and a reference image associated with at least one training image in the training database; - modification of at least one parameter of the generative model as a function of the evaluated loss. The steps of the learning phase can be repeated for multiple training images from the training database.

[0026] Thus, the generative model is derived from machine learning. This facilitates its acquisition and improves its accuracy, particularly when the training database includes data corresponding to varied driving situations.

[0027] A second aspect of the invention relates to a computer program comprising instructions for implementing the method according to the first aspect of the invention, when these instructions are executed by a processor.

[0028] A third aspect of the invention relates to a module for generating a simulation of the projection of a light pattern in a scene, the generation module comprising: - a first interface for receiving at least one image representative of a scene; - a second interface for receiving a light pattern for a motor vehicle lighting module comprising a matrix light source; - a processor configured to generate a simulation of the projection of the light pattern into the scene represented by at least one image, by applying a generative model to the at least one image and to the light pattern obtained.

[0029] A fourth aspect of the invention relates to a simulation system for the projection of a light pattern in a scene, comprising a generation module according to the third aspect of the invention and a lighting function module capable of implementing a motor vehicle lighting function to obtain the light pattern.

[0030] Other features and advantages of the invention will become apparent from an examination of the detailed description below, and the accompanying drawings in which:

[0031] [Fig.1] illustrates a simulation system for projecting a light pattern into a scene, according to embodiments of the invention;

[0032] [Fig.2] illustrates a drive system for a module generating a simulation of the projection of a light pattern in a scene, according to embodiments of the invention;

[0033] [Fig.3] illustrates a method of training a module for generating a simulation of the projection of a light pattern in a scene, according to embodiments of the invention;

[0034] [Fig.4] illustrates a method for simulating the projection of a luminous pattern in a scene, according to embodiments of the invention;

[0035] [Fig.5] illustrates the structure of a module for generating a simulation of the projection of a light pattern in a scene, according to embodiments of the invention.

[0036] The description focuses on the characteristics that distinguish the system and the method from those known in the state of the art.

[0037] Fig. 1 illustrates a simulation system 100 of a projection of a light pattern in a scene, according to embodiments of the invention.

[0038] The simulation system 100 may include a projection simulation generation module 101, according to embodiments, the generation module 101 storing or remotely accessing a generative projection simulation model, obtained by machine learning by the training system 200 described below with reference to [Fig.2].

[0039] The generative model implemented by the generation module 101 is capable of receiving as input at least one image representative of a scene, and a light pattern, preferably high-definition, generated by a lighting function module 102, and of generating as output a simulation of a projection of the received light pattern, in the scene described by said at least one image received as input.

[0040] At least one representative scene image may be taken from at least one first database 103 storing a plurality of first representative scene images facing the vehicle. The first images in the first database 103 may be color images, for example, images in RGB format, for “Red-Green-Blue”. However, there are no restrictions on the format of the color images in the first database 103.

[0041] The first images of the first database 103 may be: - simulated images of fictional scenes; - images of real scenes, captured in real driving situations, for example by cameras arranged respectively in vehicles of a group of vehicles in different driving situations.

[0042] The generative model of the generation module 101 is thus able to take as input the light pattern from the lighting function module 102 and a first color image from the first database 103.

[0043] No restrictions are attached to the format of the light pattern from the lighting function module 102: it can be a black and white image, or in greyscale, in high resolution, the brightness of each pixel of the light pattern being able to be interpreted by a lighting module with a high-definition matrix source, as a command to control the brightness of each light element of the matrix source, so as to project the light pattern into a scene facing a vehicle in which the lighting module is integrated.

[0044] Furthermore, according to embodiments of the invention, the generative model is capable of receiving as input, in addition to the first color image representing a scene and a light pattern, a second image representing the same scene as the first color image, the second image being a depth image of the scene.

[0045] The second depth image can be taken from a second database 104 storing depth images associated with the color images in the first database 103. Thus, each pair consisting of a first image from the first database 103 and a second associated image from the second database 104 is representative of a scene facing a vehicle. Alternatively, as shown in [Fig. 1], a single database can contain the first and second images stored together.

[0046] The depth images of the second database 104 can be: - simulated, or synthesized, images of fictional scenes; - images of real scenes, captured in real driving situations, for example by depth cameras, or by combinations of lidar and a depth estimation algorithm, arranged respectively in vehicles of a set of vehicles in different driving situations.

[0047] The first and second images can both be simulated in the same simulation environment. Alternatively, the first and second images are both acquired in real-world driving situations, each pair comprising a first and a second image and being obtained by a set of sensors on the same vehicle during a given driving situation. Alternatively, the first images can be acquired in real-world driving situations by a set of vehicles, and each second image is synthesized from an associated first image.

[0048] Taking into account depth images as input to the generative model for simulating light pattern projection makes it possible to improve the associated accuracy to the simulation of the projection of the light pattern from the lighting function module 102.

[0049] The simulation system 100 according to the invention can advantageously be operated by a module 105 or 106 placed at the output of the generation module 101.

[0050] In a first embodiment, the lighting function implemented by the lighting function module 102 is trained by machine learning. In this case, the lighting function is a generative model of a light pattern defined by a set of parameters that are trained by machine learning.

[0051] In this first embodiment, the projection simulation generated by the generation module 101 can be transmitted to a loss evaluation module 105, which is capable of evaluating a loss between an expected result and the projection simulation. No restrictions are attached to the expected result, which can be defined from criteria defining constraints of the lighting function, for example.

[0052] Based on the loss thus evaluated, the parameters of the lighting function are modified or optimized. The generation of the projection simulation by the generation module 101 thus allows for a real-time evaluation of the lighting function during its machine learning. The first database 103, and optionally the second database 104, can therefore be considered as containing training and test data for the lighting function implemented by the lighting function module 102.

[0053] In a second embodiment, the lighting function is manually parameterized by a human operator responsible for developing the lighting function. In the second embodiment, the projection simulation generated by the generation module 101 can be transmitted to a presentation module 106 capable of presenting the projection simulation as the result of the projected light pattern generated by the lighting function under development.

[0054] The human operator can thus, if necessary, modify parameters of the lighting function, and then restart a projection simulation with the same first image, or with the same pair of first and second images, or with a new first image, or with a new pair of first and second images. The generation of the projection simulation by the generation module 101 thus allows for real-time evaluation of the lighting function being developed by a human operator.

[0055] As previously stated, the lighting function module 102 can implement a lighting function capable of determining a light pattern to be projected, for example as a function of at least one descriptive image of the scene (the first image, and optionally as a function of a first image and second image pair).

[0056] According to embodiments of the invention, the generative model implemented by the generation module 101 is trained during a machine learning training phase, illustrated with reference to Figures 2 and 3 described below, the machine learning enabling the training of the parameters defining the generative model, the generative model having a predefined structure. The generative model may, for example, have one of the following structures: - a convolutional neural network, such as a U-Net type network for example; - an artificial neural network of the auto-encoder or variational auto-encoder type, also called VAE in English; - a generating network of a system of generative adversarial networks, also called GAN, for “Generative Adversarial Networks” in English;

[0057] - a self-attentive or transformer model; - a diffusion model - any other generative model structure capable of receiving as input one or more images representing a scene, and a light pattern generated by the lighting function.

[0058] No restrictions are attached to the machine learning applied to the generative model of module 101 to optimize its parameters. The machine learning can, for example, be of the supervised type, implemented by the training system 200 shown in [Fig. 2], using a training database 201 comprising associations between: - at least one representative image of a scene, for example a first image in RGB format of a scene, and optionally a depth image of the same scene; - a luminous pattern to be projected onto the stage; - a reference image representing the actual projection of the light pattern in the scene corresponding to at least one image, the image representing the projection indicating the “ground truth”, from which a loss function can be applied to evaluate the difference between the projection simulation of the generative model and the ground truth, which allows the optimization of parameters during supervised learning.

[0059] The associations stored in the training database 201 can be obtained by collecting, for each scene of a set of real scenes, the scenes of the set being preferably varied: - a color image, for example in RGB format, which can be obtained from a vehicle camera, optionally with an associated depth map, without projection of a light pattern; - a reference image, obtained by the same vehicle camera, after projection of the light pattern into the same scene by a vehicle lighting module; - the light pattern projected before obtaining the reference image.

[0060] The various scenes can be varied by corresponding to various driving situations, for example driving in the city, on the motorway, in the countryside, and / or in varied weather conditions.

[0061] The drive system 200 further includes a drive module 202 capable of evaluating a loss function as a function of the outputs of the generation module 101 and of optimizing the parameters of the generative model implemented in the generation module 101. Figure 3 represents the steps of a training phase of the generative projection simulation model, of a projection simulation method according to embodiments of the invention.

[0062] The training phase is implemented in the training system 200 shown in [Fig.2], and aims to obtain the generative model implemented in the generation module 101 described previously.

[0063] Initially, the generation module 101 stores a generative projection simulation model according to a given structure (among the structures described previously, for example), with non-optimized parameters.

[0064] At a step 300, the generation module 101 obtains a first training image, such as an RGB format training image, from the training database 201 described previously.

[0065] At a step 301, the generation module 101 obtains a training light pattern associated with the first training image obtained at step 300, in the training database 201.

[0066] At an optional step 302 (implemented when the generation module 101 is capable of also taking a depth image as input), the generation module 101 obtains a second training image, which is a depth image, associated with the first training image obtained in step 300 in the training database 201.

[0067] As shown in [Fig.3], steps 300 and 301, and optionally 302, can be carried out in parallel.

[0068] At a step 304, the generation module 101 applies the generative projection simulation model to the data obtained in steps 300 and 301, and optionally in step 302. Following the application of the generative model, the generation module 101 generates in step 304 a projection simulation of the training light pattern in the scene corresponding to the first training image, and optionally to the second training image.

[0069] At a step 303, implemented in parallel with the steps described above, or following the implementation of step 304, the training module 202 obtains the image reference data associated with the data obtained in steps 300 and 301, and optionally 302, in training database 201.

[0070] At a step 305, following steps 303 and 304, the training module 305 can evaluate a loss by comparing the reference image obtained at step 303, with the projection simulation from the generative model during step 304.

[0071] At a step 306, the training module 202 can update, or optimize, the parameters of the generative projection simulation model, according to the loss determined at step 305.

[0072] At a step 307, the training module 202 can determine whether a convergence criterion for the generative model is met or not, for example, based on the evaluated loss. If the training module 202 determines that the convergence criterion is met by the generative model, the training phase is completed and the trained generation module can be implemented by the generation module in the simulation system 100 described with reference to [Fig. 1]. The process can then proceed to a current phase described below with reference to [Fig. 4].

[0073] Otherwise, if the training module 202 determines in step 307 that the convergence criterion is not met, then the process returns to steps 300 to 303, to continue training the generation module on the basis of another association of training data stored in the training database 201.

[0074] Fig. 4 illustrates a common phase of a method for simulating the projection of a light pattern of a lighting function in a scene, according to embodiments of the invention.

[0075] The current phase is implemented in the simulation system 100 described with reference to [Fig.1].

[0076] At a step 401, the generation module 101 obtains a first image, preferably a color image, for example in RGB format, representative of a scene, from the first database 103.

[0077] At an optional step 402, the generation module 101 obtains a second image, preferably a depth image, representative of the same scene as the first image obtained at step 401, from the second database 104.

[0078] In parallel with steps 401 and 402, in a step 403, the lighting function module 102 implementing the lighting function under development (by machine learning or developed by a human operator) generates a light pattern of the lighting function, for example in the form of a black and white image representative of the light pattern, interpretable by a lighting module including a matrix source, preferably high-definition.

[0079] Preferably, the lighting pattern can be generated according to the scene represented in the first image obtained by the generation module 101 in step 401. For this purpose, the lighting function module 102 can obtain the same first image as that obtained by the generation module 101 in step 401, and the lighting pattern is determined according to the first image, thus enabling the realization of an adaptive lighting function.

[0080] At a step 404, the generation module 101 obtains the light pattern generated at step 403 by the lighting function module 102.

[0081] At a step 405, the generation module 101 applies the previously trained generative model to the first image obtained in step 401, to the light pattern obtained in step 404, and optionally to the second image obtained in step 402. By applying the generative model, the generation module 101 thus determines a simulation of the projection of the light pattern into the scene represented by the first image, and optionally by the second image.

[0082] Steps 406 and 407 are implemented in the first embodiment in which the lighting function implemented by the lighting function module 102 is trained by machine learning.

[0083] At a step 408, the loss evaluation module 105 evaluates a loss based on the expected result and the projection simulation generated by the generation module 101 at step 405.

[0084] At a step 409, the loss evaluation module 105 can modify the parameters of the lighting function in the lighting function module 102. Depending on the determined loss, the loss evaluation module 105 can alternatively validate the lighting function implemented by the lighting function module 102.

[0085] Steps 408 and 409 are implemented in the second embodiment in which the lighting function implemented by the lighting function module 102 is developed by a human operator.

[0086] At a step 408, the presentation module 106 presents the simulation of the projection of the light pattern in the scene to the human operator.

[0087] Based on the projection simulation thus presented, the human operator can modify the parameters of the lighting function implemented by the lighting function module 102 at a step 409.

[0088] Thus, the generative projection simulation model makes it possible to improve, facilitate and accelerate the development of a lighting function, whether by development by a human operator or by machine learning.

[0089] Figure 5 presents a structure of the generation module 101 for simulating the projection of a light pattern in a scene, according to embodiments of the invention.

[0090] The generation module 101 includes a processor 501 configured to communicate unidirectionally or bidirectionally, via one or more buses or via a direct wired connection, with a memory 502 such as a Random Access Memory (RAM), a Read Only Memory (ROM), or any other type of memory (Flash, EEPROM, etc.). Alternatively, the memory 502 comprises several memories of the aforementioned types.

[0091] Memory 502 can in particular store the generative projection simulation model described above, during the training phase and during the current phase of the process described above.

[0092] Memory 502 can, for example, store instructions enabling the execution of the generative projection simulation model described above.

[0093] The generation module 101 includes a first receiving interface 503 capable of obtaining a first image of the first database 103 during the current phase, and of obtaining a first training image of the training database 201 during the training phase.

[0094] The generation module 101 further includes a second receiving interface 504 capable of obtaining the light pattern from the lighting function module 102 during the current phase, and the training light pattern from the training database 201 during the training phase.

[0095] According to certain embodiments of the invention, the generation module 101 may further include a third receiving interface 505 capable of obtaining a second image of the second database 104 during the current phase, and of obtaining a second training image of the training database 201 during the training phase.

[0096] The present invention is not limited to the embodiments described above by way of example; it extends to other variants.

Claims

Demands

1. A method for simulating the projection of a light pattern into a scene, the method being implemented by a generation module comprising a first interface for receiving at least one image representative of a scene; a second interface for receiving a light pattern and a processor, the method comprising the following steps: - obtaining (401; 402) at least one image representative of a scene; - obtaining (404) a light pattern for a motor vehicle lighting module comprising a matrix light source; - generating (405) a simulation of the projection of the light pattern into the scene represented by the at least one image, by applying a generative model to the at least one image and to the light pattern obtained.

2. A method according to claim 1, wherein the light pattern is obtained from a lighting function module (102) capable of implementing a lighting function for a motor vehicle, and wherein the method further comprises an update (407; 409) of the lighting function as a function of the simulation of projection of the light pattern in the scene.

3. Method according to claim 2, wherein the light pattern is obtained (404) by the lighting function module as a function of at least one image representative of the scene.

4. A method according to claim 2 or 3, further comprising an evaluation (406) of a loss based on a comparison between an expected result and the projection simulation; and wherein the lighting function is updated (407) from the evaluated loss.

5. A method according to claim 2 or 3, further comprising a presentation (408) of the projection simulation to a human operator, and wherein the lighting function is updated (409) following said presentation.

6. A method according to any one of the preceding claims, wherein the at least one image obtained comprises a first color image representative of the scene.

7. A method according to claim 6, wherein the at least one image obtained further comprises a second image, said second image being a depth image representing the same scene as the first image.

8. A method according to any one of the preceding claims, wherein the generative model is: - a convolutional neural network; - an artificial neural network of the autoencoder or variational autoencoder type; or - a generating network of a system of antagonistic generative networks; - a self-attentive or transforming model; - a diffusion model.

9. A method according to claim 8, further comprising a learning phase, the learning phase comprising the following steps: - obtaining (300; 301) at least one representative training image of a scene from a training database (201); - obtaining (302) a training light pattern associated with the at least one training image in the training database; - generating (304) a training projection simulation of the training light pattern in the scene represented by the at least one training image, by applying the generative model to the training light pattern and to the at least one training image; - evaluating (305) a loss between the training projection simulation and a reference image associated with the at least one training image in the training database; - modifying (306) at least one parameter of the generative model as a function of the evaluated loss;in which the steps of the learning phase are repeated for several training images from the training database.;

10. Computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by a processor (501).

11. A generation module (101) for a simulation of the projection of a light pattern in a scene, the generation module comprising: - a first receiving interface (503) of at least one image representative of a scene; - a second receiving interface (504) of a light pattern for a motor vehicle lighting module comprising a matrix light source; - a processor (501) configured to generate a simulation of the projection of the light pattern into the scene represented by at least one image, by applying a generative model to the at least one image and to the light pattern obtained.

12. Simulation system (100) of a projection of a light pattern in a scene, comprising a generation module (101) according to claim 11 and a lighting function module (102) capable of implementing a motor vehicle lighting function to obtain the light pattern.