Method and device for controlling an augmented reality content display device embedded in a vehicle comprising a camera

Abstract

The present invention relates to a method and device for controlling an augmented reality content display (11), referred to as RX content, embedded in a vehicle (10). The vehicle (10) comprises at least one camera (101) configured for acquiring images of an environment external to the vehicle (10). To this end, image data representative of an external lighting environment of the vehicle (10) are received from at least one camera (101). Illumination data representative of the lighting of a set of virtual graphic objects of the RX content are determined based on the image data. The RX content display (11) is controlled to display the set of virtual graphic objects in augmented reality based on the illumination data. Figure 1

Description

Title of the invention: Method and device for controlling an augmented reality content display device embedded in a vehicle comprising a camera. Technical field

[0001] The invention relates to methods and devices for controlling an augmented reality content display device, referred to as RX content, embedded in a vehicle, particularly but not exclusively a motor vehicle. The invention also relates to a method and device for determining illumination data for a set of virtual objects in RX content. Technological background

[0002] Augmented reality or mixed reality content display devices, known as RX content, are experiencing significant growth, particularly in fields such as health, education, entertainment, and industry. These RX content display devices include, in particular, head-up displays, also called viewing headsets or immersive headsets, as well as vehicle-mounted display devices such as head-up displays or HUDs.Such RX content display devices are used, for example, in the automotive field to display or project virtual graphic objects superimposed on the real world in one or more image planes located at a certain distance from the person viewing these virtual graphic objects. These objects are located, in particular but not exclusively, outside the vehicle, whose lighting environment changes according to the vehicle's movement and external conditions, including weather, type of environment, or time of day. The virtual graphic objects thus displayed are therefore not always visible or are displayed with color or lighting characteristics that are inconsistent with this external environment. Summary of the present invention

[0003] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0004] An object of the present invention is, for example, to improve the rendering of virtual graphic object(s) by an augmented reality content display device embedded in a vehicle.

[0005] According to a first aspect, the present invention relates to a method for controlling an augmented reality content display device, referred to as RX content, embedded in a vehicle, the vehicle comprising at least one camera configured for a acquisition of images of an environment external to the vehicle, the process being implemented by at least one processor and comprising the following steps: - receiving image data representative of a luminous environment outside the vehicle from at least one camera; - determination of illumination data representative of the lighting of a set of virtual graphic objects of the RX content as a function of image data; - control of the RX content display device to display the set of virtual graphic objects in augmented reality based on illumination data.

[0006] Acquiring images of the vehicle's external environment using one or more vehicle cameras provides image data on the current external environment in which the vehicle is located, as the vehicle moves. Determining illumination data from this image data allows each virtual graphic object of the RX content displayed by the vehicle's onboard display to be illuminated and rendered in accordance with the current lighting conditions of the environment in which the vehicle is moving.

[0007] The use of one or more cameras equipping the vehicle to determine the illumination data also makes it possible to avoid equipping the display device with such a camera, which reduces the manufacturing and design costs of the RX content display device.

[0008] According to one variant, the determination of the illumination data includes the generation of an environment map from the image data.

[0009] According to a further variant, at least one camera belongs to a set of cameras comprising: - a roof camera mounted on the roof of the vehicle; - a front camera arranged in a vehicle's passenger compartment behind a vehicle's windshield; - a sensor for the brightness of the external environment.

[0010] According to yet another variant, the roof camera corresponds to a panoramic camera, the image data being representative of a panoramic image of the luminous environment outside the vehicle when at least one camera corresponds to the roof camera.

[0011] According to an additional variant, the image data corresponds to high dynamic range image data.

[0012] According to an additional variant, the image data is received by the RX content display device via a wireless connection linking the RX content display device to the vehicle.

[0013] According to a second aspect, the present invention relates to a control device for a display device for augmented reality content, referred to as RX content, embedded in a vehicle, the device comprising a memory associated with a processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0014] According to a third aspect, the present invention relates to a vehicle, for example a motor vehicle, comprising a device as described above according to the second aspect of the present invention.

[0015] According to one variant, the vehicle includes at least one camera and the vehicle also carries an augmented reality content display device, the at least one camera being connected in communication to the augmented reality content display device.

[0016] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0017] Such a computer program may use any programming language, and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0018] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0019] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0020] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0021] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0022] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 3, in which:

[0023] [Fig-1] schematically illustrates a vehicle equipped with a display device RX content, according to a particular embodiment of the present invention;

[0024] [Fig.2] illustrates a device configured to control the RX content display device of [Fig.1], according to a particular and non-limiting embodiment of the present invention.

[0025] [Fig.3] illustrates a flowchart of the different stages of a method for controlling the RX content display device of [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0026] A method and a control device for an augmented reality content display device, called RX content, embedded in a vehicle will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

[0027] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0028] Fig. 1 schematically illustrates a vehicle 10, according to a particular and non-limiting embodiment of the present invention.

[0029] Vehicle 10 refers to any moving vehicle, for example, a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle, such as a car, a truck, or a bus.

[0030] According to other examples, the vehicle corresponds to a bicycle, a boat, an airplane.

[0031] The vehicle 10 includes or carries one or more cameras or light-detecting devices, the term "camera" being used in the rest of the description to cover any device comprising a set of photosensitive sensors.

[0032] The vehicle 10 thus comprises one or more of the following cameras, according to all possible combinations: - one or more cameras 101 arranged on the roof of the vehicle 10 and configured for acquiring images of the external environment of the vehicle 10, i.e. the environment in which the vehicle 10 is traveling; and / or - a so-called front camera (also called a windshield camera), such a front camera being arranged in the passenger compartment of the vehicle 10 under / behind the windshield and at the top and in the middle of this windshield and having a field of vision corresponding to the space located in front of the vehicle 10 according to the direction of travel of the vehicle 10; and / or - one or more ambient light sensors, such a sensor being arranged for example in the passenger compartment behind the windshield of the vehicle 10.

[0033] The camera 101 and / or the front camera includes for example the following elements: - a photosensitive sensor corresponding for example to a matrix of photoreceptors associated for example with a Bayer filter; - an optical assembly arranged in front of the sensor with respect to the scene to be acquired by the sensor, the optical assembly comprising, for example, an arrangement of one or more lenses; and - optionally one or more computers associated with memory and configured for processing images acquired by the sensor.

[0034] According to a particular embodiment, the camera 101, referred to as the roof camera, corresponds, for example, to an arrangement of cameras, each oriented so as to cover a part of the external environment (for example, 3 cameras each covering 1 / 3 of the environment located above and around the vehicle 10, or 4 cameras each covering 1 / 4 of the environment located above and around the vehicle 10). According to such an example, an image of the external environment is obtained by compositing the images acquired at the same time instant of acquisition by the set of cameras forming the camera arrangement.

[0035] According to another particular embodiment, the roof camera 101 corresponds to a panoramic camera configured for the acquisition of images of the external environment according to a field of vision equal to 360°, such a camera being for example of the "fisheye" type.

[0036] Camera 101 (and / or the front camera) corresponds, for example, to an LDR (Low-Dynamic Range) camera, the luminance values ​​acquired by the photosensitive sensor being, for example, encoded on 8 bits for each color channel (for example, for each RGB (Red, Green, Blue)) channel). According to another example, camera 101 corresponds, for example, to an HDR (High-Dynamic Range) camera. French"), the luminance values ​​acquired by the photosensitive sensor being, for example, coded on 10 or 12 bits for each color channel.

[0037] The vehicle 10 also includes a display device for augmented reality or mixed reality content. In the remainder of this description, the term augmented reality, abbreviated as RX, is used, and RX content encompasses all augmented reality or mixed reality content.

[0038] The RX content display device corresponds for example to a visor head-mounted display device 11 worn on the head of a user (passenger or driver) of the vehicle 10.

[0039] A head-mounted display 11 corresponds to a system or device comprising a display screen for displaying one or more virtual objects of RX content. The head-mounted display 11 corresponds to a headset equipped with an integrated display device on which computer-generated synthetic images are displayed or projected under the control of a controller comprising one or more processors. The synthetic images thus displayed or projected are superimposed on the user's real vision. The head-mounted display 11 encompasses any RX content display device, for example, devices such as an immersive headset, a viewing headset, an HMD (Head-Mounted Display), RX glasses (called "see-through glasses"), or a semi-transparent headset (or "see-through HMD").

[0040] According to another example, the RX content display device corresponds to an image projection system for the vehicle 10 configured to display images comprising one or more virtual graphic objects on a glazed surface of the vehicle 10, for example on the windshield or on a transparent strip rising from the dashboard behind the steering wheel of the vehicle 10. Such an RX content display device includes, for example, a projector integrated into the dashboard. Such an RX content display device thus corresponds to a Head-Up Display (HUD) system, which allows virtual objects to be superimposed onto the field of vision of the driver seated in the driver's seat in a driving position of the vehicle 10, for example on the windshield of the vehicle 10, so as to overlay the virtual objects onto the real road scene.

[0041] A control process for an RX content display device embedded in the vehicle 10 is advantageously implemented by one or more processors, for example, by one or more processors of a device embedded in the vehicle 10 and corresponding to a computer or set of computers of the vehicle's embedded network 10. According to another example, the process is implemented by a device a computer-type device acting as an embedded rendering engine in vehicle 10 and connected to vehicle 10's embedded network via a wired or wireless connection. In another example, the device corresponds to a controller for the display device, particularly when the display device is a head-mounted display 11, the head-mounted display 11 then being connected to vehicle 10, specifically to the camera(s), via a wired or wireless connection.

[0042] In a first operation of the process, image data representative of a luminous environment outside the vehicle 10 are received from one or more cameras of the vehicle 10.

[0043] The process is described below with reference to a particular embodiment in which image data is received from the roof camera 101. Of course, the invention is not limited to such an embodiment but extends to any embodiment in which image data is received from a vehicle camera 10 other than the roof camera 101 or from several cameras.

[0044] The image data corresponds to representative luminance data coded on 8, 10, 12 or 14 bits for each color channel of the color space used by the camera 101, for example color space 101.

[0045] The image data obtained from the photosensitive sensor are thus representative of the luminous environment associated with the external environment of the vehicle 10 with colour data and light intensity data according to a set of directions belonging to the field of vision or acquisition of the camera 101.

[0046] Image data is received for example via one or more data buses of the vehicle's embedded network 10 when the device implementing the process corresponds to a computer of the vehicle 10, such as the IVI computer (from the English "In-Vehicle Infotainment" or in French "Infodivertissement épâble") for example.

[0047] According to another example, the image data is received via a wireless connection (for example, Bluetooth® or Wifi® type), in particular when the device implementing the process corresponds to a controller of the head-mounted display 11. According to this example, the controller is connected directly to the camera 101 via the wireless connection or to a wireless communication interface of the vehicle 10 connected to the camera 101 via one or more data buses of the vehicle 10's on-board network.

[0048] In a second operation of the process, illumination data is determined from the image data received in the first operation. This illumination data is representative of the lighting of a set of virtual graphic objects of the RX content to be displayed or projected via the RX content display device.

[0049] This or these virtual graphic object(s) each correspond to a 2D (two-dimensional) or 3D (three-dimensional) computer-generated object. A graphic object A virtual object, for example, represents a real-world object, such as a road sign, a vehicle, a weather event, or a point of interest like a monument, a shop, a piece of street furniture, or an intersection. In other examples, a virtual graphic object represents a shape or an icon, such as navigation icons to indicate the direction to follow when the navigation system is activated.

[0050] As known to a person skilled in the art, the lighting and / or color of a virtual graphic object to be displayed via the RX content display device is rendered by implementing a global illumination method or a local illumination method based on image data of the lighting environment.

[0051] Such methods are described for example in the document entitled "Real-time direct lighting with surface sources and arbitrary BRDFs", by Sylvain Meunier, published in 2007 in "Image Synthesis and Virtual Reality" in 2007 under the reference inria-00598444, or in the document entitled "HDR Environment Map Estimation for Real-Time Augmented Reality", by Gowri Somanath and Daniel Kurz, published on July 27, 2021 under the reference arXiv:2011.10687v5.

[0052] The determination of the illumination data includes the generation of an environment map from the image data obtained in the first operation of the process.

[0053] In a third operation of the process, the RX content display device is controlled to display the set of virtual graphic objects in augmented reality according to the illumination data obtained in the second operation.

[0054] A display control of RX content or any virtual object included in such RX content by the RX content display device includes rendering the RX content, such rendering corresponding to a set of operations performed by one or more processors on the pixels of one or more images of the graphic content to be displayed or projected as an overlay on the real world. For example, rendering consists of associating a set of pixels in an image with pixel data (for example, color data expressed in an RGB (Red, Green, Blue) color space) associated with each virtual graphic object.The control of displaying graphic content thus includes the transmission of control signals to the screen of the RX content display device to modify the values ​​associated with the screen pixels at the location intended to display the indicator (or any graphic object) or to the projector(s) of the RX content display device to project the virtual graphic object(s) onto the transparent or semi-transparent surface according to the pixel data associated with each virtual graphic object.

[0055] Displaying a set of virtual objects in augmented reality amounts to displaying one or more virtual graphic objects (for example computer-generated) superimposed on the real world visible to the user wearing the head-mounted display 11, the real world being directly visible to the user when the head-mounted display 11 is of the transparent (or "see-through") type or the real world being visible via images of this real environment acquired by a camera of the head-mounted display 11. When the RX content display device corresponds to a VTH system, the set of virtual graphic objects is visible to the driver through the windshield.

[0056] These virtual graphic objects are for example displayed or projected in one or more image planes each having a depth such that these virtual graphic objects are seen in superposition of the real world or of the real scene at a distance corresponding to the depth associated with the image plane, these virtual graphic objects appearing advantageously beyond the windshield or any glass wall through which the user looks, the virtual graphic objects thus being seen in the external environment of the vehicle 10.

[0057] Such a process makes it possible to adapt the lighting of each virtual graphic object so that this lighting corresponds to the lighting of the real world according to the movement of the vehicle 10. The light sources of the real world are numerous and depend for example on the environment, the time of day, the weather and include for example the sun, artificial light sources such as street lighting, the headlights of other vehicles, the reflection of these light sources on reflective surfaces of the environment in which the vehicle 10 travels, etc.

[0058] The acquisition of images of the luminous environment traversed by the vehicle 10 by one or more cameras of the vehicle 10 makes it possible to calculate or estimate illumination data realistically representing the lighting of the external environment traversed by the vehicle 10. The virtual graphic objects then displayed are seen in consistency with real world objects, with lighting similar to or approaching that of real world objects, thus improving the realism of these virtual graphic objects.

[0059] Figure 2 schematically illustrates a device 2 configured to control an RX content display device embedded in a vehicle, for example, vehicle 10, according to specific and non-limiting embodiments of the present invention. Device 2 corresponds, for example, to a device embedded in the vehicle 10, such as a computer or set of computers. In another example, device 2 corresponds to a control unit of the head-mounted display 11.

[0060] Device 2 is, for example, configured to carry out the operations described opposite [Fig. 1] and / or the steps of the process described opposite [Fig. 3]. Examples of such a device 2 include, but are not limited to, a Embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a computer, a server, or a combination of several of the devices listed above. The elements of Device 2, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 2 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0061] The device 2 comprises one (or more) processor(s) 20 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 2. The processor 20 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 2 further comprises at least one memory 21, corresponding, for example, to volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0062] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 21.

[0063] According to various particular and non-limiting embodiments, the device 2 is coupled in communication with other similar devices or systems and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0064] According to a particular and non-limiting embodiment, the device 2 comprises a block 22 of interface elements for communicating with external devices. The interface elements of the block 22 comprise one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - at least one wired interface of type USB (Universal Serial Bus) or CAN (Controller Area Network) » or in French « Controller Network"> or Ethernet (according to the ISO / IEC 802-3 standard); - HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0065] According to another particular and non-limiting embodiment, the device 2 includes a communication interface 23 which allows communication to be established with other devices (such as other computers in the embedded system) via a communication channel 230. The communication interface 23 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 230. The communication interface 23 corresponds, for example, to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3).

[0066] According to a particular and non-limiting embodiment, the device 2 can provide output signals to one or more external devices, such as a display screen 240, touch or not, one or more speakers 250 and / or other peripherals 260 (projection system) via output interfaces 24, 25 and 26 respectively. According to a variant, one or more of the external devices is integrated into the device 2.

[0067] Figure 3 illustrates a flowchart of the various steps in a method for controlling an RX content display device embedded in a vehicle, for example vehicle 10, according to a particular and non-limiting embodiment of the present invention. The method is implemented, for example, by a device embedded in the vehicle 10 or in the RX content display device, or by device 2 of Figure 2.

[0068] In a first step 31, image data representative of a luminous environment outside the vehicle are received from at least one camera of the vehicle and configured for image acquisition of an environment outside the vehicle.

[0069] In a second step 32, illumination data representative of the lighting of a set of virtual graphic objects of the RX content are determined as a function of the image data.

[0070] In a third step 33, the RX content display device is controlled so that the set of virtual graphic objects is displayed in augmented reality according to the illumination data.

[0071] According to one variant, the variants and examples of the operations described in relation to [Fig.1] apply to the steps of the process in [Fig.3].

[0072] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining illumination data that would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

Claims

Demands

1. Method of controlling an augmented reality content display device (11), referred to as RX content, embedded in a vehicle (10), said vehicle (10) comprising at least one camera (101) configured for image acquisition of an environment external to said vehicle (10), said method being implemented by at least one processor and comprising the following steps: - receiving (31) image data representative of a luminous environment external to said vehicle (10) from said at least one camera (101); - determining (32) illumination data representative of the lighting of a set of virtual graphic objects of said RX content as a function of said image data; - controlling (33) said RX content display device (11) to display said set of virtual graphic objects in augmented reality as a function of said illumination data.

2. A method according to claim 1, wherein said determination of illumination data comprises generating an environment map from said image data.

3. A method according to claim 1 or 2, wherein said at least one camera belongs to a camera assembly comprising: - a roof camera (101) arranged on a roof of said vehicle (10); - a front camera arranged in a passenger compartment of said vehicle (10) behind a windshield of said vehicle (10); - a light sensor of said external environment.

4. Method according to claim 3, wherein said roof camera (101) corresponds to a panoramic camera, said image data being representative of a panoramic image of said external lighting environment said vehicle (10) when said at least one camera corresponds to said roof camera.

5. A method according to any one of claims 1 to 4, wherein said image data correspond to high dynamic range image data.

6. A method according to any one of claims 1 to 5, wherein said image data are received by said RX content display device via a wireless connection linking said RX content display device to said vehicle (10).

7. Computer program comprising instructions for carrying out the method according to any one of claims 1 to 6, when such instructions are executed by at least one processor.

8. Device (2) for controlling an augmented reality content display device, referred to as RX content, embedded in a vehicle (10), said device (2) comprising a memory (21) associated with at least one processor (20) configured for carrying out the steps of the method according to any one of claims 1 to 6.

9.

10. Vehicle (10) comprising the device (2) according to claim 8. Vehicle (10) according to claim 9, said vehicle (10) comprising at least one camera (101) and further incorporating an augmented reality content display device (11), said at least one camera (101) being connected in communication to said augmented reality content display device (11).

Images