Method and device for controlling the display of augmented or virtual reality content in a headset worn by a user in a vehicle

Abstract

The present invention relates to a method and device for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display (12) worn by a user in a vehicle (10) and comprising a first inertial measurement unit (IMU). The head-mounted display (12) is wirelessly connected to a mobile communication device (11) comprising a second IMU and installed in the vehicle (10). First inertial data is received from the first IMU, and second inertial data is received from the second IMU. The display of the RX content is controlled based on third data representing the head pose of the user wearing the head-mounted display (12), determined from said first and second inertial data. Figure 1 for the abstract

Description

Title of the invention: Method and device for controlling the display of augmented or virtual reality content in a headset worn by a user in a vehicle. Technical field

[0001] The invention relates to methods and devices for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display worn by a user inside a vehicle, particularly but not exclusively a motor vehicle. The invention also relates to a method and device for determining the head position of the user wearing the head-mounted display in the vehicle. Technological background

[0002] Head-mounted displays, also called viewing headsets or immersive headsets, are experiencing significant growth, particularly in fields such as healthcare, education, entertainment, and industry. These headsets are configured to display virtual reality, augmented reality, and mixed reality content. Their portability leads to mobile use, and they are naturally worn by people traveling in vehicles such as cars, buses, coaches, or trucks. Such headsets are then used, for example, for entertainment such as playing video games, to obtain information about the journey via, for example, the overlay of virtual objects onto the real world and providing diverse and varied information, to consume immersive content, and so on.

[0003] It is known to equip the head-mounted display with devices configured to determine the head position of the person wearing the head-mounted display, including its relative position in a given space and the head orientation. This makes it possible, for example, to determine the direction in which the person is looking and to adapt the content displayed or projected in the head-mounted display to this orientation. Such devices include, for example, an inertial measurement unit comprising gyroscopes and accelerometers. Such devices are known, particularly in the context of head-mounted displays for consuming virtual, augmented, and / or mixed reality content, and are described, for example, in the document entitled "Augmented Reality Outdoors: State of the Art" by Imane Zendjebil et al., published on February 28, 2024, under reference hal-00639331.

[0004] When the person wearing the head-mounted display is in a moving vehicle, the inertial data acquired by the inertial measurement unit represents the movements of the vehicle in addition to the head movements, which can induce errors in the display or projection of virtual objects can diminish the user experience of the person using the headset, and may even cause perceptual disturbances, as the displayed content may not be consistent with head movements. Summary of the present invention

[0005] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0006] An object of the present invention is, for example, to improve the determination of the position of a visor helmet worn by a person traveling in a vehicle.

[0007] According to a first aspect, the present invention relates to a method for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display worn by a user in a vehicle, the head-mounted display comprising a first inertial measurement unit, the vehicle further carrying a mobile communication device comprising a second inertial measurement unit, the head-mounted display being connected in communication to the mobile communication device, the method being implemented by at least one processor and comprising the following steps: - reception of first inertial data from the first inertial measurement unit; - reception of second inertial data from the second inertial measurement unit via the wireless connection; - determination of third data points representative of a pose of a head of the user wearing the visio-helmet from the first and second inertial data points; - control of RX content display based on third-party data.

[0008] Taking into account inertial data received from a mobile communication device embedded in the vehicle, in addition to inertial data obtained from the inertial measurement unit of the head-mounted display (HMD), to determine the pose of the head wearing the HMD allows the vehicle's movements to be considered in the pose calculation. Thus, only head movements are taken into account for determining the pose, which makes it possible to control the display in the HMD based solely on head movements.

[0009] Using the inertial measurement unit (IMU) of the mobile communication device to measure vehicle movements eliminates the need to equip the vehicle with such an IMU, thereby reducing vehicle manufacturing and design costs. Indeed, modern mobile communication devices (e.g., a smartphone) are generally equipped with an IMU.

[0010] According to one variant, the determination of the third data includes a correction of the first inertial data from the second inertial data.

[0011] According to a further variant, the mobile communication device is stationary in a reference frame of the vehicle.

[0012] According to yet another variant, the video headset is connected in communication to the mobile communication device via a wireless connection.

[0013] According to an additional variant, the method further includes a step of receiving fourth data representing the location of the user from a camera of the head-up display, the third data being further determined according to the fourth data.

[0014] According to a further variant, the first and second data are each representative of accelerations along three axes and angular velocities around the three axes.

[0015] According to a second aspect, the present invention relates to a display control device for augmented or virtual reality content, referred to as RX content, in a head-mounted display, the device comprising a memory associated with a processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0016] According to a third aspect, the present invention relates to a head-mounted display comprising a device as described above according to the second aspect of the present invention.

[0017] According to a fourth aspect, the present invention relates to a system comprising a vehicle carrying a head-mounted display as described above according to the third aspect of the present invention and a mobile communication device connected wirelessly to the head-mounted display.

[0018] According to a fifth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0019] Such a computer program may use any programming language, and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0020] According to a sixth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0021] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a storage means, such as a ROM, a CD-ROM, or a type ROM microelectronic circuit, or even a magnetic recording device or a hard drive.

[0022] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0023] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0024] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 3, in which:

[0025] [Fig-1] schematically illustrates a vehicle carrying a user equipped with a visiocasque, according to a particular embodiment of the present invention;

[0026] [Fig.2] illustrates a device configured to control the display of content from augmented or virtual reality, said RX content, in the head-mounted display of [Fig.1], according to a particular and non-limiting embodiment of the present invention.

[0027] [Fig.3] illustrates a flowchart of the different stages of a control process displaying augmented or virtual reality content, referred to as RX content, in the head-mounted display of [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0028] A method and a display control device for augmented or virtual reality content, referred to as RX content, in a head-mounted display in a vehicle will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

[0029] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0030] Fig. 1 schematically illustrates a vehicle 10, according to a particular and non-limiting embodiment of the present invention.

[0031] Vehicle 10 corresponds, for example, to a vehicle with an internal combustion engine, with electric motor(s), or even a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle, for example a car, a truck, a bus.

[0032] According to other examples, the vehicle corresponds to any moving vehicle, for example a bicycle, a boat, an airplane.

[0033] The vehicle 10 carries a mobile communication device 11, that is to say that the mobile communication device 11 is present in the vehicle 10. The mobile communication device 11 corresponds to any device configured for data communication via a wireless connection, such a mobile communication device 11 corresponding for example to a tablet, a smartphone or a smartwatch, such a smartwatch being for example placed on a support provided for this purpose in the vehicle.

[0034] The mobile communication device 11 is, for example, placed stationary in the passenger compartment of the vehicle 10, for example on a support provided for this purpose so that the driver can view the graphic content displayed on the screen of the mobile communication device 11 while driving, on a seat in the vehicle, in the center console, etc.

[0035] The mobile communication device 11 is thus advantageously immobile with respect to the vehicle 10, that is to say that the mobile communication device 11 is immobile in the reference system or frame of reference of the vehicle 10, that is to say in a three-dimensional orthonormal frame associated with the vehicle 10 and having as its origin a point of the vehicle 10 (for example the barycenter of the vehicle 10 or the middle of the front axle of the vehicle 10).

[0036] A person is on board the vehicle 10, this person corresponding for example to a passenger seated in the front passenger seat or in a seat in the rear of the vehicle 10. This person advantageously wears a visor helmet 12 on his head.

[0037] A visiohelmsque 12 corresponds to a system or device comprising a display screen for displaying one or more virtual objects of augmented reality, virtual reality and / or mixed reality content.

[0038] In the rest of the description, the term augmented reality, denoted RX, is used and RX content encompasses all augmented reality, virtual reality and mixed reality content.

[0039] The head-mounted display 12 corresponds to a headset equipped with an integrated display device on which computer-generated synthetic images are displayed or projected under the control of a controller comprising one or more processors. The images The syntheses thus displayed or projected immerse the user wearing the 12-inch head-mounted display or are superimposed on the user's real vision.

[0040] The visiohelmsque 12 encompasses any RX content display device, for example devices such as an immersive headset, a viewing headset, an HMD (from the English "Head-Mounted Display" or "Head-Mounted Display Device" in French), RX glasses (called "see-through glasses" in English, or "transparent glasses" in French) or even a semi-transparent headset (or in English "see-through HMD").

[0041] The 12-headset visor advantageously comprises: - a display screen; - a controller configured to control the display or projection of images onto the display screen; - a first inertial unit (or IMU, from the English "Inertial Measurement Unit"), also called an inertial control unit, comprising 3 gyroscopes or gyrometers configured to measure the 3 components of the angular velocity vector (rates of change of the angles of roll, pitch and yaw) and 3 accelerometers configured to measure the 3 components of the displacement vector, i.e. accelerations along the 3 axes of a three-dimensional frame associated with the visiohelmet 12; - a wireless communication unit comprising one or more wireless communication interfaces for the implementation of wireless connection of type Bluetooth® or Wifi® for example.

[0042] According to one embodiment, the controller corresponds to a data processing device external to the head-mounted display 12 and connected to the latter via a wired or wireless link.

[0043] The mobile communication device 11 advantageously comprises an inertial measurement unit (IMU), referred to as a second IMU, similar to the first IMU and also comprising three gyroscopes configured to measure the three components of the angular velocity vector of the mobile communication device 11 (and consequently of the vehicle 10 when the mobile communication device 11 is stationary in the vehicle 10) and three accelerometers configured to measure the three components of the displacement vector of the mobile communication device 11 (and consequently of the vehicle 10 when the mobile communication device 11 is stationary in the vehicle 10). The mobile communication device 11 also comprises a wireless communication unit comprising one or more wireless communication interfaces for implementing wireless connections such as Bluetooth® or Wi-Fi®.

[0044] According to one embodiment, the first inertial measurement unit and / or the second inertial measurement unit comprise a receiver for a geolocation system by satellite, for example a GPS (Global Positioning System) type system, Galileo or GLONASS.

[0045] The mobile communication device 11 and the video headset 12 are thus configured to communicate data between them via a wireless connection or link, for example of the Bluetooth® type (after an initial pairing operation).

[0046] According to one embodiment, the mobile communication device 11 and the video headset 12 are further configured to communicate with each other via a wired connection or link, for example of the USB type (Universal Serial Bus), CAN type (Controller Area Network) or Ethernet type (according to ISO / IEC 802-3).

[0047] A process for controlling the display of RX content in the head-mounted display 12 is advantageously implemented by one or more processors, for example by one or more processors of the controller of the head-mounted display 12.

[0048] In a first step of the process, the mobile communication device 11 and the video headset 12 connect via a wireless connection, for example of the Bluetooth® type. The connection is, for example, automatic following the detection of one device by the other.

[0049] Such an operation is optional, in particular when the mobile communication device 11 and the video headset 12 are connected via a wired connection.

[0050] In a third operation of the process, initial inertial data are received from the first inertial measurement unit of the head-mounted display 12. This initial inertial data includes, for example: - 3 representative acceleration values ​​along the 3 axes of a three-dimensional coordinate system associated with the visiohelmet 12; and - 3 angular velocity values ​​around the three axes of the three-dimensional frame associated with the visiohelmsque 12.

[0051] These first data are for example received via one or more data buses linking the first inertial measurement unit to the visio-headset controller 12.

[0052] When the controller is external to the head-mounted display 12, this first inertial data is received via the wired or wireless link connecting this controller to the head-mounted display 12.

[0053] In a second operation of the process, second inertial data are received by the visio-helmet controller 12, this second inertial data being transmitted by the mobile communication device 11, which obtained them from the second inertial unit.

[0054] The second inertial data is, for example, received from the second inertial measurement unit in response to a request transmitted by the head-mounted display 12 to the mobile communication device 11 via the wireless connection. According to one embodiment, the second inertial data is transmitted automatically by the mobile communication device 11 to the head-mounted display 12 as soon as the wireless connection is established between the two devices. This second inertial data is transmitted, for example, at regular intervals, for example every 10, 20, 50, or 100 ms.

[0055] When the controller is external to the head-mounted display 12, these second inertial data are received via a wired or wireless link connecting this controller to the mobile communication device 11.

[0056] These second inertial data include, for example: - 3 representative acceleration values ​​along the 3 axes of a three-dimensional frame of reference associated with the mobile communication device 11, which corresponds to the frame of reference associated with the vehicle 10 when the mobile communication device 11 is stationary in the vehicle 10; and - 3 angular velocity values ​​around the three axes of the three-dimensional frame associated with the mobile communication device 11.

[0057] According to a particular embodiment, the marker associated with the head-mounted display 12 is identical to the marker associated with the mobile communication device 11.

[0058] In a fourth operation of the process, third data representing a pose of a head of the user wearing the visor helmet 12 are determined from the first and second inertial data, which makes it possible to determine the pose of the head only without taking into account the movements of the vehicle 10 represented by the second inertial data.

[0059] The determination of the third data includes, for example, a correction of the first inertial data from the second inertial data.For example, each component described in the second inertial data (i.e., the 3 accelerations and the 3 angular velocities) is subtracted from the corresponding component described in the first inertial data: each acceleration measured by the second inertial measurement unit along each of the X, Y, and Z axes is subtracted from the corresponding acceleration measured by the first inertial measurement unit along each of the X, Y, and Z axes (after a possible change of reference frame when the reference frame associated with the first inertial measurement unit is different from that of the second inertial measurement unit), and each angular velocity measured by the second inertial measurement unit around each of the X, Y, and Z axes is subtracted from the corresponding angular velocity measured by the first inertial measurement unit around each of the X, Y, and Z axes (after a possible change of reference frame when the . (the reference frame associated with the first inertial measurement unit is different from that of the second inertial measurement unit).

[0060] According to a particular embodiment, fourth data points representing the location of the user wearing the head-mounted display 12 are received from a camera on the head-mounted display 12, the third data points then being further determined based on the fourth data points. Such a camera corresponds to a camera arranged in the head-mounted display and configured to acquire images of the external environment of the head-mounted display 12 according to the position of the user's head. According to this embodiment, these fourth data points are merged with the first inertial data points once the latter have been corrected using the second inertial data points.

[0061] Such fusion is implemented according to any method known to a person skilled in the art, such methods being described in the document entitled "Outdoor Augmented Reality, State of the Art" by Imane Zendjebil et al., published on February 28, 2024 under reference hal-00639331.

[0062] In a fifth operation of the process, the controller controls the display of the RX content according to the third data determined in the fourth operation, the displayed content thus being a function of the position of the head of the user wearing the visor helmet 12 excluding the movements of the vehicle 10.

[0063] Display control of RX content or any virtual object included in such RX content comprises rendering the RX content, such rendering corresponding to a set of operations performed by one or more processors on the pixels of one or more images of the graphic content to be displayed or projected onto the screen(s) of the head-mounted display 12. For example, rendering consists of associating pixel data (for example, color data expressed in an RGB (Red, Green, Blue) color space) with each graphic object. Display control of the graphic content thus includes the transmission of control signals to the screen to modify the values ​​associated with the screen pixels at the location intended to display the indicator (or any graphic object).

[0064] Displaying a set of virtual objects in augmented reality amounts, for example, to displaying one or more virtual graphic objects (for example, computer-generated) superimposed on the real world visible to the user wearing the visor head 12, the real world being directly visible to the user when the visor head 12 is of the transparent (or "see-through") type or the real world being visible via images of this real environment acquired by a camera of the visor head 12.

[0065] The displayed RX content is adapted to the head pose information obtained from the third data, the displayed RX content corresponding in particular to the part of immersive content or a set of virtual objects selected according to the pose, for example the orientation of the head indicating the direction of the user's gaze.

[0066] Figure 2 schematically illustrates a device 2 configured to control the display of augmented, mixed and / or virtual reality content in a head-mounted display such as the head-mounted display 12. Device 2 corresponds for example to a device embedded in the head-mounted display, for example a controller.

[0067] According to a particular embodiment, device 2 corresponds to a mobile communication device such as device 11.

[0068] Device 2 is, for example, configured to carry out the operations described opposite [Fig. 1] and / or the steps of the process described opposite [Fig. 3]. Examples of such a device 2 include, but are not limited to, embedded electronic equipment such as a controller, a rendering engine, a smartphone, a tablet, or a laptop computer. The elements of device 2, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 2 can be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0069] The device 2 comprises one (or more) processor(s) 20 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 2. The processor 20 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 2 further comprises at least one memory 21, corresponding, for example, to volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0070] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 21.

[0071] According to a particular and non-limiting embodiment, the device 2 comprises a block 22 of interface elements for communicating with external devices. The interface elements of the block 22 comprise one or more of the following interfaces: - Radio frequency (RF) interface, for example, Wi-Fi® (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox® using UBN (Ultra Narrow Band) radio technology narrow), or LoRa® in the 868 MHz frequency band, LTE (from the English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-avancé); - at least one wired interface of type USB (Universal Serial Bus), CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (according to ISO 17458) or Ethernet (according to ISO / IEC 802-3); - HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French).

[0072] According to another particular and non-limiting embodiment, the device 2 includes a communication interface 23 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 230. The communication interface 23 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 230. The communication interface 23 corresponds, for example, to a wired Ethernet network (standardized by ISO / IEC 802-3).

[0073] According to a particular and non-limiting embodiment, the device 2 can provide output signals to one or more external devices, such as a display screen 240, one or more speakers 250 and / or other peripherals 260 (projection system) via output interfaces 24, 25 and 26 respectively. According to a variant, one or more of the external devices is integrated into the device 2.

[0074] Figure 3 illustrates a flowchart of the different stages of a method for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display worn by a user in a vehicle. The head-mounted display includes a first inertial measurement unit (IMU), and the vehicle also carries a mobile communication device comprising a second IMU. The head-mounted display is connected to the mobile communication device via a wireless connection, according to a particular and non-limiting embodiment of the present invention. The method is implemented, for example, by a device integrated into the head-mounted display 12 or by the device 2 of Figure 2.

[0075] In a first step 31, first inertial data are received from the first inertial unit.

[0076] In a second step 32, second inertial data are received from the second inertial unit via the wireless connection.

[0077] In a third step 33, third data representing a pose of a head of the user wearing the visio-helmet are determined from the first and second inertial data.

[0078] In a fourth step 34, the display of the RX content is controlled according to the third data.

[0079] According to one variant, the variants and examples of the operations described in relation to [Fig.1] apply to the steps of the process in [Fig.3].

[0080] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining head position information for a person wearing the head-mounted display, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0081] The present invention also relates to a system comprising the vehicle 10 carrying the mobile communication device 11 and the video headset 12, the mobile communication device 11 being connected wirelessly to the video headset 12.

Claims

Demands

1. A method for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display (12) worn by a user in a vehicle (10), said head-mounted display (12) comprising a first inertial measurement unit (IMU), said vehicle (10) further carrying a mobile communication device (11) comprising a second IMU, said head-mounted display (12) being connected in communication to said mobile communication device (11), said method being implemented by at least one processor and comprising the following steps: - receiving (31) first inertial data from the first IMU; - receiving (32) second inertial data from the second IMU via said wireless connection; - determining (33) third data representing a pose of the head of said user wearing said head-mounted display (12) from said first and second inertial data;- control (34) of displaying the RX content according to said third data.;

2. Method according to claim 1, wherein said determination of the third data includes a correction of said first inertial data from said second inertial data.

3. Method according to claim 1 or 2, wherein said mobile communication device (11) is stationary in a reference frame of said vehicle (10).

4. A method according to any one of claims 1 to 3, wherein said head-mounted display (12) is connected in communication to said mobile communication device (11) via a wireless connection.

5. A method according to any one of claims 1 to 4, further comprising a step of receiving fourth data representing the location of said user from a camera of said head-mounted display (12), the third data being further determined as a function of the fourth data.

6. A method according to any one of claims 1 to 5, wherein said first data and said second data are each representative of accelerations along three axes and angular velocities around the three axes.

7. Computer program comprising instructions for carrying out the method according to any one of claims 1 to 6, when such instructions are executed by at least one processor.

8. Device (2) for controlling the display of augmented or virtual reality content, referred to as RX content, in a head-mounted display, said device (2) comprising a memory (21) associated with at least one processor (20) configured for carrying out the steps of the method according to any one of claims 1 to 6.

9. A head-mounted display (12) comprising the device (2) according to claim e

10. O. System comprising a vehicle (10) carrying the head-mounted display (12) according to claim 9 and a mobile communication device (11) connected wirelessly to said head-mounted display (12).

Images