In-vehicle immersive-reality system
Patent Information
- Application Number
- EP2023828712
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-12-23
- Filing Date
- 2023-12-12
- Publication Date
- 2025-10-29
AI Technical Summary
Existing immersive reality systems in vehicles lack comprehensive hand and head movement detection, leading to blind spots and requiring multiple sensors per head-mounted display, which limits interaction and immersion in virtual environments.
A detection device with sensors placed at fixed locations in the passenger compartment, including three-dimensional and two-dimensional sensors, to detect the movement, orientation, and position of passengers' hands and heads, allowing 360° detection and interaction without the need for controllers or sensors integrated into head-mounted displays.
Enables precise and extensive detection of passengers' hands for managing reality interactions, allowing immersive and interactive virtual or augmented reality experiences, including 360° head movement and hand gestures, while avoiding blind spots and reducing sensor redundancy.
Smart Images
Figure 1.1
Abstract
Description
Description Title of the invention: Immersive reality system on board a vehicle [1] The present invention relates in particular to an immersive reality system, for example virtual reality, intended to be installed on board a vehicle. [2] The vehicle can be land, sea or air. [3] Patent application US2015378156 describes a virtual reality system in a real vehicle. With this system, a vehicle salesperson can travel with a test vehicle to a potential customer's home and the system can virtually display a vehicle configuration required by the customer. The potential customer can thus sit in the vehicle, experience the relevant materials of the test vehicle, specify an interior and customized equipment, and touch elements of the real vehicle, for example the steering wheel and controls, while seeing the chosen virtual configuration, thanks to virtual reality glasses worn by the customer. [4] Glasses are known from patent application DE102016225262 which can display an image on display surfaces in the field of vision of the wearer of the glasses. The display surfaces of the glasses can be semi-transparent, so that the actual environment can be perceived in the usual way through the glasses and information such as text, symbols, graphics or video displays can be displayed by overlapping the perception of the environment. In order to have the spatial position of the glasses data in the coordinate system of the vehicle and their three-dimensional orientation, a control unit is provided which is connected to a camera attached to a temple of the glasses, and directed approximately in the direction of vision of the wearer of the glasses. The control unit evaluates the camera images and, using pattern recognition methods, allows to locate distinctive structures of the vehicle interior in the camera image, the positions of which are known, and to determine the three-dimensional orientation of the glasses. The spatial positions and / or movements of both hands of the wearer of the data glasses are preferably determined when they are in the detection field of the glasses camera. [5] Patent application US2016070966 describes a vehicle system that uses a driver's head pose as a metric for estimating driver attention. The system may be configured to measure the driver's head pose with non-intrusive or intrusive methods. Non-intrusive measurement techniques may include, for example, interior cameras and image processing performed based on captured images of the driver's head. Intrusive methods may include, for example, head-mounted devices configured to more directly measure the orientation or location of the driver's head.Intrusive methods of measuring driver gaze may include glasses worn by the driver, which include motion sensors (e.g., accelerometer, gyroscope) that the vehicle system may use to provide information to estimate the driver's head position and activity. [6] The invention aims in particular to propose a new immersive reality system, for example virtual reality, intended to be installed on board a vehicle. [7] The invention thus relates to an immersive reality system, for example virtual reality, intended to be installed on board a vehicle, this system comprising: - a detection device configured to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger, preferably of a plurality of passengers on board the vehicle, this detection device comprising at least one sensor installed at a fixed location in a passenger compartment of the vehicle, - an immersive reality image display device configured to be worn on a passenger's head, this display device being in particular a head-mounted display (HMD) or glasses configured to display images and allow the passenger to see through them, - a data processing unit configured to provide immersive reality images to the display device, based on data obtained by the detection device, in particular to enable interaction of the hands with an object in an immersive reality scene displayed by the display device. [8] The term "passenger" can refer to the driver himself or to a person sitting next to him, or in the back. [9] According to one aspect of the invention, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger seated in any seat in the passenger compartment. For example, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger seated in a seat at the front or at the rear of the vehicle.
[0010] According to one aspect of the invention, the detection device comprises several sensors fixed in the passenger compartment, at several locations in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger anywhere in the passenger compartment. These sensors may be identical or, alternatively, be different, combining for example a three-dimensional type sensor and a two-dimensional type sensor. The system may comprise, where appropriate, several three-dimensional sensors to cover the entire passenger compartment by offering several points of view which complement each other. Similarly, the system may comprise, where appropriate, several two-dimensional sensors to cover the entire passenger compartment by offering several points of view which complement each other.
[0011] According to one aspect of the invention, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of several passengers at the same time, installed in the passenger compartment, whether on the seats at the front or the seats at the rear.
[0012] In the invention, the detection device and the display device are mechanically disjointed from each other, namely the sensor of the detection device is not mechanically linked to the display device, unlike certain virtual reality devices in which the head-mounted display incorporates sensors to determine the position, orientation and movement of the wearer of the head-mounted display. In the invention, the sensor(s) of the detection device are placed at fixed locations in the passenger compartment, so that these sensors are capable of detecting the position, orientation and movement of body parts (head and hands in particular) of passengers in the passenger compartment, avoiding having to manage blind spots which remain when the sensors are integrated into the head-mounted display.In a way, the sensor(s) placed, according to the invention, at suitable locations in the passenger compartment, allow 360° detection in the passenger compartment, where sensors integrated into traditional head-mounted displays do not, in general, allow detection in an area behind the head-mounted display.
[0013] The invention also makes it possible to do without controllers to interact with objects in the virtual environment, thanks to sensors placed in the passenger compartment around the passenger(s), which directly detect hand gestures.
[0014] The sensors mounted in the passenger compartment can be used for multiple passengers at once. Conversely, if the sensor(s) were mounted on the head-mounted display, as many groups of sensors would be required as there are headsets to be tracked. The invention thus makes it possible to share sensors for multiple head-mounted displays or glasses.
[0015] Advantageously, the immersive reality image display device is devoid of any motion and / or orientation and / or head or hand position detection sensors. All these sensors are placed in fixed locations in the passenger compartment.
[0016] According to one aspect of the invention, the sensor of the detection device is configured to output a three-dimensional image of a detection area. This is preferred for detecting the position, orientation and hand movement of the passenger.
[0017] According to one aspect of the invention, the sensor of the detection device is chosen from: - a time-of-flight sensor, such as a time-of-flight camera, ("Time of Flight" or TOF according to the English acronym) adapted to send light towards the passenger and to measure the time taken by this light to return to said time-of-flight sensor to deduce the three-dimensional image in a detection zone, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to derive a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection zone and to analyze the deformation of this pattern to deduce the three-dimensional image of the driver.
[0018] According to another aspect of the invention, the sensor of the detection device is configured to output a two-dimensional image of a detection area. This is suitable for detecting the position, orientation and movement of the passenger's head.
[0019] According to one aspect of the invention, the sensor of the detection device is an RGB camera, or an infrared camera, in particular of the near infrared type (or NIR according to the English acronym).
[0020] According to one aspect of the invention, the detection device comprises a plurality of sensors chosen from the sensors described above, in particular among which are at least one two-dimensional detection sensor and one three-dimensional detection sensor. The two-dimensional detection sensor may advantageously be used to locally increase the resolution of the three-dimensional detection sensor.
[0021] According to one aspect of the invention, the sensor is positioned on a ceiling of the vehicle, in particular in a front ceiling light of the vehicle.
[0022] Alternatively, the sensor is arranged in a lower part of the vehicle's windshield, in particular substantially in front of the driver.
[0023] According to one aspect of the invention, the sensor(s) are placed on a dashboard of the vehicle, in particular in a central area of this dashboard. It is also possible to have the sensor(s) placed on a rearview mirror of the vehicle, notably integrated into the central rearview mirror inside the passenger compartment.
[0024] According to one aspect of the invention, the immersive reality system is configured to allow the passenger wearing the immersive reality display device to interact with an object of the virtual environment or the real environment, in particular with a hand, for example by touching it with a hand or grasping it with a hand.
[0025] The invention thus makes it possible, thanks to the sensor(s) mounted in the passenger compartment, to detect the hands of the passenger(s) precisely and extensively (for example at 360°) for the purposes of managing virtual reality interactions.
[0026] According to one aspect of the invention, the immersive reality system is configured to generate a fully virtual environment which is perceptible in the virtual reality headset, which is totally occulting, namely the real environment is totally occulted.
[0027] In this system according to the invention, it is possible, while being totally immersed in the virtual environment, to turn the head 360°, move in space and interact with the virtual environment using the hands whose movements are detected by the sensor(s) of the detection device.
[0028] According to another aspect of the invention, the immersive reality system is configured to generate an augmented reality environment in which three-dimensional or two-dimensional virtual elements are superimposed on a real environment.
[0029] According to another aspect of the invention, the immersive reality system is configured to generate an augmented reality environment allowing, in addition, interaction between a virtual environment and a real environment. This type of system is also called mixed reality.
[0030] In other words, in a mixed reality application, three-dimensional or two-dimensional virtual elements are superimposed on a real environment (augmented reality) and it is possible to interact with the virtual elements, in particular by hand gestures, as in virtual reality.
[0031] According to one aspect of the invention, the immersive reality system is configured to operate an entertainment or relaxation session for the passenger wearing the display device, which passenger can interact, in particular by his hands, with a virtual entertainment or relaxation environment.
[0032] According to another aspect of the invention, the immersive reality system can be used for driving training purposes or for commercial promotion of a vehicle.
[0033] In the present invention, a head-mounted display is a display device, worn on the head or in a helmet of the passenger, comprising a display screen in front of one eye (monocular head-mounted display), or two display screens in front of each eye (binocular head-mounted display).
[0034] Display screens are miniaturized and can, for example, be liquid crystal displays (LCDs), liquid crystal on silicon (LCoS) displays or organic light-emitting diode (OLED) displays.
[0035] According to one aspect of the invention, the data processing unit is wiredly connected to the display device.
[0036] Alternatively, the data processing unit is connected wirelessly to the display device, in particular by a wireless communication protocol of the Bluetooth or WIFI type.
[0037] According to one aspect of the invention, the data processing unit is part of an on-board computer of the vehicle.
[0038] According to another aspect of the invention, the data processing unit is integrated into the display device. Thus, for example, the head-mounted display may include the data processing unit which communicates with the sensors and performs the calculations to determine the interactions between, for example, the passenger's hands and the immersive reality scene.
[0039] The invention also relates to a method for managing an immersive reality system, for example virtual reality, on board a vehicle, this method comprising the following steps: - detecting a movement and / or an orientation and / or a position of the hands and / or the head of a passenger, preferably of a plurality of passengers on board the vehicle, using a detection device of the immersive reality system, comprising at least one sensor installed at a fixed location in a passenger compartment of the vehicle, - displaying immersive reality images in a display device of the system, worn on the head of a passenger, based on data obtained by the detection device, in particular to enable interaction of the hands with an object in an immersive reality scene displayed by the display device.
[0040] According to one aspect of the invention, the immersive reality scene is a virtual reproduction of the vehicle interior. Thus, the passenger can be immersed in the vehicle environment, with additional virtual objects in this virtual scene, for example dedicated to entertainment (for example a virtual pencil for drawing) or to a relaxation session (for example a virtual ball that the passenger can grasp).
[0041] The virtual reality scene may contain an image of another passenger. This allows two or more people in the passenger compartment to interact in the virtual reality world.
[0042] Sensors installed in the passenger compartment can detect the movements of all passengers seated in the vehicle.
[0043] The invention is particularly well-suited to a self-driving vehicle. This allows passengers, including the driver, to benefit from the immersive reality system without having to concentrate on driving the vehicle.
[0044] Other characteristics and advantages of the invention will appear more clearly on reading the following description, given as an illustrative and non-limiting example, and the appended drawing in which:
[0045] - [Figure 1] illustrates, schematically and partially, an immersive reality system intended to be installed on board a vehicle, according to an exemplary implementation of the invention.
[0046] Figure 1 shows an immersive reality system 1, here virtual reality, intended to be carried on board a vehicle V, this system 1 comprising: - a detection device 2 configured to detect a movement and / or an orientation and / or a position of the hands 3 and / or the head 4 of a plurality of passengers 5 on board the vehicle V, this detection device 2 comprising sensors 7 and 8 installed at fixed locations 9 and 10 respectively in a passenger compartment H of the vehicle V, - an immersive reality image display device 15 configured to be worn on the head of a passenger 5, this display device here being a head-mounted display (HMD), - a data processing unit 16 configured to provide immersive reality images to the display device 15, based on data obtained by the detection device 2, to enable interaction of the hands 3 with an object in an immersive reality scene displayed by the display device 15.
[0047] The term "passenger" can refer to the driver himself or to a person sitting next to him, or in the back.
[0048] The detection device 2 is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger 5 installed in a place at the front or at the rear of the vehicle.
[0049] The sensors 7 and 8 fixed in the passenger compartment are at several locations 9 and 10 respectively, in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger anywhere in the passenger compartment.
[0050] The sensor 7 is positioned on a location 9 of a ceiling 18 of the vehicle, in particular in a front ceiling light of the vehicle or a dome.
[0051] The sensor 8 is arranged in a lower part 10 of the windshield 19 of the vehicle, in particular substantially in front of the driver. Alternatively, a camera-type sensor 8 can be integrated into the central rearview mirror.
[0052] The detection device 2 is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of several passengers 5 at a time, installed in the passenger compartment, whether on the seats at the front or the seats at the rear.
[0053] The detection device 2 and the display device 15 are mechanically disjointed from each other, namely the sensor of the detection device is not mechanically linked to the display device, unlike certain virtual reality devices in which the head-mounted display incorporates sensors to determine the position, orientation and movement of the wearer of the head-mounted display.
[0054] The immersive reality image display device 15 does not have a sensor for detecting movement and / or orientation and / or position of the head 4 or hands 3.
[0055] The sensor 7 of the detection device 2 is configured to deliver a three-dimensional image of a detection zone 21 which is the entire passenger compartment H.
[0056] Sensor 7 is chosen from: - a time-of-flight sensor, such as a time-of-flight camera, ("Time of Flight" or TOF according to the English acronym) adapted to send light towards the passenger and to measure the time taken by this light to return to said time-of-flight sensor to deduce the three-dimensional image in a detection zone, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to derive a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection zone and to analyze the deformation of this pattern to deduce the three-dimensional image of the driver.
[0057] The sensor 8 is configured to deliver a two-dimensional image of the detection area 21.
[0058] Sensor 8 is an RGB camera, or an infrared camera, particularly of the near infrared type (or NIR according to the Anglo-Saxon acronym).
[0059] The immersive reality system 1 is configured to allow the passenger 6 carrying the immersive reality display device 16, to interact with an object of the virtual environment or the real environment, in particular with a hand, for example by touching it with a hand or by grasping it with a hand.
[0060] The immersive reality system 1 is configured to generate a fully virtual environment which is perceptible in the virtual reality headset 16, which is totally occulting, namely the real environment is totally occulted.
[0061] In this system according to the invention, it is possible, while being totally immersed in the virtual environment, to turn the head 360°, move in space and interact with the virtual environment using the hands whose movements are detected by the sensor(s) of the detection device.
[0062] Alternatively, the immersive reality system could be configured to generate an augmented reality or mixed reality environment in which three-dimensional or two-dimensional virtual elements are superimposed on a real environment.
[0063] Depending on the needs, the immersive reality system is configured to generate an augmented reality or mixed reality environment, also allowing interaction between a virtual environment and a real environment.
[0064] The immersive reality system 1 is configured to operate an entertainment or relaxation session for the passenger 5 who is wearing the display device 16, which passenger 5 can interact, by his hands, with a virtual entertainment or relaxation environment.
[0065] The immersive reality system 1 could be used for driving training purposes or for commercial vehicle promotions.
[0066] The head-mounted display 15 is a display device, worn on the head 5 or in a passenger's helmet, comprising a display screen in front of one eye (monocular head-mounted display), or two display screens in front of each eye. (binocular head-mounted display). The head-mounted display can be completely opaque or pass-through, allowing you to see through it, depending on your needs.
[0067] Display screens are miniaturized and can, for example, be liquid crystal displays (LCDs), liquid crystal on silicon (LCoS) displays or organic light-emitting diode (OLED) displays.
[0068] The data processing unit 16 is wiredly connected to the display device 15.
[0069] Alternatively, the data processing unit 16 is connected wirelessly to the display device 15, in particular by a wireless communication protocol of the Bluetooth or WIFI type.
[0070] For example, the data processing unit 16 is part of an on-board computer of the vehicle.
[0071] It is also possible that the data processing unit 16 is integrated into the display device 15. Thus, for example, the head-mounted display 15 may include the data processing unit 16 which communicates with the sensors 7 and 8 and performs the calculations to determine the interactions between, for example, the hands 4 of the passenger and the immersive reality scene.
[0072] In the example described, the immersive reality scene is a virtual reproduction of the passenger compartment H of the vehicle. Thus, the passenger 6 can be immersed in the environment of the vehicle, with additional virtual objects in this virtual scene, for example dedicated to entertainment (for example a virtual pencil for drawing) or to a relaxation session (for example a virtual ball that the passenger can grasp).
[0073] The virtual reality scene may contain an image of another passenger. This allows two or more people in the passenger compartment to interact in the virtual reality world.
[0074] Sensors 7 and 8 installed in the passenger compartment H allow the movements of all passengers seated in the vehicle to be detected.
[0075] The invention is particularly well suited to an autonomous driving vehicle. This allows passengers, including the driver, to benefit from the immersive reality system, without having to concentrate on driving the vehicle.
Claims
Claims
1. Immersive reality system (1), for example virtual reality, intended to be carried on board a vehicle, this system comprising: - a detection device (2) configured to detect a movement and / or an orientation and / or a position of the hands (3) and / or the head (4) of a passenger (5), preferably of a plurality of passengers on board the vehicle, this detection device comprising at least one sensor (7; 8) installed at a fixed location (9; 10) in a passenger compartment of the vehicle, - an immersive reality image display device (15) configured to be worn on the head of a passenger, this display device being in particular a head-mounted display or glasses configured to display images and allow the passenger to see through it, - a data processing unit (16) configured to provide immersive reality images to the display device (15), based on data obtained by the detection device (2), in particular to enable interaction of the hands with an object in an immersive reality scene displayed by the display device.
2. Immersive reality system (1) according to the preceding claim, wherein the detection device (2) comprises several sensors (7, 8) fixed in the passenger compartment, at several locations in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger anywhere in the passenger compartment.
3. Immersive reality system (1) according to one of the preceding claims, wherein the detection device (2) is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a plurality of passengers at a time, installed in the passenger compartment, whether on the seats at the front or the seats at the rear.
4. Immersive reality system (1) according to one of the preceding claims, in which the immersive reality image display device (15) is devoid of a sensor for detecting movement and / or orientation and / or position of the head or hands.
5. Immersive reality system (1) according to one of the preceding claims, wherein the sensor (7) of the detection device is configured to deliver a three-dimensional image of a detection zone.
6. Immersive reality system (1) according to the preceding claim, in which the sensor (7) of the detection device is chosen from: - a time-of-flight sensor, such as a time-of-flight camera, ("Time of Flight" or TOF according to the English acronym) adapted to send light towards the passenger and to measure the time taken by this light to return to said time-of-flight sensor to deduce the three-dimensional image in a detection zone, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to derive a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection zone and to analyze the deformation of this pattern to deduce the three-dimensional image of the conductor 4.
7. Immersive reality system (1) according to one of claims 1 to 4, wherein the sensor (8) of the detection device is configured to deliver a two-dimensional image of a detection zone.
8. Immersive reality system (1) according to one of the preceding claims, wherein the sensor (8) is positioned on a ceiling (18) of the vehicle, in particular in a front ceiling light of the vehicle.
9. Immersive reality system (1) according to one of claims 1 to 7, wherein the sensor (8) is arranged in a part lower part of the windshield (19) of the vehicle, in particular substantially in front of the driver.
10. Immersive reality system (1) according to one of the preceding claims, wherein the immersive reality system (1) is configured to allow the passenger wearing the immersive reality display device to interact with an object of the virtual environment or the real environment, in particular with a hand, for example by touching it with a hand or grasping it with a hand.
11. Immersive reality system (1) according to one of the preceding claims, wherein the immersive reality system (1) is configured to generate a fully virtual environment which is perceptible in the virtual reality headset, which is totally occulting, namely the real environment is totally occulted.
12. Immersive reality system (1) according to one of claims 1 to 10, wherein the immersive reality system (1) is configured to generate an augmented reality environment in which three-dimensional or two-dimensional virtual elements are superimposed on a real environment.
13. Immersive reality system (1) according to the preceding claim, wherein the immersive reality system (1) is configured to generate an augmented reality environment allowing, in addition, interaction between a virtual environment and a real environment.
14. Immersive reality system (1) according to one of the preceding claims, wherein the immersive reality system (1) is configured to operate an entertainment or relaxation session for the passenger who is wearing the display device (15), which passenger can interact, in particular by his hands, with a virtual entertainment or relaxation environment.
15. Method for managing an immersive reality system (1), for example virtual reality, on board a vehicle, this method comprising the following steps: - detecting a movement and / or an orientation and / or a position of the hands (3) and / or the head (4) of a passenger, preferably of a plurality of passengers on board the vehicle, using a detection device (2) of the immersive reality system, comprising at least one sensor installed in a fixed location in a passenger compartment of the vehicle, - displaying immersive reality images in a display device (15) of the system, worn on the head of a passenger, based on data obtained by the detection device, in particular to allow interaction of the hands with an object in an immersive reality scene displayed by the display device.