Immersive reality system embedded in a vehicle
A vehicle-based immersive reality system with fixed compartment sensors addresses blind spots in traditional headsets, allowing comprehensive passenger interaction and enhanced immersion, particularly beneficial for autonomous driving.
Patent Information
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- VALEO COMFORT & DRIVING ASSISTANCE
- Filing Date
- 2022-12-23
- Publication Date
- 2026-06-19
AI Technical Summary
Existing immersive reality systems in vehicles often suffer from blind spots in sensor detection due to integrated sensors within headsets, limiting comprehensive passenger interaction and immersion, especially when multiple passengers are involved.
A detection system with sensors placed at fixed locations within the vehicle passenger compartment, such as the ceiling, windshield, and dashboard, provides 360-degree detection of passengers' hands and head movements, enabling precise interaction without the need for sensors on head-mounted displays.
Enables comprehensive 360-degree detection and interaction with virtual environments, supporting multiple passengers and enhancing immersive experiences without blind spots, suitable for autonomous driving scenarios.
Smart Images
Figure 00000014_0000
Abstract
Description
Title of the invention: Immersive reality system embedded in a vehicle
[0001] The present invention relates in particular to an immersive reality system, for example virtual reality, intended to be embedded in a vehicle.
[0002] The vehicle may be of land, sea or air type.
[0003] US patent application 2015378156 describes a virtual reality system within a real vehicle. With this system, a vehicle salesperson can drive a test vehicle to a potential customer's location, and the system can virtually display a vehicle configuration requested by the customer. The potential customer can then sit in the vehicle, explore the relevant materials of the test vehicle, specify a personalized interior and equipment, and touch elements of the real vehicle, such as the steering wheel and controls, all while viewing the chosen virtual configuration through virtual reality glasses worn by the customer.
[0004] Patent application DE102016225262 discloses glasses that can display an image on display surfaces within the wearer's field of vision. The display surfaces of the glasses can be semi-transparent, so that the real environment can be perceived in the usual way through the glasses, and information such as text, symbols, graphics, or video displays can be shown overlaid on the perception of the environment. To obtain the spatial position of the glasses in the vehicle's coordinate system and their three-dimensional orientation, a control unit is provided, connected to a camera mounted on one temple of the glasses and directed approximately in the wearer's line of sight.The control unit evaluates the camera images and, using pattern recognition techniques, locates distinctive structures inside the vehicle within the camera image, whose positions are known, and determines the three-dimensional orientation of the glasses. The spatial positions and / or movements of both hands of the wearer of the data glasses are preferably determined when they are within the detection field of the glasses' camera.
[0005] US patent application 2016070966 describes a vehicle system that uses a driver's head posture as a measure for estimating driver attention. The system can be configured to measure the driver's head posture using non-intrusive or intrusive methods. Non-intrusive measurement techniques may include, for example, interior cameras and image processing performed on captured images of the driver's head. The methods Intrusive methods can include, for example, head-mounted devices configured to more directly measure the orientation or location of the driver's head. Intrusive methods of measuring driver gaze can include glasses worn by the driver that incorporate motion sensors (e.g., an accelerometer, a gyroscope) that the vehicle's system can use to provide information for estimating head position and driver activity.
[0006] The invention aims in particular to propose a new immersive reality system, for example virtual reality, intended to be embedded in a vehicle.
[0007] The invention thus relates to an immersive reality system, for example virtual reality, intended to be installed on board a vehicle, this system comprising: - a detection device configured to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger, preferably of a plurality of passengers on board the vehicle, this detection device comprising at least one sensor installed in a fixed location in a passenger compartment of the vehicle, - an immersive reality image display device configured to be worn on a passenger's head, this display device being in particular a head-mounted display (HMD) or glasses configured to display images and allow the passenger to see through them, - a data processing unit configured to provide immersive reality images to the display device, based on data obtained by the detection device, in particular to enable hand interaction with an object in an immersive reality scene displayed by the display device.
[0008] The term “passenger” may refer to the driver himself or a person sitting next to him, or in the back.
[0009] According to one aspect of the invention, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or head of a passenger seated in any position in the passenger compartment. For example, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or head of a passenger seated in a seat in the front or rear of the vehicle.
[0010] According to one aspect of the invention, the detection device comprises several sensors fixed in the passenger compartment, at several locations within the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or head of a passenger anywhere in the passenger compartment. These sensors may be The sensors may be identical or, alternatively, different, for example, combining a three-dimensional sensor and a two-dimensional sensor. The system may, where appropriate, include several three-dimensional sensors to cover the entire passenger compartment, providing multiple complementary viewpoints. Similarly, the system may, where appropriate, include several two-dimensional sensors to cover the entire passenger compartment, providing multiple complementary viewpoints.
[0011] According to one aspect of the invention, the detection device is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of several passengers at the same time, installed in the passenger compartment, whether in the front seats or the rear seats.
[0012] In the invention, the detection device and the display device are mechanically separate from each other; namely, the sensor of the detection device is not mechanically linked to the display device, unlike certain virtual reality devices in which the headset incorporates sensors to determine the position, orientation, and movement of the headset wearer. In the invention, the sensor(s) of the detection device are placed in fixed locations within the passenger compartment, so that these sensors are capable of detecting the position, orientation, and movement of body parts (particularly the head and hands) of passengers within the passenger compartment, thus avoiding the need to manage blind spots that remain when the sensors are integrated into the headset.In a way, the sensor(s) placed, according to the invention, in suitable locations in the passenger compartment, allow 360° detection within the passenger compartment, whereas sensors integrated into traditional helmet-mounted displays generally do not allow detection in an area behind the helmet.
[0013] The invention also makes it possible to do without controllers to interact with objects in the virtual environment, thanks to sensors placed in the passenger compartment around the passenger(s), which directly detect hand gestures.
[0014] The sensors mounted in the passenger compartment can be used for several passengers at once. Conversely, if the sensor(s) were mounted on the helmet-mounted display, as many groups of sensors would be needed as there are helmets to track. The invention thus makes it possible to share the sensors for several helmet-mounted displays or goggles.
[0015] Advantageously, the immersive reality image display device is devoid of motion and / or orientation and / or head or hand position detection sensors. All these sensors are placed in fixed locations within the passenger compartment.
[0016] According to one aspect of the invention, the sensor of the detection device is configured to deliver a three-dimensional image of a detection area. This is preferred for detecting the position, orientation, and hand movements of the passenger.
[0017] According to one aspect of the invention, the sensor of the detection device is chosen from: - a time-of-flight sensor, such as a time-of-flight camera (TOF), adapted to send light towards the passenger and measure the time it takes for this light to return to said time-of-flight sensor in order to deduce the three-dimensional image in a detection area, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to deduce a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection area and to analyze the deformation of this pattern to deduce the three-dimensional image of the conductor.
[0018] According to another aspect of the invention, the sensor of the detection device is configured to deliver a two-dimensional image of a detection area. This is suitable for detecting the position, orientation, and movement of the passenger's head.
[0019] According to one aspect of the invention, the sensor of the detection device is an RGB camera, or an infrared camera, in particular of the near-infrared type (or NIR according to the Anglo-Saxon acronym).
[0020] According to one aspect of the invention, the detection device comprises a plurality of sensors selected from among those described above, including at least one two-dimensional detection sensor and one three-dimensional detection sensor. The two-dimensional detection sensor may advantageously be used to locally increase the resolution of the three-dimensional detection sensor.
[0021] According to one aspect of the invention, the sensor is positioned on a ceiling of the vehicle, in particular in a front ceiling light of the vehicle.
[0022] Alternatively, the sensor is located in a lower part of the vehicle's windshield, in particular substantially in front of the driver.
[0023] According to one aspect of the invention, the sensor(s) are placed on the vehicle's dashboard, particularly in a central area of that dashboard. It is also possible to have the sensor(s) placed on a vehicle's rearview mirror, in particular integrated into the central rearview mirror inside the passenger compartment.
[0024] According to one aspect of the invention, the immersive reality system is configured to allow the passenger wearing the immersive reality display device to interact with an object in the virtual or real environment, in particular with a hand, for example by touching it with a hand or by grasping with one hand.
[0025] The invention thus makes it possible, thanks to the sensor(s) mounted in the passenger compartment, to detect in a precise and extensive manner (for example at 360°) the hands of the passenger(s) for the purposes of managing virtual reality interactions.
[0026] According to one aspect of the invention, the immersive reality system is configured to generate a fully virtual environment which is perceptible in the virtual reality vis-headset, which is totally occulting, namely the real environment is totally occulted.
[0027] In this system according to the invention, it is possible, while being totally immersed in the virtual environment, to turn the head 360°, move in space and interact with the virtual environment by means of the hands whose movements are detected by the sensor(s) of the detection device.
[0028] According to another aspect of the invention, the immersive reality system is configured to generate an augmented reality environment in which three-dimensional or two-dimensional virtual elements are superimposed on a real environment.
[0029] According to another aspect of the invention, the immersive reality system is configured to generate an augmented reality environment, further enabling interaction between a virtual environment and a real environment. This type of system is also called mixed reality.
[0030] In other words, in a mixed reality application, three-dimensional or two-dimensional virtual elements are superimposed on a real environment (augmented reality) and it is possible to interact with the virtual elements, in particular by hand gestures, as in virtual reality.
[0031] According to one aspect of the invention, the immersive reality system is configured to operate an entertainment or relaxation session for the passenger wearing the display device, who can interact, in particular with their hands, with a virtual entertainment or relaxation environment.
[0032] According to another aspect of the invention, the immersive reality system can be used for driver training purposes or for commercial promotion of a vehicle.
[0033] In the present invention, a visor helmet is a display device, worn on the head or in a helmet of the passenger, comprising a display screen in front of one eye (monocular visor helmet), or two display screens in front of each eye (binocular visor helmet).
[0034] The display screens are miniaturized and can, for example, be liquid crystal displays (LCDs), liquid crystal displays on silicon (LCoS) or organic light-emitting diode (OLED) displays.
[0035] According to one aspect of the invention, the data processing unit is connected by wire to the display device.
[0036] Alternatively, the data processing unit is connected wirelessly to the display device, in particular by a wireless communication protocol such as Bluetooth or WIFI.
[0037] According to one aspect of the invention, the data processing unit is part of a vehicle on-board computer.
[0038] According to another aspect of the invention, the data processing unit is integrated into the display device. Thus, for example, the head-mounted display may include the data processing unit which communicates with the sensors and performs the calculations to determine the interactions between, for example, the passenger's hands and the immersive reality scene.
[0039] The invention also relates to a method for managing an immersive reality system, for example virtual reality, embedded in a vehicle, this method comprising the following steps: - detect a movement and / or orientation and / or position of the hands and / or head of a passenger, preferably of multiple passengers on board the vehicle, using an immersive reality system detection device, comprising at least one sensor installed in a fixed location within the vehicle's passenger compartment, - display immersive reality images in a system display device, worn on a passenger's head, based on data obtained by the detection device, in particular to allow hand interaction with an object in an immersive reality scene displayed by the display device.
[0040] According to one aspect of the invention, the immersive reality scene is a virtual reproduction of the vehicle's interior. Thus, the passenger can be immersed in the vehicle's environment, with additional virtual objects in this virtual scene, for example, objects dedicated to entertainment (e.g., a virtual pencil for drawing) or to a relaxation session (e.g., a virtual ball that the passenger can grab).
[0041] The virtual reality scene may contain an image of another passenger. Thus, two or more people in the passenger compartment can interact in the virtual reality world.
[0042] The sensors installed in the passenger compartment make it possible to detect the movements of all the passengers seated in the vehicle.
[0043] The invention is particularly well suited to an autonomous driving vehicle. This allows passengers, including the driver, to benefit from the reality system immersive, without having to concentrate on driving the vehicle.
[0044] Other features and advantages of the invention will become more apparent upon reading the following description, given by way of illustrative and non-limiting example, and the accompanying drawing on which:
[0045] - [Fig. 1] illustrates, schematically and partially, a system of reality immersive intended to be carried on board a vehicle, according to an example of implementation of the invention.
[0046] Figure 1 shows an immersive reality system 1, here a virtual reality system, intended to be embedded in a vehicle V, this system 1 comprising: - a detection device 2 configured to detect a movement and / or an orientation and / or a position of the hands 3 and / or the head 4 of a plurality of passengers 5 on board the vehicle V, this detection device 2 comprising sensors 7 and 8 installed at fixed locations 9 and 10 respectively in a passenger compartment H of the vehicle V, - an immersive reality image display device 15 configured to be worn on the head of a passenger 5, this display device being here a head-mounted display (HMD), - a data processing unit 16 configured to provide immersive reality images to the display device 15, based on data obtained by the detection device 2, to enable interaction of the hands 3 with an object in an immersive reality scene displayed by the display device 15.
[0047] The term "passenger" may refer to the driver himself or a person sitting next to him, or in the back.
[0048] The detection device 2 is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger 5 installed in a seat in the front or rear of the vehicle.
[0049] The sensors 7 and 8 fixed in the passenger compartment are at several locations 9 and 10 respectively, in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger anywhere in the passenger compartment.
[0050] The sensor 7 is positioned on a location 9 of a ceiling 18 of the vehicle, in particular in a front ceiling light of the vehicle or a dome.
[0051] The sensor 8 is located in a lower part 10 of the windshield 19 of the vehicle, specifically substantially in front of the driver. Alternatively, a camera-type sensor 8 may be integrated into the rearview mirror.
[0052] The detection device 2 is configured to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of several passengers 5 to at the same time, installed in the passenger compartment, whether on the front seats or the rear seats.
[0053] The detection device 2 and the display device 15 are mechanically separated from each other, namely the sensor of the detection device is not mechanically linked to the display device, unlike some virtual reality devices in which the head-mounted display incorporates sensors to determine the position, orientation and movement of the wearer of the head-mounted display.
[0054] The immersive reality image display device 15 is devoid of motion and / or orientation and / or position detection sensors for the head 4 or hands 3.
[0055] The sensor 7 of the detection device 2 is configured to deliver a three-dimensional image of a detection area 21 which is the entire passenger compartment H.
[0056] Sensor 7 is selected from: - a time-of-flight sensor, such as a time-of-flight camera (TOF), adapted to send light towards the passenger and measure the time it takes for this light to return to said time-of-flight sensor in order to deduce the three-dimensional image in a detection area, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to deduce a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection area and to analyze the deformation of this pattern to deduce the three-dimensional image of the conductor.
[0057] The sensor 8 is configured to deliver a two-dimensional image of the detection area 21.
[0058] The sensor 8 is an RGB camera, or an infrared camera, in particular of the near-infrared type (or NIR according to the Anglo-Saxon acronym).
[0059] The immersive reality system 1 is configured to allow the passenger 6 wearing the immersive reality display device 16 to interact with an object in the virtual or real environment, in particular with a hand, for example by touching it with a hand or by grasping it with a hand.
[0060] The immersive reality system 1 is configured to generate a fully virtual environment which is perceptible in the virtual reality headset 16, which is totally occulting, namely the real environment is totally occulted.
[0061] In this system according to the invention, it is possible, while being totally immersed in the virtual environment, to turn one's head 360°, move in space and interact with the virtual environment using hands whose movements are detected by the sensor(s) of the detection device.
[0062] Alternatively, the immersive reality system could be configured to generate an augmented reality or mixed reality environment in which three-dimensional or two-dimensional virtual elements are superimposed on a real environment.
[0063] Depending on the needs, the immersive reality system is configured to generate an augmented reality or mixed reality environment allowing, in addition, an interaction between a virtual environment and a real environment.
[0064] The immersive reality system 1 is configured to operate an entertainment or relaxation session for the passenger 5 who is wearing the display device 16, which passenger 5 can interact, with his hands, with a virtual entertainment or relaxation environment.
[0065] The immersive reality system 1 could be used for driver training purposes or for commercial promotion of a vehicle.
[0066] The visor helmet 15 is a display device, worn on the head 5 or in a passenger's helmet, comprising a display screen in front of one eye (monocular visor helmet), or two display screens in front of each eye (binocular visor helmet). The visor helmet can be completely opaque or translucent, allowing visibility through it, depending on the requirements.
[0067] Display screens are miniaturized and can, for example, be liquid crystal displays (LCDs), liquid crystal displays on silicon (LCoS) or organic light-emitting diode (OLED) displays.
[0068] The data processing unit 16 is connected by wire to the display device 15.
[0069] Alternatively, the data processing unit 16 is connected wirelessly to the display device 15, in particular by a wireless communication protocol such as Bluetooth or WIFI.
[0070] For example, the data processing unit 16 is part of a vehicle's on-board computer.
[0071] It is also possible that the data processing unit 16 is integrated into the display device 15. Thus, for example, the head-mounted display 15 may include the data processing unit 16 which communicates with the sensors 7 and 8 and performs the calculations to determine the interactions between, for example, the passenger's hands 4 and the immersive reality scene.
[0072] In the example described, the immersive reality scene is a virtual reproduction of the vehicle's passenger compartment H. Thus, passenger 6 can be immersed in the vehicle's environment, with additional virtual objects in this virtual scene, by examples dedicated to entertainment (e.g., a virtual pencil for drawing) or to a relaxation session (e.g., a virtual balloon that the passenger can grab).
[0073] The virtual reality scene may contain an image of another passenger. Thus, two or more people in the passenger compartment can interact in the virtual reality world.
[0074] Sensors 7 and 8 installed in the passenger compartment H allow the movements of all passengers seated in the vehicle to be detected.
[0075] The invention is particularly well suited to an autonomous driving vehicle. This allows passengers, including the driver, to enjoy the immersive reality system without having to concentrate on driving the vehicle.
Claims
Demands
1. An immersive reality system (1), for example virtual reality, intended to be carried in a vehicle, this system comprising: - a detection device (2) configured to detect a movement and / or an orientation and / or a position of the hands and / or head of a plurality of passengers at once, installed in the passenger compartment, whether in the front seats or the rear seats, this detection device having several sensors (7, 8) fixed in the passenger compartment, at several locations in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or head of a passenger anywhere in the passenger compartment, - an immersive reality image display device (15) configured to be worn on the head of a passenger, this display device being in particular a head-mounted display or glasses configured to display images and allow the passenger to see through them,- a data processing unit (16) configured to provide immersive reality images to the display device (15), based on data obtained by the detection device (2), to enable hand interaction with an object in an immersive reality scene displayed by the display device,
2. Immersive reality system (1) according to the preceding claim, wherein the immersive reality image display device (15) is devoid of motion and / or orientation and / or head or hand position detection sensors.
3. Immersive reality system (1) according to any one of the preceding claims, wherein the sensor (7) of the detection device is configured to deliver a three-dimensional image of a detection area.
4. Immersive reality system (1) according to the preceding claim, wherein the sensor (7) of the detection device is selected from: - a time-of-flight sensor, such as a time-of-flight camera, (“Time of Flight” or TOF according to the English acronym) Saxon) adapted to send light towards the passenger and to measure the time it takes for this light to return to said time-of-flight sensor in order to deduce the three-dimensional image in a detection area, - a stereoscopic sensor comprising at least two cameras, each capturing an image of a detection area from its own point of view, the images from each camera then being combined to deduce a three-dimensional image of the detection area, - a structured light sensor adapted to project a pattern onto the detection area and to analyze the deformation of this pattern to deduce the three-dimensional image of the driver 4.
5. Immersive reality system (1) according to any one of claims 1 or 2, wherein the sensor (8) of the detection device is configured to deliver a two-dimensional image of a detection area.
6. Immersive reality system (1) according to any one of the preceding claims, wherein the sensor (8) is positioned on a ceiling (18) of the vehicle, in particular in a front ceiling light of the vehicle.
7. Immersive reality system (1) according to any one of claims 1 to 5, wherein the sensor (8) is disposed in a lower part of the windshield (19) of the vehicle, in particular substantially in front of the driver.
8. Immersive reality system (1) according to any one of the preceding claims, wherein the immersive reality system (1) is configured to allow the passenger wearing the immersive reality display device to interact with an object in the virtual or real environment, including with a hand, for example by touching it with a hand or grasping it with a hand.
9. Immersive reality system (1) according to any one of the preceding claims, wherein the immersive reality system (1) is configured to generate a fully virtual environment that is perceptible in the virtual reality headset, which is totally occulting, namely the real environment is totally obscured.
10. Immersive reality system (1) according to any one of claims 1 to 8, wherein the immersive reality system (1) is configured to generate an augmented reality environment in which three-dimensional or two-dimensional virtual elements are superimposed in a real-world environment.
11. Immersive reality system (1) according to the preceding claim, wherein the immersive reality system (1) is configured to generate an augmented reality environment allowing, in addition, interaction between a virtual environment and a real environment.
12. Immersive reality system (1) according to any one of the preceding claims, wherein the immersive reality system (1) is configured to operate an entertainment or relaxation session for the passenger wearing the display device (15), which passenger can interact, including with their hands, with a virtual entertainment or relaxation environment.
13. A method for managing an immersive reality system (1), for example virtual reality, embedded in a vehicle, this method comprising the following steps: - detecting a movement and / or an orientation and / or a position of the hands and / or the head of a plurality of passengers at once, seated in the passenger compartment, whether in the front seats or the rear seats, by means of a detection device (2) of the immersive reality system, comprising several sensors (7, 8) fixed in the passenger compartment, at several locations in the passenger compartment, so as to be able to detect a movement and / or an orientation and / or a position of the hands and / or the head of a passenger anywhere in the passenger compartment, - displaying immersive reality images in a display device (15) of the system, worn on the head of a passenger, according to data obtained by the detection device,to enable hand interaction with an object in an immersive reality scene displayed by the display device.
14.