Vehicle-mounted immersive reality system

Abstract

The invention relates to an immersive reality system (1), such as a virtual reality system, intended to be embedded on a vehicle, comprising: a detector (2) configured to detect a movement and / or orientation and / or position of a hand (3) and / or head (4) of a passenger (5), and preferably a movement and / or orientation and / or position of a hand (3) and / or head (4) of a plurality of passengers located on the vehicle, the detector comprises at least one sensor (7, 8) mounted in a fixed position (9, 10) in a passenger compartment of the vehicle; -a display (15) for displaying an immersive realistic image configured to be worn on the head of the passenger, in particular a head-mounted display or glasses configured to display an image and to allow the passenger to see through; -a data processing unit (16) configured to deliver an immersive reality image to the display (15) as a function of the data obtained by the detector (2), in particular in order to allow a hand to interact with an object in an immersive reality scene displayed by the display.

Description

Technical Field

[0001] The invention relates in particular to immersive reality systems, such as virtual reality systems, intended to be mounted on vehicles. Background Art

[0002] The vehicle may be a land vehicle, a sea vehicle or an air vehicle.

[0003] Patent application US2015378156 describes a virtual reality system for an actual vehicle. This system allows a vehicle supplier to bring a test vehicle to a potential customer's site, where the system virtually displays the vehicle's configuration based on the customer's specifications. The potential customer can sit inside the vehicle, familiarize themselves with the test vehicle's materials, specify the interior and any custom features, and touch elements of the actual vehicle, such as the steering wheel and controls, while viewing the selected virtual configuration through virtual reality glasses worn by the customer.

[0004] Patent application DE102016225262 discloses glasses that can display images on a display surface within the wearer's field of view. The display surface of the glasses can be translucent, allowing the real environment to be perceived through the glasses as usual, and any information, such as text, symbols, graphics, or video displays, can be displayed by overlaying the perceived environment. In order to obtain the spatial position of the data glasses in the vehicle coordinate system and their three-dimensional orientation, a control unit is provided, which is connected to a camera mounted on the temple of the glasses and is approximately oriented in the direction of the wearer's line of sight. The control unit evaluates the camera image and, using shape recognition methods, allows the location of distinctive structures in the vehicle interior in the camera image, where the position of the structures is known and allows the three-dimensional orientation of the glasses to be determined. Preferably, the spatial position and / or movement of the hands of the person wearing the data glasses is determined when the hands of the person wearing the data glasses are within the detection range of the camera on the glasses.

[0005] Patent application US2016070966 describes a vehicle system that uses the position of the driver's head as a metric for estimating the driver's attention. The system can be configured to measure the position of the driver's head using non-invasive or invasive methods. Non-invasive measurement techniques can include, for example, internal cameras and image processing performed based on captured images of the driver's head. Invasive methods can include, for example, head-mounted devices configured to more directly measure the orientation or position of the driver's head. Invasive methods for measuring the driver's gaze can include glasses worn by the driver, which contain motion sensors (e.g., accelerometers, gyroscopes), which the vehicle system can use to provide information to assess the driver's head position and activity. Summary of the Invention

[0006] The present invention is particularly intended to propose a new immersive reality system, such as a virtual reality system, intended to be installed on board a vehicle.

[0007] The object of the present invention is therefore an immersive reality system, such as a virtual reality system, intended to be installed on a vehicle, comprising:

[0008] a detection device configured to detect movements and / or orientations and / or positions of hands and / or heads of a passenger, preferably of a plurality of passengers on board the vehicle, wherein the detection device comprises at least one sensor mounted in a fixed position in the passenger compartment of the vehicle;

[0009] - a device for displaying immersive reality images, which is configured to be worn on the passenger's head, the display device being in particular a head-mounted display (or HMD) or glasses configured to display images and allow the passenger to see through them;

[0010] - a data processing unit configured to provide an immersive reality image to a display device based on the data obtained by the detection device, in particular allowing the hand to interact with objects in the immersive reality scene displayed by the display device.

[0011] The term "passenger" may designate the driver himself or an individual sitting next to the driver or in the rear of the vehicle.

[0012] According to one aspect of the present invention, the detection device is configured to detect the movement and / or orientation and / or position of the hands and / or head of a passenger located anywhere in the passenger compartment. For example, the detection device is configured to detect the movement and / or orientation and / or position of the hands and / or head of a passenger located at the front or rear of the vehicle.

[0013] According to one aspect of the invention, the detection device comprises a plurality of sensors fixed in the passenger cabin at several locations in the passenger cabin so as to be able to detect the movement and / or orientation and / or position of the hands and / or head of a passenger anywhere in the passenger cabin. These sensors may be identical or, as a variant, different, for example combining sensors of the three-dimensional type and sensors of the two-dimensional type. If applicable, the system may comprise a plurality of three-dimensional sensors so as to cover the entire passenger cabin by providing several viewpoints that complement each other. Similarly, if applicable, the system may comprise a plurality of two-dimensional sensors so as to cover the entire passenger cabin by providing several viewpoints that complement each other.

[0014] According to one aspect of the invention, the detection device is configured to be able to simultaneously detect movements and / or orientations and / or positions of the hands and / or heads of multiple passengers, wherein the passengers are located in the passenger compartment, whether they are on the front seats or on the rear seats.

[0015] In the present invention, the detection device and the display device are mechanically separated from each other, that is, the sensors of the detection device are not mechanically linked to the display device, unlike some virtual reality devices in which the head-mounted display is integrated with sensors for determining the position, orientation, and movement of the individual wearing the head-mounted display. In the present invention, the one or more sensors of the detection device are placed at fixed locations in the passenger compartment, allowing these sensors to detect the position, orientation, and movement of body parts (particularly the head and hands) of passengers in the passenger compartment, while avoiding the need to manage blind spots that remain when the sensors are integrated into the head-mounted display. In some aspects, the one or more sensors placed at appropriate locations in the passenger compartment according to the present invention allow for 360° detection in the passenger compartment, wherein sensors integrated into conventional head-mounted displays generally do not allow any detection in the area behind the head-mounted display.

[0016] The present invention also eliminates the need to use a joystick to interact with objects in the virtual environment because one or more sensors are placed around the passenger in the passenger compartment, which directly detect hand gestures.

[0017] Sensors installed in the passenger compartment can be used for multiple passengers simultaneously. In contrast, if one or more sensors were installed on a head-mounted display, as many sensor arrays as head-mounted devices to be monitored would be required. Therefore, the present invention allows sensors to be pooled for several head-mounted displays or glasses.

[0018] Advantageously, the device for displaying immersive reality images has no sensors for detecting movement and / or orientation and / or position of the head or hands. All of these sensors are placed at fixed locations in the passenger compartment.

[0019] According to one aspect of the invention, the sensor of the detection device is configured to provide a three-dimensional image of the detection area. This is preferred for detecting the position, orientation and movement of the passenger's hand.

[0020] According to one aspect of the present invention, the sensor of the detection device is selected from:

[0021] a time-of-flight sensor, such as a time-of-flight (TOF) camera, adapted to send light towards the passenger and measure the time it takes for the light to return to said time-of-flight sensor in order to determine a three-dimensional image of the detection area;

[0022] a stereo sensor comprising at least two cameras, each camera capturing an image of a detection area from its own viewpoint, and combining the images of each camera in order to determine a three-dimensional image of the detection area;

[0023] - a structured light sensor adapted to project a pattern onto the detection area and to analyze deformations of the pattern in order to determine a three-dimensional image of the driver.

[0024] According to another aspect of the invention, the sensor of the detection device is configured to provide a two-dimensional image of the detection area. This is suitable for detecting the position, orientation and movement of the passenger's head.

[0025] According to one aspect of the invention, the sensor of the detection device is an RGB camera or an infrared camera, in particular an infrared camera of the near infrared (or NIR) type.

[0026] According to one aspect of the present invention, the detection device comprises a plurality of sensors selected from the above sensors, in particular comprising at least one two-dimensional detection sensor and one three-dimensional detection sensor. The two-dimensional detection sensor can advantageously be used to partially increase the resolution of the three-dimensional detection sensor.

[0027] According to one aspect of the invention, the sensor is positioned on the ceiling of the vehicle, in particular in the front ceiling lighting of the vehicle.

[0028] As a variant, the sensor is arranged in the lower part of the windshield of the vehicle, in particular substantially in front of the driver.

[0029] According to one aspect of the invention, one or more sensors are placed on the dashboard of the vehicle, in particular in the central area of ​​the dashboard. One or more sensors can also be placed on the rearview mirror of the vehicle, in particular on the central rearview mirror integrated into the interior of the passenger compartment.

[0030] According to one aspect of the invention, the immersive reality system is configured to allow a passenger wearing an immersive reality display device to interact with an object in a virtual environment or a real environment, in particular with their hands, such as by touching it with their hands or by grasping it with their hands.

[0031] Thus, the present invention allows accurate and extensive detection (e.g. 360°) of the hands of one or more passengers by means of one or more sensors installed in the passenger compartment for the purpose of managing virtual reality interactions.

[0032] According to one aspect of the present invention, an immersive reality system is configured to generate a fully virtual environment perceivable in a virtual reality head-mounted display, wherein the fully virtual environment is fully masked, that is, the real environment is completely blocked.

[0033] In the system according to the invention, it is possible to turn the head 360° while being fully immersed in the virtual environment, move in space and interact with the virtual environment using the hands, the movements of the hands being detected by one or more sensors of the detection device.

[0034] According to another aspect of the present invention, an immersive reality system is configured to generate an augmented reality environment in which virtual three-dimensional or two-dimensional elements are superimposed on a real environment.

[0035] According to another aspect of the present invention, an immersive reality system is configured to generate an augmented reality environment that additionally allows interaction between a virtual environment and a real environment. This type of system is also referred to as a mixed reality system.

[0036] In other words, in mixed reality applications, virtual three-dimensional or two-dimensional elements are superimposed on the real environment (augmented reality), and interaction with the virtual elements is possible, in particular with the help of gestures, as in virtual reality.

[0037] According to one aspect of the invention, an immersive reality system is configured to operate an entertainment or relaxation session intended for a passenger wearing a display device, who can interact with the virtual entertainment or relaxation environment, in particular with their hands.

[0038] According to another aspect of the present invention, the immersive reality system may be used for learning to drive a vehicle or for marketing promotions of vehicles.

[0039] In the present invention, a head-mounted display is a display device worn on a passenger's head or in headphones, and includes a display screen opposite one eye (monocular head-mounted display) or two display screens opposite each eye (binocular head-mounted display).

[0040] The display screen is miniaturized and may be, for example, a liquid crystal display (LCD) screen, a liquid crystal on silicon (LCoS) screen, or an organic light emitting diode (OLED) screen.

[0041] According to one aspect of the present invention, the data processing unit is connected to the display device through a wire.

[0042] As a variant, the data processing unit is connected to the display device wirelessly, in particular via a Bluetooth or Wi-Fi wireless communication protocol.

[0043] According to one aspect of the invention, the data processing unit forms part of an on-board computer of a vehicle.

[0044] According to another aspect of the present invention, a data processing unit is integrated into the display device. Thus, for example, a head-mounted display may include a data processing unit that communicates with sensors and performs calculations to determine, for example, the interaction between a passenger's hands and the immersive reality scene.

[0045] Another object of the present invention is a method for managing an immersive reality system (e.g., a virtual reality system) installed on a vehicle, the method comprising the following steps:

[0046] - detecting movements and / or orientations and / or positions of hands and / or heads of passengers, preferably of a plurality of passengers on board a vehicle, using a detection device of the immersive reality system, the detection device comprising at least one sensor mounted in a fixed position in the passenger compartment of the vehicle;

[0047] - Based on the data obtained by the detection device, displaying an immersive reality image in a display device of the system worn on the passenger's head, in particular allowing hand interaction with objects in the immersive reality scene displayed by the display device.

[0048] According to one aspect of the present invention, the immersive reality scene is a virtual representation of the vehicle's passenger compartment. Passengers can thus be immersed in the vehicle's environment, with the virtual scene including additional virtual objects, for example, specifically designed for entertainment (e.g., a virtual pencil for drawing) or relaxation sessions (e.g., a virtual ball that the passenger can hold).

[0049] The virtual reality scene can include an image of another passenger, so two or more people in the passenger compartment can interact in the virtual reality world.

[0050] Sensors installed in the passenger compartment allow the detection of the movements of all passengers seated in the vehicle.

[0051] The present invention is particularly well-suited for use in autonomous vehicles, allowing passengers (including the driver) to enjoy the immersive reality system without having to focus on driving the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other characteristics and advantages of the invention will become more clearly apparent on reading the following description provided by way of non-limiting illustrative example and from the accompanying drawings, in which:

[0053] [ Figure 1 ] schematically and partially illustrates an immersive reality system intended to be mounted on a vehicle according to one embodiment of the present invention. DETAILED DESCRIPTION

[0054] Figure 1 An immersive reality system 1, in this case a virtual reality system, intended to be mounted on a vehicle V is shown, the system 1 comprising:

[0055] a detection device 2 configured to detect movements and / or orientations and / or positions of hands 3 and / or heads 4 of a plurality of passengers 5 on board a vehicle V, wherein the detection device 2 comprises sensors 7 and 8 mounted at fixed positions 9 and 10 , respectively, in a passenger compartment H of the vehicle V;

[0056] a device 15 for displaying immersive reality images, configured to be worn on the head of the passenger 5 , wherein the display device is in this case a head-mounted display (or HMD);

[0057] a data processing unit 16 configured to provide an immersive reality image to the display device 15 based on the data obtained by the detection device 2 , in particular allowing the hand 3 to interact with objects in the immersive reality scene displayed by the display device 15 .

[0058] The term "passenger" may designate the driver himself or an individual sitting next to the driver or in the rear of the vehicle.

[0059] The detection device 2 is configured to be able to detect the movement and / or orientation and / or position of the hands and / or head of the passenger 5 located at the front or rear of the vehicle.

[0060] Sensors 7 and 8 fixed in the passenger compartment are located at several positions 9 and 10 respectively in the passenger compartment so as to be able to detect movements and / or orientations and / or positions of the hands and / or heads of passengers anywhere in the passenger compartment.

[0061] The sensor 7 is positioned at a position 9 in a ceiling 18 of the vehicle, in particular in a front ceiling lighting or dome of the vehicle.

[0062] The sensor 8 is arranged in the lower part 10 of the vehicle's windshield 19, in particular substantially in front of the driver. As a variant, the camera-type sensor 8 can be integrated into the central rearview mirror.

[0063] The detection device 2 is configured to be able to simultaneously detect movements and / or orientations and / or positions of hands and / or heads of a plurality of passengers 5 located in the passenger compartment, whether they are in the front seat or in the back seat.

[0064] The detection device 2 and the display device 15 are mechanically separated from each other, i.e. the sensors of the detection device are not mechanically connected to the display device, unlike some virtual reality devices in which the head-mounted display is integrated with sensors for determining the position, orientation and movement of the individual wearing the head-mounted display.

[0065] The device 15 for displaying immersive reality images has no sensors for detecting movement and / or orientation and / or position of the head 4 or the hands 3 .

[0066] The sensor 7 of the detection device 2 is configured to provide a three-dimensional image of a detection area 21 , which is the entire passenger compartment H.

[0067] The sensor 7 is selected from:

[0068] a time-of-flight sensor, such as a time-of-flight (TOF) camera, adapted to send light towards the passenger and measure the time it takes for the light to return to said time-of-flight sensor in order to determine a three-dimensional image of the detection area;

[0069] a stereo sensor comprising at least two cameras, each camera capturing an image of a detection area from its own viewpoint, and combining the images of each camera in order to determine a three-dimensional image of the detection area;

[0070] - a structured light sensor adapted to project a pattern onto the detection area and to analyze deformations of the pattern in order to determine a three-dimensional image of the driver.

[0071] The sensor 8 is configured to provide a two-dimensional image of the detection area 21 .

[0072] The sensor 8 is an RGB camera or an infrared camera, in particular of the near infrared (or NIR) type.

[0073] The immersive reality system 1 is configured to allow a passenger 5 wearing an immersive reality display device 16 to interact with an object in a virtual environment or in a real environment, in particular with the hands, eg by touching it with the hands or by grasping it with the hands.

[0074] The immersive reality system 1 is configured to generate a fully virtual environment perceivable in a virtual reality head mounted display 16 , wherein the fully virtual environment is fully masked, ie the real environment is completely blocked.

[0075] In the system according to the invention, it is possible to turn the head 360° while being fully immersed in the virtual environment, move in space and interact with the virtual environment using the hands, the movements of the hands being detected by one or more sensors of the detection device.

[0076] As a variation, the immersive reality system may be configured to generate an augmented reality or mixed reality environment in which virtual three-dimensional or two-dimensional elements are superimposed on a real environment.

[0077] As desired, the immersive reality system is configured to generate an augmented reality or mixed reality environment that additionally allows interaction between the virtual environment and the real environment.

[0078] The immersive reality system 1 is configured to operate an entertainment or relaxation session intended for a passenger 5 wearing a display device 16 , who can interact with the virtual entertainment or relaxation environment, in particular with their hands.

[0079] The immersive reality system 1 may be used for learning to drive a vehicle or for marketing promotions of vehicles.

[0080] The head-mounted display 15 is a display device worn on the head or head-mounted device of the passenger 5, and includes a display screen opposite each eye (monocular head-mounted display) or two displays opposite each eye (binocular head-mounted display). Depending on the needs, the head-mounted display can be fully obscured (opaque) or see-through.

[0081] The display screen is miniaturized and may be, for example, a liquid crystal display (LCD) screen, a liquid crystal on silicon (LCoS) screen, or an organic light emitting diode (OLED) screen.

[0082] The data processing unit 16 is connected to the display device 15 by means of wires.

[0083] As a variant, the data processing unit 16 is connected wirelessly to the display device 15 , in particular via a Bluetooth or Wi-Fi wireless communication protocol.

[0084] For example, the data processing unit 16 forms part of an on-board computer of the vehicle.

[0085] It is also possible that the data processing unit 16 is integrated into the display device 15. Thus, for example, the head mounted display 15 may include a data processing unit 16 which communicates with the sensors 7 and 8 and performs calculations for determining, for example, the interaction between the passenger's hand 3 and the immersive reality scene.

[0086] In the example described, the immersive reality scene is a virtual representation of the vehicle's passenger compartment H. Thus, the passenger 5 can be immersed in the vehicle's environment, with the virtual scene including additional virtual objects, for example, dedicated to entertainment (e.g., a virtual pencil for drawing) or relaxation sessions (e.g., a virtual ball that the passenger can hold).

[0087] The virtual reality scene can include an image of another passenger, so two or more people in the passenger compartment can interact in the virtual reality world.

[0088] The sensors 7 and 8 installed in the passenger compartment H allow the detection of the movements of all passengers seated in the vehicle.

[0089] The present invention is particularly well-suited for use in autonomous vehicles, allowing passengers (including the driver) to enjoy the immersive reality system without having to focus on driving the vehicle.

Claims

1. An immersive reality system (1), such as a virtual reality system, intended to be mounted on a vehicle, the system comprising: - a detection device (2) configured to detect movements and / or orientations and / or positions of the hands (3) and / or head (4) of a passenger (5), preferably of a plurality of passengers on board the vehicle, wherein the detection device comprises at least one sensor (7; 8) mounted in a fixed position (9; 10) in the passenger compartment of the vehicle; - a device (15) for displaying immersive reality images, which is configured to be worn on the head of a passenger, wherein the display device is in particular configured as a head-mounted display or glasses that displays images and allows the passenger to view them; - a data processing unit (16) configured to provide an immersive reality image to the display device (15) based on the data obtained by the detection device (2), in particular allowing the hand to interact with objects in the immersive reality scene displayed by the display device.

2. An immersive reality system (1) according to the preceding claim, wherein the detection device (2) comprises a plurality of sensors (7, 8) which are fixed in the passenger cabin at several locations in the passenger cabin so as to be able to detect movements and / or orientations and / or positions of the hands and / or heads of a passenger at any location in the passenger cabin.

3. An immersive reality system (1) according to any one of the preceding claims, wherein the detection device (2) is configured to be able to simultaneously detect movements and / or orientations and / or positions of the hands and / or heads of a plurality of passengers, wherein the passengers are located in the passenger compartment, whether they are on the front seats or on the rear seats.

4. The immersive reality system (1) according to any of the preceding claims, wherein the device (15) for displaying an immersive reality image is free of sensors for detecting the movement and / or the orientation and / or the position of the head or the hand.

5. The immersive reality system (1) according to any one of the preceding claims, wherein the sensor (7) of the detection device is configured to provide a three-dimensional image of a detection area.

6. Immersive reality system (1) according to the preceding claim, wherein the sensor (7) of the detection device is selected from: a time-of-flight sensor, such as a time-of-flight (TOF) camera, adapted to send light towards the passenger and measure the time it takes for the light to return to said time-of-flight sensor in order to determine a three-dimensional image of the detection area; - a stereo sensor comprising at least two cameras, each camera capturing an image of the detection area from its own viewpoint, and then combining the images of each camera in order to determine a three-dimensional image of the detection area; - a structured light sensor adapted to project a pattern onto a detection area and to analyze the deformation of the pattern in order to determine a three-dimensional image of the driver (4).

7. The immersive reality system (1) according to any one of claims 1 to 4, wherein the sensor (8) of the detection device is configured to provide a two-dimensional image of a detection area.

8. The immersive reality system (1) according to any of the preceding claims, wherein the sensor (8) is positioned on a ceiling (18) of the vehicle, in particular in a front ceiling lighting of the vehicle.

9. The immersive reality system (1) according to any one of claims 1 to 7, wherein the sensor (8) is arranged in a lower part of a windshield (19) of the vehicle, in particular substantially in front of the driver.

10. An immersive reality system (1) according to any one of the preceding claims, wherein the immersive reality system (1) is configured to allow a passenger wearing the immersive reality display device to interact with an object in the virtual environment or the real environment, in particular to interact with it with his hands, for example by touching it with his hands or grasping it with his hands.

11. An immersive reality system (1) according to any one of the preceding claims, wherein the immersive reality system (1) is configured to generate a fully virtual environment perceivable in the virtual reality head mounted display, the fully virtual environment being fully masked, i.e. the real environment is fully obscured.

12. The immersive reality system (1) according to any one of claims 1 to 10, wherein the immersive reality system (1) is configured to generate an augmented reality environment in which virtual three-dimensional or two-dimensional elements are superimposed on a real environment.

13. The immersive reality system (1) according to the preceding claim, wherein the immersive reality system (1) is configured to generate an augmented reality environment that additionally allows interaction between a virtual environment and a real environment.

14. An immersive reality system (1) according to any of the preceding claims, wherein the immersive reality system (1) is configured to operate an entertainment or relaxation session intended for a passenger wearing the display device (15), the passenger being able to interact with the virtual entertainment or relaxation environment, in particular with their hands.

15. A method for managing an immersive reality system (1) installed on a vehicle, the immersive reality system (1) being, for example, a virtual reality system, the method comprising the following steps: - detecting the movement and / or orientation and / or position of a passenger's hands (3) and / or head (4), preferably of a plurality of passengers on board the vehicle, using a detection device (2) of the immersive reality system, the detection device comprising at least one sensor mounted in a fixed position in the passenger compartment of the vehicle; - Based on the data obtained by the detection device, an immersive reality image is displayed in a display device (15) of the system worn on the passenger's head, in particular allowing hand interaction with objects in the immersive reality scene displayed by the display device.

Images