Adaptive augmented reality vision system for automotive vehicle cockpit

Abstract

Augmented reality vision system comprising an augmented reality virtual object display management module (MRA) embedded in a vehicle cockpit (COK) capable of cooperating with an augmented reality optical display device worn in front of the eyes of a user (UTI) in the cockpit (COK), enabling the display of a set of a determined number of virtual objects (1-6) superimposed on the instrument panel (PDB), or interfaced with the instrument panel (PDB) and the windshield (PRB) of the vehicle (VHL), arranged according to a determined arrangement, in the field of view (CVL) of the equipment (LRA) which is smaller than the field of view of the user (UTI); said module (MRA) comprising computing means (MDC) capable of dynamically adjusting the size and positioning of the virtual objects (1-6) in the field of view of the equipment according to the position or orientation of the user's head. (Figure 4)

Description

Title of the invention: Adaptive augmented reality vision system for automotive vehicle cockpit

[0001] The present invention relates generally to augmented reality vision systems for motor vehicle cockpits and relates more particularly to an adaptive augmented reality vision system capable of displaying virtual objects in various locations in the cockpit by dynamically and scalably adapting the positioning and sizing of the objects according to the position or orientation of the user's head.

[0002] By virtual objects, we mean graphic representations and / or images in two or three dimensions relating to vehicle control / status information, multimedia information, driver assistance information, ... which will be displayed spatially in the user's environment.

[0003] Today, this information is mainly displayed on fixed physical media such as screens, generally touch screens, installed on the dashboard, the central console, ..., in the driver's field of vision.

[0004] With the development of augmented reality techniques and in particular those using a display support such as glasses, helmet, ... which are all optical equipment worn on the head of a user, driver or passenger of the vehicle, and which covers at least the eyes of the user, it becomes possible to position this visual information in different locations in the passenger compartment or cockpit of the vehicle or even outside the cockpit, outside the windshield.

[0005] However, the field of vision of the display support worn by the user, for example augmented reality glasses, is limited to about 50° today, which remains less than the human field of vision which is around 180°, which limits, on the one hand, the number and size of virtual objects representing the visual information to be displayed and, on the other hand, some virtual objects may no longer be visible to the user when the latter turns his head or changes position.

[0006] The present invention aims to overcome these drawbacks by offering a solution enabling the user, wearing augmented reality glasses, to view virtual objects in his environment at different locations, including in the cockpit or even outside the cockpit, with an evolving, dynamic arrangement, depending on the position or orientation of the user's head.

[0007] To this end, the invention's first object is an augmented reality vision system comprising a module for managing the display of virtual objects in reality augmented capable of cooperating with an augmented reality optical display equipment worn in front of the eyes of a user of said equipment, present in the cockpit of a motor vehicle, allowing the display of a set of a determined number of virtual objects, in the environment of the user, from the cockpit, arranged according to a determined arrangement in the field of vision of the equipment which is less than the field of vision of the user; said module comprising computing means capable of dynamically adjusting the size and positioning of the virtual objects in the field of vision of the equipment according to the position or orientation of the user's head.

[0008] According to one feature, the system further comprises a device for detecting the position or orientation of the user's head, coupled to the module, enabling the provision of position or orientation data of the user's head to the computing means to adapt the arrangement of virtual objects in the field of vision of the equipment with the movement of the field of vision of the equipment.

[0009] According to another feature, the detection device is a camera arranged in the vehicle cockpit or on the equipment so as to be able to detect the position or orientation of the user's head.

[0010] According to another feature, the management module is hosted in the equipment coupled to the module by means of waves or by wired connection.

[0011] According to another feature, the management module is hosted in a remote server communicating wirelessly with the equipment.

[0012] According to another feature, the optical equipment is a pair of augmented reality glasses.

[0013] According to another feature, the computing means implement a first software for repositioning and resizing virtual objects and a second software, cooperating with the first, for tracking the position or orientation of the user's head.

[0014] According to another feature, the virtual objects are provided by a database of virtual objects defining a library coupled to the management module.

[0015] According to another feature, the virtual objects are predefined by default or selectable by the user in the library; some selected virtual objects can be chosen to be fixed and therefore potentially not visible for certain positions or orientations of the user's head.

[0016] According to another characteristic, the virtual objects define human-machine interfaces chosen from: a display panel of at least one piece of vehicle information, a three-dimensional object representing an avatar in relation to a virtual assistant, a map of a navigation route, a parameter vehicle function settings, driver assistance information, infotainment information.

[0017] The present invention has the advantage of dynamically positioning and resizing virtual objects so that all or some of them, previously defined, are always fully visible to the user regardless of the user's head position or orientation.

[0018] Other advantages and features of the present invention will become clearer from the following description, given solely by way of non-limiting example and with reference to the drawings in which:

[0019] [Fig-1] illustrates a first schematic view of a vehicle cockpit with a first fictitious arrangement of a set of virtual objects that would appear to a user wearing augmented reality glasses without limitation of the user's field of vision;

[0020] [Fig.2] illustrates a second schematic view of the cockpit with the same first arrangement but with limitation of the field of vision by the augmented reality glasses;

[0021] [Fig.3] illustrates a third schematic view of the cockpit with a second arrangement of all the virtual objects processed by the system according to the invention; and

[0022] [Fig.4] illustrates a fourth schematic view of the cockpit with a third arrangement of all the virtual objects processed by the system according to the invention following a change in position or orientation of the user's head.

[0023] Figure 1 schematically illustrates a first view of the interior of a cockpit (COK) of a vehicle in which a user wearing augmented reality glasses, here the driver of the vehicle, is seated at the steering wheel and therefore facing the instrument panel. It should be noted that the user is not necessarily the driver but may be a passenger in any of the vehicle's seats.

[0024] The UTI user, here the driver, is schematically represented by his head which, in the figure, is in a first driving position called "head straight", that is to say, a driving position in which the face of the UTI user is oriented towards the front of the vehicle VHL with a field of vision substantially centered with respect to the steering wheel VOL.

[0025] In the example of [Fig.1], a first arrangement of a set of six virtual objects 1 to 6 that could be visualized by the UTI user wearing LRA augmented reality glasses (represented by dashed line) has been "fictitiously" represented if the LRA glasses had a field of vision greater than or equal to the human field of vision.

[0026] These virtual objects 1-6 are all virtual Human-Machine Interfaces (HMIs) which relate, for example and not limited to, virtual panels containing information about the VHL vehicle, three-dimensional (3D) objects such as an avatar linked to a virtual assistant, a map of a navigation route, a photo, driver assistance information and any other infotainment (multimedia) information, a setting parameter for a vehicle function, ...

[0027] Augmented reality glasses allow the user to perceive the real environment in which they find themselves with visual cues (virtual objects) superimposed on the real environment, displayed by augmented reality. The real environment here is essentially the instrument panel (PDB) and its interface with the windshield (PRB) (primarily the lower part of the windshield). It should be noted that the real environment can be more broadly defined, encompassing everything inside the vehicle as well as everything outside the vehicle (VHL) as seen through the windshield (PRB) and the other windows of the vehicle (VHL).

[0028] The lenses of the LRA glasses are used as a display support, or screen, for virtual objects 1-6.

[0029] In this example, the arrangement of the six virtual objects 1-6 extends roughly along the entire length of the dashboard PDB, slightly overlapping the left side window of the vehicle VHL (driver's side) and the bottom of the windshield PRB. The information carried by the virtual objects, their positioning relative to each other, and their size (dimensioning) were predefined.

[0030] In the example considered, an augmented reality display management module (ARVM) is implemented in the vehicle, for example in a multimedia system of the vehicle (VLM), not shown. A database defining the BOV library of virtual objects available for display is coupled to the ARVM.

[0031] The MRA module communicates wirelessly, via Bluetooth or Wi-Fi, or via a wired connection with the LRA augmented reality glasses to display specific virtual objects with a predetermined arrangement defined by the MRA module. Alternatively, the MRA management module could be embedded in the LRA glasses or even in a remote "cloud" server (not shown); the LRA glasses would then have wireless communication capabilities (not shown) with the "cloud".

[0032] A device for detecting the position or orientation of the UTI user's (here the driver's) head, comprising, for example, a camera fixed in the cockpit, on the dashboard, facing the UTI driver, or on the roof near the interior rearview mirror (as shown in the figure) and pointing towards the UTI driver's head, is coupled to the MRA module. The LRA glasses can also be equipped with a motion detection device, not shown, coupled to the MRA module by radio waves or by wired connection.

[0033] Fig. 2 illustrates the same first arrangement of the six virtual objects 1-6 introduced above, but taking into account the CVL field of view of the LRA glasses, represented in solid line, which is less than the human field of view, and which is represented on the figure by a rectangular frame in dashed line.

[0034] The user UTI, wearing the LRA glasses, only sees the portion of the virtual objects that are located inside the CVL frame, here objects 1, 2, 3 and 5, and object 6 in its entirety. Object 4, which is located outside the CVL frame, is therefore not visible to the user UTI.

[0035] The objective of the present invention is therefore to reposition and possibly dynamically resize these objects 1-6 to make them always fully visible to the UTI user within the CVL frame.

[0036] In this way, the UTI user does not need to turn their head to view either a virtual object that was partially visible in its entirety, or a virtual object that was not visible at all.

[0037] Fig. 3 illustrates an example of a second arrangement in which the six objects 1-6 are contained within the CVL frame with new positionings and dimensions of the objects 1-6.

[0038] For this purpose, the MRA module includes MDC computing means implementing a first software which manages the positioning and dimensioning of objects 1-6 to display them on the LRA augmented reality glasses, represented in solid line, in the CVL field of vision of the LRA glasses.

[0039] This first software is capable of adjusting objects 1-6 in size and positioning according to the field of vision of the LRA glasses.

[0040] Advantageously, the CAM camera and the MRA module also allow the user's head to be tracked to display a new arrangement of the six virtual objects 1-6 in another location of the COK cockpit corresponding to the new position or orientation of the user's head, allowing the user to always view all the objects or some of them only if certain objects from among all the objects have been selected to remain fixed regardless of the position or orientation of the user's head.

[0041] Fig. 4 illustrates an example of a third arrangement of the set of six objects 1-6 in the CVL frame which is here superimposed on the right part of the PDB instrument panel and on the interface between the PDB instrument panel and the PRB windscreen; the UTI user having turned their head to the right.

[0042] For this purpose, the MDC computing means also implement a second software, cooperating with the first software, which, from the position of the user's head UTI, obtained for example by a "head tracking" type process, or other, manages the movement of the CVL field of vision by repositioning and resizing objects 1-6 dynamically and adaptively to display them on the LRA augmented reality glasses, represented in solid lines, according to the movement of the user's head UTI.

[0043] The MDC computing means are capable of adjusting objects in size and positioning, possibly taking into account the viewing angle, according to the data of change of position or orientation of the head of the UTI user.

[0044] The present invention thus makes it possible to create new information display configurations in a motor vehicle cockpit integrating virtual HMIs and potentially makes it possible to reduce or even eliminate screens and other "physical" display devices (saving in mass and cost).

Claims

Demands

1. Augmented reality vision system comprising an augmented reality virtual object display management module (ARM) capable of cooperating with an augmented reality optical display device (ARD) worn in front of the eyes of a user (UTI) of said device (ARD), present in the cockpit of a motor vehicle (HV), enabling the display of a set of a determined number of virtual objects (1-6) in the environment of the user (UTI), from the cockpit (COK), arranged according to a determined arrangement, in the field of view (FOV) of the device (ARD) which is smaller than the field of view of the user (UTI); said module (ARM) comprising computing means (CM) capable of dynamically adjusting the sizing and positioning of the virtual objects (1-6) in the field of view (FOV) of the device (ARD) according to the position or orientation of the head of the user (UTI).

2. System according to the preceding claim, further comprising a device (CAM) for detecting the position or orientation of the user's head (UTI), coupled to the module (MRA), enabling the provision of user's head (UTI) position or orientation data to the computing means (MDC) to adapt the arrangement of virtual objects (1-6) in the field of view (CVL) of the equipment (LRA) with the displacement of the field of view (CVL) of the equipment (LRA).

3. System according to the preceding claim, wherein the detection device is a camera (CAM) arranged in the cockpit (CPK) of the vehicle (VHL) or on the equipment (LRA) so as to be capable of detecting the position or orientation of the user's head (UTI).

4. System according to any one of the preceding claims, wherein the management module (MRA) is housed in the equipment (LRA) coupled to the module (MRA) by means of waves or by wired connection.

5. System according to any one of claims 1 to 3, wherein the management module (MRA) is hosted in a remote server communicating wirelessly with the equipment (LRA).

6. System according to any one of the preceding claims wherein the optical equipment is a pair of augmented reality (AR) glasses.

7. System according to any one of the preceding claims, wherein the computing means (MDC) implement a first software for repositioning and resizing virtual objects (1-6) and a second software, cooperating with the first, for tracking the position or orientation of the user's head (UTI).

8. System according to any one of the preceding claims, wherein the virtual objects (1-6) are provided by a virtual object database defining a library (BOV) coupled to the management module (MRA).

9. System according to the preceding claim, wherein the virtual objects (1-6) are predefined by default or user-selectable (UTI) in the library (BOV); certain selected virtual objects can be chosen to be fixed and therefore potentially not visible for certain positions or orientations of the user's (UTI) head.

10. System according to any one of the preceding claims, wherein the virtual objects (1-6) define human-machine interfaces selected from: a display panel of at least one vehicle information (VHL), a three-dimensional object representing an avatar in relation to a virtual assistant, a map of a navigation route, a setting parameter of a vehicle function (VHL), a driver assistance information, an infotainment information.

Images