Method and control device for operating a head-mounted display device in a motor vehicle

Abstract

The invention relates to a method for operating a head-mounted display device (3) in a motor vehicle (1), wherein the display device (3) is controlled to display a virtual environment (8), in which a vehicle occupant (2) wearing the display device (3) moves virtually in a manner coordinated with the detected movement of the motor vehicle (1), the method comprising the following steps: determining at least one potential parking point (6) of the motor vehicle (1); upon detecting a deceleration of the motor vehicle (1) in a predetermined environment (7) of the determined potential parking point (6): controlling the display device (3) to output a virtual scene (12) in the virtual environment (8), which virtual scene causes a deceleration of the virtual movement in the virtual environment (8) as a function of the deceleration of the motor vehicle (1). The invention also relates to a control device (4) designed to control the head-mounted display device (3).

Description

Technical Field

[0001] The invention relates to a method for operating a head-mounted display device in a motor vehicle. The invention also relates to a control device for operating a head-mounted display device in a motor vehicle and a motor vehicle having such a control device. Background Art

[0002] The future is likely to offer more and more possibilities with which vehicle occupants can enjoy virtual experiences while traveling in a motor vehicle. In the context of such virtual experiences in motor vehicles (also known as In-Car VR Experiences), the displayed virtual content is usually coordinated with the movement of the motor vehicle in question in order to increase the sense of immersion and avoid so-called simulator sickness.

[0003] Thus, for example, DE 10 2014 220 053 A1 discloses a method for operating a head-mounted display device in a motor vehicle, in which the display device is controlled to display a virtual environment in which a vehicle occupant wearing the display device is virtually moved as a function of detected movements of the motor vehicle.

[0004] DE 10 2017 005 982 A1 describes a method for preventing or reducing motion sickness symptoms when using virtual reality glasses in a vehicle while driving. In addition to displaying the playback content, an additional movable grid structure is displayed, wherein the grid structure is also moved so that its balance corresponds consistently to the current dynamic driving dynamics movement state of the vehicle.

[0005] DE 10 2014 220 001 A1 discloses an autonomous vehicle media control device. Here, it is determined whether an event has occurred that indicates stopping or modifying the playback of media content. This may be the case, for example, if the vehicle is in a traffic situation that requires more attention from the vehicle occupants who are consuming media content. In this case, the playback of the media content can be stopped. Summary of the invention

[0006] The object of the present invention is to provide a solution which enables a particularly immersive and convincing virtual experience by means of a display device mounted on the head of a vehicle occupant.

[0007] In the method according to the invention for operating a head-mounted display device in a motor vehicle, the display device is controlled to display a virtual environment in which a vehicle occupant wearing the display device moves virtually in a manner coordinated with the detected movement of the motor vehicle. The head-mounted display device can be, for example, virtual reality glasses, augmented display glasses, or also augmented reality contact lenses. Data characterizing the movement of the motor vehicle can be provided, for example, by the motor vehicle itself and transmitted at least indirectly to the display device, so that the virtual movement can be coordinated with the real movement of the motor vehicle. If the motor vehicle accelerates or brakes, the vehicle occupant wearing the display device accelerates accordingly virtually in the virtual environment shown. This applies not only to longitudinal acceleration and lateral acceleration, but also to acceleration in the height direction. Similarly, rotational movements and accelerations, such as yaw acceleration or yaw movement of the motor vehicle, can also be realized in the virtual environment.

[0008] The method according to the invention is characterized in that at least one potential stopping point of the motor vehicle is determined. Once a deceleration of the motor vehicle in a predetermined environment of the determined potential stopping point is detected, a display device is controlled to output a virtual scene in the virtual environment, wherein the virtual scene causes a deceleration of the virtual movement in the virtual environment as a function of the deceleration of the motor vehicle.

[0009] The invention is based on the recognition that in the context of passive and / or interactive virtual experiences, so-called in-vehicle virtual reality experiences, the content of which is coordinated with the movement of the vehicle in order to increase the sense of immersion and avoid simulator sickness, a particular challenge for convincing storytelling is posed by the difficulty or unpredictability of the parking of the relevant vehicle. That is, it is difficult or impossible to predict in advance when, where and for how long the motor vehicle in question will stop along a determined route. This makes it difficult to create a coherent and inherently credible, i.e., convincing, virtual experience.

[0010] The present invention is addressed to this recognition and the related problems by determining at least one potential, i.e., possible future parking point of a motor vehicle and, as soon as a deceleration of the motor vehicle is detected in a predetermined environment of the determined potential parking point, controlling a display device worn on the head of a vehicle occupant so that a virtual scene occurs in the previously displayed virtual environment, which causes a deceleration of the virtual movement in the virtual environment in accordance with the deceleration of the motor vehicle.

[0011] Thus, the virtual scene is integrated into the virtual experience, i.e., into the displayed virtual environment, before the potential parking point is determined. If, for example, the motor vehicle actually slows down before the determined potential parking point and, for example, stops at the parking point, the output virtual scene results in a particularly immersive and convincing virtual experience for the vehicle occupant wearing the head-mounted display device. At least one potential parking point for the motor vehicle can be, for example, a traffic light, an intersection, a street junction, a jam, or other parked vehicles. Thus, even in the case of, for example, stopping directly and unexpectedly at a traffic light, a consistent virtual overall experience, i.e., a particularly immersive and convincing virtual experience, is created with the aid of the head-mounted display device.

[0012] Therefore, according to the invention, it is provided that the recognition of an impending stop when approaching a potential stop is linked to the reproduction of a virtual experience by means of a display device that can be worn on the head. In order to make the virtual experience feel plausible, the following content is presented in the displayed virtual environment in the form of a virtual scene: it plausibly represents the parking process of the motor vehicle before or at a determined potential stop. Therefore, the parking of the motor vehicle, especially even if it is a sudden stop, does not cause an interruption of the content in the virtual experience, which would destroy the immersion that is very important for the virtual experience. In this case, the virtual scene matches the previously presented virtual experience in terms of content and visuals. If the presented virtual experience is, for example, a computer game, the virtual scene includes one or more virtual elements, which are seamlessly integrated into the story of the computer game. Therefore, the virtual experience of the vehicle occupant wearing the display device on his head is also particularly immersive and convincing, especially when the motor vehicle stops suddenly, because the virtual deceleration caused by the deceleration of the motor vehicle in the virtual environment is convincingly represented by the virtual scene.

[0013] An advantageous embodiment of the invention provides that at least one potential stopping point of the motor vehicle is determined by evaluating at least one of the following data:

[0014] - sensor data characterizing the environment of the motor vehicle;

[0015] - Digital map data;

[0016] - Traffic flow data;

[0017] - Vehicle-to-vehicle data;

[0018] - Data characterizing the driving behavior of a driver of a motor vehicle.

[0019] Various sensors of a motor vehicle can be used, for example, to detect the environment of the motor vehicle and provide corresponding sensor data. By evaluating the sensor data, potential parking points of the motor vehicle can be determined predictively, for example, because of traffic lights, crossroads, junctions, streets, congestion, or, for example, early identification of other vehicles that are parking. Digital map data can also be used, so that based on this, for example, crossroads, stop signs, etc. that will appear can also be identified early along the route traveled by the motor vehicle. In addition, traffic flow data can also be received and evaluated online to determine potential parking points of the motor vehicle in advance. It is also possible that the motor vehicle obtains corresponding data from other motor vehicles through vehicle-to-vehicle communication. Therefore, the vehicle traveling in front can, for example, send information about its own movement to the relevant motor vehicle, so that the motor vehicle can recognize in advance that the vehicle traveling in front is braking or has stopped. In addition, it is also possible to evaluate and analyze data characterizing the driving behavior of the driver of the motor vehicle if the driver of the motor vehicle is driving personally. For example, it can be estimated whether, when and where to consider the potential parking point of the motor vehicle based on the driver's immediate driving behavior. Of course, the above data can also be used in various combinations different from each other to determine one or more potential parking points of the motor vehicle in a particularly reliable manner.

[0020] Another advantageous embodiment of the invention provides that the virtual scene is output only when the deceleration of the motor vehicle reaches at least one predetermined deceleration intensity. This ensures, for example, that the virtual scene is always introduced in the case of particularly sudden decelerations, in order to achieve a particularly convincing virtual experience. On the other hand, if the motor vehicle decelerates very slowly in reality before a potential stopping point, it may be provided that the virtual scene is not introduced, as this is not absolutely necessary for a convincing virtual experience.

[0021] According to another advantageous embodiment of the invention, it is provided that the virtual scene ends as soon as the motor vehicle accelerates at least at a predetermined acceleration after deceleration and the virtual movement is carried out in accordance with the movement of the motor vehicle. In other words, it can be provided that the virtual scene ends again if the motor vehicle exhibits a defined starting dynamic after deceleration. By ending the virtual scene, a particularly plausible virtual experience is created in the virtual environment for the vehicle occupant wearing the display device, which expresses the continued travel and acceleration of the motor vehicle and thus the acceleration that is actually perceived by the vehicle occupant.

[0022] In another advantageous embodiment of the invention, it is provided that the virtual scene also does not appear if the motor vehicle does not slow down when passing a parking spot. If the motor vehicle, for example, drives towards a traffic light which has previously been determined as a potential parking spot for the motor vehicle and the traffic light, for example, changes from red to green again at the right time so that the motor vehicle can pass the traffic light without slowing down, then the virtual scene does not appear, which otherwise convincingly represents the real virtual environment of the motor vehicle in the virtual experience.

[0023] Another advantageous embodiment of the present invention is provided that the virtual environment includes a space scene, and the vehicle occupant wearing the display device virtually flies through the space scene in a spaceship, wherein as a virtual scene, a defect that occurs on the spaceship is repaired as long as the motor vehicle stops at a potential parking point. Therefore, the real dynamic changes and virtual dynamic changes due to the deceleration of the motor vehicle at the parking point can be reasonably presented to the vehicle occupant wearing the display device in the virtual environment. Once the motor vehicle continues to drive after stopping at the relevant parking point, the virtual repair of the previously staged defect at the spaceship is ended, and the spaceship can also continue to move in the virtual environment corresponding to the movement of the motor vehicle. In this regard, it is also conceivable that, for example, in the cockpit of the spaceship, when approaching the potential parking point accordingly, a warning light is gradually lit, which signals that there is a potential defect at the spaceship. If the parking state of the motor vehicle actually occurs, then the defect that actually occurs at the spaceship in the scene at this time is repaired as long as the motor vehicle is at the parking point. Therefore, a particularly reasonable expression is produced in the virtual experience, because the effect does not suddenly appear on the spaceship. On the other hand, if the motor vehicle does not stop at all at a potential stopping point, the warning light can be extinguished, for example, in this case, and the defect at the spacecraft does not appear in the virtual environment.

[0024] An alternative advantageous embodiment of the invention provides that, when the motor vehicle approaches the predetermined environment of a determined potential parking point in a predetermined manner, at least one virtual element is displayed in the virtual environment, which is designed as a component of the scene to cause a deceleration of the virtual movement in the virtual environment as a function of the deceleration of the motor vehicle. The virtual element can be any virtual element that can make the virtual scene appear plausible when it appears. This can also be the warning light already mentioned above.

[0025] Another advantageous embodiment of the invention provides that the previously displayed virtual elements disappear after passing the parking point if no deceleration of the motor vehicle occurs when passing the parking point. This results in a particularly realistic virtual experience for the vehicle occupant wearing the display device, since the previously displayed virtual elements are cancelled or disappear again, which helps to convincingly represent the deceleration in the virtual environment when parking.

[0026] In a further advantageous embodiment of the invention, it is provided that the virtual environment comprises a space scene, through which the vehicle occupant wearing the display device virtually flies in a first spaceship, wherein a second spaceship is displayed as at least one virtual element, and as a virtual scene, the first spaceship is decelerated by means of the tractor beam of the second spaceship in accordance with the deceleration of the motor vehicle. If the motor vehicle actually stops at a potential parking point, the virtual movement with the first spaceship is decelerated by the tractor beam in accordance with the deceleration of the motor vehicle. Thus, for example, it can be reasonably represented that the motor vehicle stops at a traffic light. If the motor vehicle does not stop at the traffic light, for example, because the traffic light turns green in due time, it can be provided that the second spaceship only flies parallel to the first spaceship and does not activate the tractor beam at all.

[0027] The control device according to the invention is configured to control a head-mounted display device in order to carry out the method according to the invention or an advantageous embodiment of the method according to the invention. Advantageous embodiments of the method according to the invention can be seen as advantageous embodiments of the control device according to the invention and vice versa, wherein the control device has in particular components for carrying out the method steps.

[0028] The motor vehicle according to the invention comprises a control device according to the invention or an advantageous embodiment of the control device according to the invention. Alternatively, it is also possible that the head-mounted display device itself comprises the control device. In this case, the control device can have various data interfaces, by means of which in particular sensor data of the motor vehicle, digital map data, traffic flow data, vehicle-to-vehicle data and data characterizing the driving behavior of the driver of the motor vehicle can be received.

[0029] Further advantages, features and details of the invention are apparent from the following description of preferred exemplary embodiments and with the aid of the drawings. The features and feature combinations mentioned above in the description and the features and feature combinations mentioned below in the description of the drawings and / or shown only in the drawings can be used not only in the respectively given combination but also in other combinations or alone without departing from the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In the accompanying drawings,

[0031] Figure 1 A schematic diagram of a motor vehicle with a vehicle occupant seated therein, the vehicle occupant wearing virtual reality glasses, wherein the motor vehicle is driven toward a potential parking spot;

[0032] Figure 2 A schematic diagram of a virtual environment displayed with the aid of virtual reality glasses is shown. DETAILED DESCRIPTION

[0033] exist Figure 1 , a motor vehicle 1 is shown schematically in which a vehicle occupant 2 wearing virtual reality glasses 3 is seated. A control device 4 is also schematically shown, which is designed to control the virtual reality glasses 3. The motor vehicle 1 can be, for example, an autonomous motor vehicle 1, so that the vehicle occupant 2 can enjoy virtual content without any trouble with the help of the virtual reality glasses 3 without having to take on the task of driving. It can also be a non-autonomous motor vehicle 1, in which case a driver (not shown here) drives the motor vehicle 1 and the vehicle occupant 2 is only a passenger or co-driver. The motor vehicle 1 travels on a certain route 5, along which various potential parking points 6 may be present. Potential parking points 6 can be, for example, traffic lights, intersections, street intersections, traffic jams, or other parked vehicles. In addition, the environment 7 of the only potential parking point 6 shown here is schematically indicated, wherein the environment 7 will be discussed in more detail later.

[0034] exist Figure 2 , a virtual environment 8 displayed by means of virtual reality glasses 3 is shown in a highly schematic manner. While the motor vehicle 1 is traveling, the control device 4 controls the virtual reality glasses 3 to display a virtual environment 8 in which the vehicle occupant 2 wearing the virtual reality glasses 3 virtually moves in a manner coordinated with the detected movement of the motor vehicle 1. In the case shown schematically here, the virtual environment 8 includes a space scene through which the vehicle occupant 2 wearing the virtual reality glasses 3 virtually flies in a first spacecraft 9. The present illustration does not correspond to the virtual perspective of the vehicle occupant 2.

[0035] In the present illustration, there is also a second spaceship 10 flying above the first spaceship 9, which slows down or stops the first spaceship 9 via a tractor beam 11. The second spaceship 10 together with the tractor beam 11 forms a virtual scene 12 which causes a deceleration of the virtual movement within the virtual environment 8. This serves to slow down the virtual movement at potential stopping points 6 (see Figure 1 ) When an actual deceleration process or parking process is carried out, the virtual deceleration generated in the virtual environment 8 appears reasonable.

[0036] The method for operating the virtual reality glasses 3 is explained in more detail below. During the travel of the motor vehicle 1, the control device 4 continuously acquires data characterizing the movement of the motor vehicle 1. Based on the data characterizing the movement of the motor vehicle, the control device 4 controls the virtual reality glasses 3 to display a virtual environment 8 in which the vehicle occupant 2 moves with the first spacecraft 9 based on the detected actual movement of the motor vehicle 1. While traveling along the route 5, the control device 4 continuously determines potential stopping points 6 for the motor vehicle, which can be, for example, traffic lights, intersections, street junctions, jams, or other parked vehicles.

[0037] The control device 4 can receive sensor data characterizing the environment of the motor vehicle 1, for example, from the motor vehicle 1. Based on the sensor data, the control device 4 can predictively identify potential parking points 6 in the direction of travel. It is also possible for the control device 4 to receive digital map data, so that the potential parking points 6 can be determined based on these data. The control device 4 can also receive traffic flow data, for example, from a server or the like, not shown here, in order to recognize potential parking points 6 along the route 5 at an early stage. It is also possible for the control device 4 to receive vehicle-to-vehicle data from other motor vehicles, not shown here, in particular ahead, in order to thereby determine the deceleration or parking of the vehicle ahead based thereon and thus determine the potential parking point 6. If the motor vehicle 1 is driven manually by a driver, not shown here, the control device 4 can also receive data characterizing the driving behavior of the driver, in order to determine the potential parking point 6 based thereon.

[0038] As soon as a deceleration of the motor vehicle 1 is detected in the predetermined environment 7 of the correspondingly determined potential parking point 6, the control device 4 controls the virtual reality glasses 3 to output a virtual scene 12 in the virtual environment 8. Figure 2 In the case described in , this means, for example, that as soon as the motor vehicle 1 has definitely approached the environment 7, the second spacecraft 10 appears in the virtual field of vision of the vehicle occupant 2 in the virtual environment 8. If the motor vehicle 1 in the environment 7 then actually brakes when approaching the potential parking point 6, the control device 4 controls the virtual reality glasses 3 so that the tractor beam 11 in the virtual environment 8 is activated in the virtual scene 12, whereby the movement of the first spacecraft 9 is decelerated by the tractor beam 11 in accordance with the actual deceleration of the motor vehicle 1, and the virtual movement is decelerated.

[0039] For example, if the motor vehicle 1 starts again at the potential parking point 6 after the parking maneuver, the virtual scene 12 ends again at least to the extent that the tractor beam 11 ends, and a virtual movement of the first spacecraft 9 in the virtual environment 8 can be realized according to the real movement of the motor vehicle 1. On the other hand, if the motor vehicle 1 approaches the potential parking point 6 but does not stop there and passes the potential parking point 6, for example, without deceleration, the tractor beam 11 is not activated at all in the virtual environment 8. It can then be provided, for example, that the second spacecraft 10 that was previously present disappears again from the virtual field of vision and thus from the virtual environment 8 after a certain flight phase parallel to the first spacecraft 9.

[0040] Different from Figure 2, it is also possible, for example, that the vehicle occupant 2 in the virtual environment 8 is again flying with a spacecraft in a space scene, wherein a defect however occurs on the spacecraft as a virtual scene and is repaired as soon as the motor vehicle 1 is at a potential parking point 6. This makes it possible to explain to the vehicle occupant 2 wearing the virtual reality glasses 3 in a virtual experience why he or she is parking in the virtual environment 8 in a particularly reasonable manner.

[0041] In this case, it can also be provided, for example, that when approaching the potential stopping point 6 sufficiently, for example when entering the environment 7 of the potential stopping point 6, a warning light is lit in the cockpit of the spacecraft, which indicates that the spacecraft has a potential defect. The warning light can, for example, first light up green in the cockpit of the spacecraft, wherein the light changes to orange as soon as the motor vehicle 1 enters the environment 7. If the motor vehicle 1 then actually slows down in order to stop at the potential stopping point 6, the light can, for example, change to red, and the defect appears in the form of a virtual scene, which is repaired as long as the motor vehicle 1 is at the potential stopping point 6. On the other hand, if the situation arises that the motor vehicle simply passes the potential stopping point 6, the light that was previously lit up orange can change back to green, which can make the spacecraft's unproblematic movement in the virtual environment 8 appear plausible.

[0042] The described embodiments in conjunction with the space scene or the spacecraft are to be understood as examples. In principle, various virtual scenes can be displayed in the form of different virtual sequences by means of virtual reality glasses 3 in order to make the actual deceleration and parking of the motor vehicle 1 at the potential parking point 6 and thus the virtual deceleration and parking in the virtual environment 8 shown plausible. Instead of virtual reality glasses 3, other head-mounted display devices can also be used, such as augmented reality glasses or augmented reality contact lenses. Overall, the described method and the described control device 4 provide a solution that enables a particularly immersive and convincing virtual experience by means of a head-mounted display device that can be worn by the vehicle occupant 2.

Claims

1. A method for operating a head-mounted display device (3) in a motor vehicle (1), wherein: A display device (3) is operated to display a virtual environment (8), in which a vehicle occupant (2) wearing the display device (3) performs virtual movements in a manner coordinated with the detected movement of the motor vehicle (1), Characterized in that the method comprises the following steps: - determining at least one potential parking point (6) for the motor vehicle (1) and a predetermined environment (7) of the determined potential parking point (6); - upon detecting a deceleration of the motor vehicle (1) in a predetermined environment (7) of the determined potential parking point (6): controlling the display device (3) to output a virtual scene (12) in the virtual environment (8), which virtual scene causes a deceleration of the virtual movement in the virtual environment (8) in accordance with the deceleration of the motor vehicle (1), As soon as the motor vehicle (1) approaches a predetermined environment (7) of a determined potential parking point (6) to a predetermined extent, at least one virtual element appears in a virtual environment (8), which virtual element is designed to cause a deceleration of the virtual movement in the virtual environment (8) as a component of the scene (12) in response to a deceleration of the motor vehicle (1).

2. The method according to claim 1, characterized in that At least one potential stopping point (6) for the motor vehicle (1) is a traffic light, an intersection, a street junction, a jam, or other parked vehicles.

3. The method according to claim 1 or 2, characterized in that: At least one potential stopping point (6) for the motor vehicle (1) is determined by evaluating at least one of the following data: - sensor data characterizing the environment of the motor vehicle (1); - Digital map data; - Traffic flow data; - Vehicle-to-vehicle data; - Data characterizing the driving behavior of a driver of a motor vehicle (1).

4. The method according to claim 1 or 2, characterized in that: The virtual scene (12) is output only when the deceleration of the motor vehicle (1) reaches at least a predetermined deceleration intensity.

5. The method according to claim 1 or 2, characterized in that: As soon as the motor vehicle (1) accelerates at least at a predetermined acceleration after deceleration, the virtual scene (12) is ended and a virtual movement is performed in accordance with the movement of the motor vehicle (1).

6. The method according to claim 1 or 2, characterized in that: If the motor vehicle (1) does not decelerate when passing the parking point (6), the virtual scene (12) also does not appear.

7. The method according to claim 1 or 2, characterized in that: The virtual environment (8) comprises a space scene, through which a vehicle occupant (2) wearing a display device (3) virtually flies in a spacecraft, wherein a defect occurring on the spacecraft is presented as a virtual scene (12), which can be repaired as soon as the motor vehicle (1) stops at a potential parking point (6).

8. The method according to claim 1, characterized in that If the motor vehicle (1) does not decelerate when passing the parking point (6), the previously displayed virtual element disappears after passing the parking point (6).

9. The method according to claim 1, characterized in that: The virtual environment (8) comprises a space scene, through which a vehicle occupant (2) wearing a display device (3) virtually flies in a first spaceship (9), wherein a second spaceship (10) is displayed as at least one virtual element, and as a virtual scene (12) the first spaceship (9) is decelerated by means of a tractor beam (11) of the second spaceship (10) in response to the deceleration of the motor vehicle (1).

10. A control device (4) designed to control a head-mounted display device (3) to perform the method according to any one of claims 1 to 9.

11. A motor vehicle (1) having a control device (4) according to claim 10.

Images