Method for operating a head-mounted display device in a motor vehicle, control device, and head-mounted display device

By integrating sensors into the head-mounted display device to obtain and output the internal images of the motor vehicle, the problem that users find it difficult to perceive the condition in the car during online car-hailing is solved, and safety and comfort are improved, ensuring that users can see it in virtual reality Interior environment.

CN114868071BActive Publication Date: 2024-11-05AUDI AG
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202080070934.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-10
Filing Date
2020-08-11
Publication Date
2024-11-05
Estimated Expiration
2040-08-11

AI Technical Summary

Technical Problem

When using head-mounted display devices in motor vehicles such as online car-hailing, it is difficult for users to sense the condition in the car at the same time, resulting in safety and comfort problems, especially in immersive virtual reality environments where strange drivers or other passengers are not clearly seen action.

Method used

By integrating sensors into the head-mounted display device to obtain the internal image of the vehicle and insert it into the entertainment content to output it, ensuring that users can still see the condition in the car during the immersive experience, enhancing security and comfort.

Benefits of technology

It effectively reduces unpleasant touch in the car, improves the safety and comfort of the user, and allows users to maintain an understanding of the interior environment in virtual reality without interfering with the auditory experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114868071B_ABST
    Figure CN114868071B_ABST
Patent Text Reader

Abstract

The invention relates to a method for operating a head mounted display device (14) in the interior of a motor vehicle (10). A control device (12) receives (S1) a sensor signal from a sensor device (18), wherein the received sensor signal describes at least one image of the current interior condition of the motor vehicle (10). The sensor device (18) generates an insertion signal (S2) describing the insertion (S4) of at least one image of the current interior condition during the output of a prescribed output content, and transmits (S3) the generated insertion signal to a display element of a display device (14), thereby causing the display device (14) to insert (S4) at least one image of the current interior condition during the output of the prescribed output content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for operating a head-mounted display device in a motor vehicle. Background Art

[0002] With the development of autonomous vehicles, head-mounted output devices are becoming increasingly important for entertainment during driving and for the current growing number of driving services. Therefore, a challenge especially with the use of immersive in-car entertainment, for example in ride-hailing, is that the user no longer has an overview of the situation in the vehicle when wearing, for example, VR glasses. What is not a problem in conventional passenger entertainment may well be a problem in, for example, ride-hailing, where the user may want a clear overview of the situation in order to feel more comfortable and safer.

[0003] For example, when immersed in virtual reality ("VR") in such a vehicle, the user has no idea what the driver or other passengers, who are usually strangers, are doing. For example, if an unfamiliar driver wants to talk to a passenger and the driver looks at him, he cannot see this. If the driver strikes up a conversation with him, he can only know if the display device does not cover his ears and the display device does not have headphones for the sound experience of the entertainment program. If the unfamiliar driver is also not perceptible, for example, the driver can only tap the shoulder or touch the arm of the user of the display device, which may not only make the user of the display device very uncomfortable, but also may make the user of the display device feel scared.

[0004] DE 10 2014 010 309 A1 describes a device having a display unit that is worn on the head of a user and is designed to shield the user's eyes from light from the surroundings and to stereoscopically display virtual objects arranged in a virtual space in front of the user's eyes.

[0005] DE 10 2009 049 073 A1 relates to a method for displaying virtual information in a view of a real environment.

[0006] DE 10 2014 213 285 A1 describes a method for displaying content on data glasses.

[0007] One possible way to avoid the above problems is to omit the headphones, but this prevents the user from hearing the sound of the virtual or partially virtual entertainment program, so that for example virtual reality or augmented reality is less immersive. Summary of the invention

[0008] The invention is based on the object of increasing the feeling of safety and comfort when using an entertainment product in a motor vehicle.

[0009] The stated object is achieved by the method according to the invention and the device according to the invention. Advantageous developments are given by the dependent claims.

[0010] The invention is based on the idea that when using a head-mounted display device in a motor vehicle, in addition to the output of the original entertainment content, i.e. in addition to the output of the main content, an image (and / or a sequence of images) of the interior of the motor vehicle acquired by means of one or more sensors is inserted. In particular, the image or the sequence of images is a current image or a current sequence of images. The disadvantages described above are significantly reduced or even eliminated. The user of the head-mounted display device feels safe because, despite being immersed in, for example, virtual reality, he still has an overview of the interior of the motor vehicle. Unpleasant situations in which he is touched by a stranger without warning can be effectively prevented.

[0011] The method according to the invention for operating a head-mounted display device, such as augmented reality glasses, mixed reality glasses or virtual reality glasses, is accordingly a method for operating a head-mounted display device in the interior of a motor vehicle when the head-mounted display device is located in the interior of a motor vehicle. The display device is configured to output a specified output content describing an entertainment product, such as a computer game or a movie. The output content can be specified, for example, by a user selecting the output content and thus the entertainment product, or by a standard setting.

[0012] The method is performed by a control device, wherein a control device can be understood as a device or device component for receiving and analyzing signals and for generating control signals. Therefore, the method can be correspondingly referred to as a computer-implemented method for operating a head-mounted display device. The control device can be configured, for example, as a control chip or a controller. The control device can preferably be a control device of a motor vehicle or a control device of a head-mounted display device.

[0013] The control device receives sensor signals from the sensor device, wherein the received sensor signals record at least one image of the current interior situation of the motor vehicle. A sensor device is to be understood as a device or a device group for detecting environmental properties, in particular for detecting interior properties, so the sensor device can preferably include, for example, at least one interior camera of the motor vehicle and / or, for example, at least one infrared sensor. The at least one image can be, for example, a photograph, a thermal image or an image sequence, for example a video of what is happening in the interior.

[0014] The control device generates an insertion signal that describes the insertion of at least one image of the current interior space situation during the output of the specified output content, and transmits the generated insertion signal to the display element of the display device, thereby causing the display device to insert at least one image of the current interior space situation during the output of the specified output content. The display element is a component or assembly or device part for displaying image content, such as a screen. In other words, the control device causes the display device to output at least one image of the interior space of the motor vehicle in addition to displaying the output content, for example in addition to displaying virtual reality, wherein the at least one image of the interior space is preferably output simultaneously with the output content, so that the at least one image of the interior space is only inserted in a sub-area of ​​the field of view of the user of the display device.

[0015] The advantages described above result here.

[0016] Optionally, the control device can additionally provide defined output content. This is preferably desirable, for example, when the control device is a control device of a motor vehicle and the output content is provided by the motor vehicle, for example, as an entertainment service. However, if the control device is a control device of a display device, the control device can also optionally provide defined output content.

[0017] According to one embodiment of the method according to the invention, it can be provided that the generated insertion signal describes the insertion of at least one image of the current interior situation in only a sub-area of ​​the field of vision of a user of the display device, preferably in the main or central field of vision, so that the user can immediately understand the interior situation.

[0018] Optionally, the received sensor signals can describe at least one camera image of an interior camera of the sensor device. The user can also see, for example, micro-gestures and expressions of other passengers on the inserted image.

[0019] Preferably, the control device can receive sensor signals from a sensor device of the motor vehicle. As a result, the interior can be recorded from different angles and the method is also compatible with such head-mounted display devices that do not have a corresponding sensor system.

[0020] The specified output content can, for example, describe a virtual reality, in particular a 360-degree virtual reality. This allows the user of the display device to have a particularly immersive experience during the ride.

[0021] According to a preferred embodiment of the method according to the invention, the control device can generate or transmit an interrupt signal only when the control device determines that an occupant, preferably an occupant different from the user of the display device, is speaking. For this purpose, for example, a directional microphone can be installed in the motor vehicle and connected to the control device.

[0022] Preferably, an insertion signal can additionally be generated or transmitted as a function of the determined volume of speech. Thus, the user only gets an overview of the interior when there is a high probability that other people want to speak to the user of the display device. While the user listens to the sound of the output content in his ears, he can see from the inserted image that he is being spoken to and decide whether he wants to take off his glasses briefly. Since in this embodiment of the method the content spoken by other passengers is not communicated to the user of the display device as the case may be, the auditory experience is not disturbed.

[0023] According to another preferred embodiment of the method according to the invention, the control device can generate an insertion signal or transmit an insertion signal to a display element of the display device only when the control device determines that the motor vehicle is approaching the driving destination of the motor vehicle (for example, the distance to the driving destination is less than a specified threshold or reaches specified geographical coordinates). Thus, the user is only "pulled back" from his virtual reality when the driver may soon want to speak to the user, for example, to pay for the journey or the user may now have to pack his belongings because he will soon reach the journey destination.

[0024] In one variant, the control device can generate an interrupt signal or transmit an interrupt signal to a display element of the display device only if the control device receives an activation signal from the motor vehicle, for example a signal indicating that the hazard warning light system has been switched on. Alternatively, if the control device receives a deactivation signal from the motor vehicle, at least one image of the interior conditions can be hidden.

[0025] Preferably, the control device can additionally receive a sensor signal from the sensor device, which describes at least one image of the external environment of the motor vehicle, which image can preferably be received from an external camera. The insertion signal generated by the control device can then describe this additional at least one image of the external environment, or the control device can generate another insertion signal describing at least one image of the external environment. The image of the external environment can then be inserted, for example, at a different position than the image of the interior. The user of the display device thus also has an overview of the surroundings of the motor vehicle. For this purpose, the sensor device can, for example, have a camera arranged on the outer shell or on the roof of the motor vehicle.

[0026] According to another embodiment, the control device can generate a map image signal, which can describe an image of a navigation map with, for example, the current position of the motor vehicle. This image can be displayed in addition to the image of the interior situation by the display device, for which purpose the control device can transmit the generated map insertion signal to a display element of the display device.

[0027] Alternatively, the map image signal can be output, for example, only in a predetermined time period, for example only at the start of driving and / or only at the end of driving, or only in a predetermined time period after the start of driving and before the end of driving. The interior image can be displayed only when the image of the navigation map is hidden.

[0028] The above-mentioned purpose is achieved by a control device, which is configured to execute one or more embodiments of the method according to the present invention. The control device can preferably have a processor device and / or a data memory. Here, the processor device is understood as a device component for electronic data processing, which has at least one microcontroller and / or a microprocessor. A program code for executing the method according to the present invention can preferably be stored on an optional data memory. The program code can then be designed to cause the control device to execute one of the above-mentioned embodiments of the method according to the present invention when running through the processor device. The control device can preferably be constructed as a control chip, a controller or an application ("App").

[0029] The above object is achieved by a motor vehicle having an embodiment of the control device according to the invention. The motor vehicle can preferably be designed as a motor vehicle, in particular a passenger car or a truck, or as a passenger bus or a motorcycle.

[0030] The above objects are achieved by a head-mounted display device having an embodiment of a control device according to the invention. A head-mounted display device, a so-called "Head-Mounted-Display", is a visual output device to be worn on the head. It is designed and arranged to either present images on a screen close to the eye or to project images directly onto the retina. The head-mounted display device can preferably be configured as virtual reality glasses, augmented reality glasses or mixed reality glasses.

[0031] The invention also relates to improvements of the control device according to the invention, the motor vehicle according to the invention and the display device according to the invention, which have the features already described in conjunction with the improvements of the method according to the invention. For this reason, corresponding improvements of the control device according to the invention, the motor vehicle according to the invention and the display device according to the invention are not described here.

[0032] The invention also includes combinations of features of the described embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In the following, an exemplary embodiment of the invention is described. To this end, the single FIGURE (“FIG”) shows a schematic diagram of a first exemplary embodiment of a method according to the invention and a device according to the invention. DETAILED DESCRIPTION

[0034] The embodiments described below are preferred embodiments of the present invention. In the embodiments, the components described in the embodiments respectively constitute the individual features of the present invention that can be considered independently of each other, and these features also improve the present invention independently of each other. Therefore, the present disclosure also includes the combination of the features of the embodiments except for the combinations shown. In addition, the embodiments can also be supplemented by other features in the features already described in the present invention.

[0035] In the figures, the same reference numerals respectively denote elements with the same function.

[0036] The figures illustrate the principle of the method according to the invention and the device according to the invention with the aid of a first embodiment. To this end, the figures show a motor vehicle 10, for example a car of a ride-hailing service. In the example of the figures, the control device 12 can be, for example, a controller or a control chip of the motor vehicle 10, but can alternatively also be installed in a head-mounted display device 14. The head-mounted display device 14 can, for example, have a display screen as a display element (not shown in the figures).

[0037] The communication with the display device 14 takes place via a data communication connection 16. If, for example, the control device 12 is installed in the display device 14, the data communication connection 16 can be a wired data communication connection, such as a cable or a data bus. If the control device 12 is an equipment component of the motor vehicle 10, the data communication connection 16 for communication between the control device 12 and the display device 14 can also be a wired data communication connection 16, such as a cable, but preferably a wireless data communication connection 16, such as a WLAN connection, a mobile radio connection, an Internet connection or a Bluetooth connection. The control device 12 in the figure shows an optional processor device 20 and an optional data memory 22.

[0038] If control device 12 is part of motor vehicle 10 , data communication connection 16 to sensor device 18 , which is shown in the figures as an interior camera, can be realized, for example, via a data bus of motor vehicle 10 , for example a CAN bus.

[0039] In the example in the figure, a user of an exemplary online ride-hailing service may have booked a motor vehicle 10 for a journey of, for example, 10 kilometers or half an hour. Entertainment with the aid of a head-mounted display device 14 may, for example, be an additional service to such a driving service. The user may bring his own display device 14, or the display device 14 may also be provided by the driving service. During the journey, for example, an entertainment product may be provided, which may describe a journey into a virtual reality during the journey, in particular a 360-degree virtual reality. The output content may, for example, be a Jurassic Park virtual reality, i.e., an entertainment describing a virtual journey through a pristine landscape.

[0040] For example, the output of the specified output content can start at the beginning of the journey. In method step S1, the control device 12 receives at least one sensor signal, which ideally describes the interior from multiple interior cameras of the motor vehicle 10. Although only one camera of the sensor device 18 is shown by way of example in the figure, multiple cameras can ideally be arranged so that the interior is captured from different viewing angles, for example. The images or image sequences of the individual cameras can then be recorded either in one sensor signal or in multiple sensor signals.

[0041] The insertion signal generated in method step S2 can describe a camera image or camera images of a plurality of interior cameras. After the generated insertion signal has been transmitted (S3) to a display element of a display device 14, the display element is caused to insert (S4) one or more corresponding images, preferably in the user's peripheral field of vision. If the control device 12 is the control device 12 of the motor vehicle 10, the generated insertion signal is therefore also first transmitted (S3) to the display device 14.

[0042] The user can, for example, see other strange passengers and strange drivers according to an inserted image (or multiple images or image series inserted). If the driver, for example, wants to talk to him for some reason, such as asking questions, the user of the display device 14 sees that the driver is looking at him. Even if the user of the display device 14 also additionally listens to the audio content of the virtual reality of the display device 14, for example, using headphones, he can also see that the driver wants to talk to him. If the driver does not get the user's reply for a while and decides to shake the user's shoulder of the display device 14, this will not make the user feel defenseless and will not scare the user of the display device 14. In addition, the user of the display device 14 can also react earlier and avoid being shaken by the shoulder, for example, by taking off the display device 14 to talk to the driver.

[0043] In order to direct the attention of the user of the display device 14 to the inserted image not permanently but only in a specific situation, the control device 12 can generate (S2) an insertion signal or transmit (S3) only when the control device 12 determines (S5) that the driver has started to speak, for example, using a microphone (not shown in the figure). Optionally, a threshold value for the volume is additionally stored here, so that, for example, no image is inserted when the driver is only whispering to himself.

[0044] Alternatively or additionally, the interruption signal may be generated (S2) or the interruption signal generated may be transmitted (S3) only when the control device 12 determines (S6), for example, with the aid of navigation data and GPS data determined by a navigation device of the motor vehicle 10, that the motor vehicle 10 is, for example, only one kilometer away from the destination. If the interior signal is interrupted only then, the user of the display device 14 will know that he will soon arrive and can therefore better determine whether the disturbance he perceives in the interior of the motor vehicle 10 is due to the other unknown passengers slowly starting to pack up.

[0045] In summary, these examples show how a head mounted display device 14, preferably VR glasses, can be provided by the present invention.

[0046] In another example, when VR glasses are used in a motor vehicle 10, in addition to the main content, a (e.g., usually small) camera image can be inserted (S4) into, for example, a VR environment, which can convey the conditions in the vehicle interior and, if necessary, also the conditions outside the motor vehicle 10 to the VR user, so that the VR user always has a rough grasp of the current conditions in and around the motor vehicle 10.

Claims

1. A method for operating a head-mounted display device (14) when the head-mounted display device (14) is located in the interior of a motor vehicle (10), wherein: The display device (14) is configured to output a prescribed output content describing an entertainment product, wherein the control device (12): - receiving sensor signals from a sensor device (18), wherein the received sensor signals describe at least one image of the current interior conditions of the motor vehicle (10), - generating an insertion signal, the insertion signal describing the insertion of at least one image of the current internal space situation during the output of the prescribed output content, and - transmitting the generated insertion signal to a display element of a display device (14), thereby causing the display device (14) to insert at least one image of the current interior situation during the output of the specified output content, The control device (12) generates or transmits an interrupt signal only when the control device (12) determines that an occupant is speaking or only when the control device (12) determines that the motor vehicle (10) is approaching a destination, or the control device (12) generates or transmits an interrupt signal only when the control device (12) determines that an occupant is speaking and the control device (12) determines that the motor vehicle (10) is approaching a destination.

2. The method according to claim 1, wherein: The generated insertion signal describes the insertion of at least one image of the current interior situation in only a sub-region of the visual field of a user of the display device (14).

3. The method according to claim 2, wherein: The sub-area is the central field of view.

4. The method according to any one of claims 1 to 3, wherein: The received sensor signal describes at least one camera image of an interior camera of the sensor device (18).

5. The method according to any one of claims 1 to 3, wherein: The control device (12) receives sensor signals from a sensor device (18) of the motor vehicle (10).

6. The method according to any one of claims 1 to 3, wherein: The specified output content describes a virtual reality.

7. The method according to claim 6, wherein: The specified output content describes a 360 degree virtual reality.

8. The method according to any one of claims 1 to 3, wherein: The occupant is a different occupant from a user of the display device (14).

9. The method according to any one of claims 1 to 3, wherein: The control device (12) generates or transmits an interrupt signal only when the control device (12) determines that the occupant is speaking and additionally as a function of the determined speaking volume.

10. A control device (12) configured to carry out the method according to any one of claims 1 to 9.

11. A motor vehicle (10) having a control device (12) according to claim 10.

12. A head-mounted display device (14) comprising a control device (12) according to claim 10.

Citation Information

Patent Citations

  • Methods for displaying virtual information in a view of a real environment

    DE102009049073A1

  • display additional content in a virtual scenery

    DE102014010309A1

  • Head - direction - dependent display of content on a data glasses

    DE102014213285A1

  • On-board head-up display system and vehicle comprising the same

    CN105785570A

  • Vehicular human machine interfaces

    CN106573540A