TRAFFIC SITUATION-Dependent CONTROL OF A VIRTUAL REALITY DEVICE

Description

[0001] The invention relates to a method for controlling a VR device, in particular VR glasses, of a user in a vehicle interior of a vehicle via a communication connection and to a vehicle arrangement.

[0002] There are currently various theoretical applications for virtual reality headsets or VR devices in the automotive sector that are currently under development. For example, program libraries are known that allow developers of VR media formats to adapt their content to vehicle movements. By implementing vehicle movement into the output of the VR media formats, a more intense immersion of the VR device user can be achieved. Furthermore, the inclusion of vehicle movement can minimize the tendency towards so-called motion sickness.

[0003] When using a VR device in a vehicle, the user is shielded from events in and around the vehicle. While this enhances the immersive experience, it also detaches the VR device user from significant events occurring within the vehicle. For example, the user cannot directly perceive an impending accident situation in the vehicle or prepare for it. This can frighten or surprise the VR device user.

[0004] WO 2020 / 035380 A1 relates to a method and system for operating VR glasses during an autonomous journey of a vehicle, in which a vehicle drives autonomously, a driver of the vehicle wears VR glasses connected to a multimedia device, and the VR glasses display an entertainment program of the multimedia device for the driver.

[0005] US 2017 / 113702 A1 discloses a safety system for an autonomous vehicle that can be used to warn an occupant about a specific activity involving handling an object that could harm the occupant in the event of a sudden deterioration in driving smoothness (an SDRS event). The safety system includes a camera that captures images of the occupant and a computer that uses the images to estimate whether the occupant is engaging in a specific activity. The computer receives an indication from an autonomous driving control system as to whether an SDRS event is imminent. In response to receiving an indication of an impending SDRS event and estimating that the occupant is engaged in a specific activity, the computer instructs a user interface to warn the occupant shortly before the SDRS event occurs.

[0006] From DE 10 2013 222580 A1 a method for detecting whether a driver is wearing data glasses in a vehicle is known, in which a sensor system in the vehicle determines whether the driver is wearing data glasses and, in this case, a warning signal is generated.

[0007] It is the object of the present invention to provide a method and a vehicle arrangement by which a user of a virtual reality device is protected from being startled or surprised.

[0008] According to the invention, this object is achieved by a method having the features of claim 1 and a vehicle arrangement having the features of claim 7. Advantageous embodiments and further developments emerge from the dependent claims.

[0009] The method according to the invention serves to control a VR device, in particular VR glasses, of a user in a vehicle interior via a communication connection. In the method according to the invention, vehicle data is received and evaluated via the communication connection, or previously evaluated vehicle data is received. Based on the evaluated vehicle data, at least one traffic event is classified, wherein control commands for controlling the VR device are generated, by which the user of the VR device is informed of an unforeseeable reaction of the vehicle to the traffic event. According to the invention, control commands for controlling the VR device are generated, which signal a need for communication between a vehicle passenger and the user of the VR device.

[0010] The method can be used to forward specific information from the vehicle to the VR device and display it. For example, a data channel can be used between the vehicle and the VR device, which can be used for motion compensation when using the VR device. The data channel can be based on a wireless or wired communication technology, such as Wi-Fi, Bluetooth, NFC, radio, infrared, Ethernet, USB, and the like.

[0011] The VR device can be configured, for example, as a virtual reality headset. VR media content can be output via at least one display integrated into the virtual reality headset or via a portable device, such as a smartphone, inserted into the virtual reality headset. The VR device can also include headphones, camera sensors, microphones, and the like to optimize user immersion.

[0012] The communication connection can preferably be based on a wireless transmission standard, such as Bluetooth, and can be established between a vehicle-mounted control unit and the VR device. The data-conducting coupling of the VR device to the vehicle still allows the user to be provided with an immersive VR experience. The method according to the invention can inform the user about special incidents in and around the vehicle. The VR device can thus react to traffic events to prevent the user of the VR device from being startled. The method can thus increase the comfort of using VR media content while driving.

[0013] The at least one traffic event can be defined or classified based on vehicle signals or vehicle data. The vehicle data can also be provided indirectly to the user of the VR device in the vehicle to enable traffic-situation-dependent control of the VR device.

[0014] Traffic events, such as impending rear-end collisions or successful pedestrian detection, are registered by vehicle control units before the vehicle reacts. The corresponding evaluation results can be transmitted to the VR device in the form of evaluated vehicle data or directly as control commands. This allows vehicle-side entities that access the vehicle-side sensors to control the VR device. Alternatively or additionally, the vehicle data can be evaluated by the VR device.

[0015] According to one embodiment, the control commands for outputting a message on a display of the VR device, activating transparency of the VR device, outputting an acoustic or graphical notification, pausing or terminating an application of the VR device, and / or deactivating a noise suppression function are generated. This can generate a notification for the user of the VR device, preventing the user from being startled.

[0016] The message on the VR device's display can be in the form of a warning or a notification. Acoustic, haptic, and / or visual messages or notifications can be issued as responses to the control commands generated by the VR device.

[0017] In a further embodiment, the user of the VR device is notified of a vehicle reaction to the traffic event in the form of activation of an emergency braking assistant, an evasive maneuver by the vehicle, arrival or approaching a destination, triggered fatigue detection, or an object detected by vehicle sensors. In such traffic events, the vehicle can initiate a braking maneuver or an evasive maneuver in response, which is perceived by the user of the VR device suddenly and without warning. The method allows the user of the VR device to be considered and notified in advance of such reactions by the vehicle. Such control of the VR device enables advance warning or adjustment of the user for the impending reaction of the vehicle, thereby increasing driving comfort for the user.

[0018] If the vehicle's drowsiness detection system detects the driver as tired, the VR device user can be notified directly or indirectly, allowing them to, for example, contribute to keeping the vehicle driver awake and active. For example, the VR device user could be offered media offerings that require collaboration with another non-VR user, such as the driver or another vehicle passenger. Such VR media content could be designed, for example, as a quiz.

[0019] The user of the VR device can preferably be a vehicle passenger sitting in a rear seat or a passenger seat in the vehicle interior. The vehicle can be controlled automatically, semi-automatically, or manually by a driver. The user of the VR device is preferably excluded from controlling the vehicle.

[0020] In a further embodiment, the communication requirement of the vehicle passenger with the user of the VR device is initiated by an input from the vehicle passenger.

[0021] When communication is needed, at least one vehicle passenger can initiate an input via the vehicle's media control system to discreetly communicate this communication need to the VR device user. This can be implemented, for example, by activating a button in the graphical interface of the media control system, by pressing a key, or by using a voice-assisted vehicle assistant.

[0022] The VR device user can receive a notification via the communication link. The notification or indication of the need for communication can be displayed as a text message on the VR device, played as a voice message over headphones, or signaled by deactivating the headphones' noise cancellation. Unlike direct contact with the VR device user, for example, through touch, indirect communication via the communication link with the vehicle can prevent the VR device user from being startled.

[0023] According to a further embodiment, the control commands for controlling the VR device are generated by a vehicle-mounted control unit and / or by a processor unit of the VR device. This measure allows the calculation of the control commands and the classification of the at least one traffic event to be performed on the vehicle or by the VR device. In particular, calculations, warnings, and calculation results already performed by the vehicle can be used by the VR device to reduce the power consumption of the VR device and increase the battery life of the VR device.

[0024] According to a further embodiment, vehicle data and / or evaluated vehicle data are transmitted between the vehicle-side control unit and the VR device via a communication connection configured as a Bluetooth connection or a Wi-Fi connection. This measure allows a communication connection to be established based on different wireless transmission standards, which enables the transmission of vehicle data for driving dynamics-dependent VR experiences.

[0025] The communication connection can preferably be implemented as a bidirectional data channel between the vehicle or the vehicle-side control unit and the VR device. The corresponding communication connection can be based on one or more in-vehicle or external transmission technologies, such as Bluetooth, Wi-Fi, LTE, 5G, and the like.

[0026] The vehicle assembly according to the invention comprises a vehicle with a vehicle-mounted control unit and at least one VR device. The vehicle assembly is configured to carry out the method according to the invention.

[0027] The vehicle layout allows the use of VR devices, such as VR glasses or headsets, in a rear or front passenger seat of the vehicle, regardless of whether the ride is automated or manually controlled. The VR device user can be alerted to an unpredictable reaction of the vehicle to a traffic event, thus avoiding startling or surprising the user and increasing the ride comfort for the user.

[0028] The invention is illustrated schematically in the drawings using embodiments and will be further described with reference to the drawings. It shows: Figure 1a vehicle arrangement according to an embodiment of the invention.

[0029] Figure 1 shows a vehicle assembly 1 according to an exemplary embodiment of the invention. The vehicle assembly 1 comprises a vehicle 2, which can be configured as an automated or manually controlled vehicle. In a vehicle interior 4 of the vehicle 2, two vehicle passengers 6, 8 are located, for example.

[0030] A first vehicle passenger is a driver of the vehicle 2. A second vehicle passenger 8 wears a virtual reality headset or a VR device 10. The second vehicle passenger 8 is thus a user 8 of the VR device 10. The first vehicle passenger 6 is a so-called non-VR user.

[0031] A data-conducting communication connection 14, based, for example, on a Bluetooth transmission standard, is established between a vehicle-mounted control unit 12 and the VR device 10. Vehicle data can be received and evaluated, or evaluated vehicle data can be received, via the communication connection 14. The vehicle data can be determined or received, for example, by a vehicle sensor system 16. Additionally, route guidance information can be taken into account in the form of vehicle data.

[0032] The vehicle sensor system 16 may include, for example, camera sensors, LIDAR sensors, radar sensors and the like.

[0033] Based on the evaluated vehicle data, a traffic situation or traffic event can be classified by the vehicle-side control unit 12 or by a processor unit 11 of the VR device 10. Based on the traffic event, a reaction of the vehicle 2 to the traffic event can be estimated.

[0034] Critical driving situations, for example, can be classified as possible traffic events. In critical driving situations, a safe distance from vehicle 2 is exceeded, resulting in an evasive maneuver or emergency braking maneuver being initiated. Such driving maneuvers can be initiated automatically by the vehicle-mounted control unit 12 or by an emergency braking assistance function, or by the driver 6. The corresponding driving situation can often be determined in advance using the vehicle's sensor system 16.

[0035] After classifying the traffic event, control commands are generated to control the VR device 10 in order to alert the user 8 of the VR device 10 to an unpredictable reaction of the vehicle 2 or the driver 6 to the traffic event and to prevent the user 8 from being startled. For this purpose, a warning message can be displayed in the VR display of the VR device 10 using the generated control commands.

[0036] Depending on the design of the VR device 10, a see-through function can be activated by the generated control commands. To position the VR device 10 in space, some VR devices 10 have camera systems with which the surroundings are recorded and evaluated. A see-through function displays the video stream of such a camera system instead of the VR display, allowing the user 8 a view behind the VR device 10. The see-through function can thus be activated and deactivated depending on the traffic situation in order to prepare the user 8 for possible reactions of the vehicle 2.

[0037] In another exemplary traffic event, vehicle 2 may reach a destination. This may be recorded based on vehicle data in the form of navigation data. Upon approaching the destination, a corresponding notification may be displayed on VR device 10 so that user 8 can prepare for the end of VR use. LIST OF REFERENCE SYMBOLS:

[0038] 1Vehicle arrangement 2Vehicle 4Vehicle interior 6First vehicle passenger / driver 8Second vehicle passenger / VR device user 10VR device / virtual reality headset 11VR device processor unit 12Vehicle control unit 14Communication connection 16Vehicle sensors

Claims

1. A method for controlling a VR device (10), in particular VR glasses, of a user (8) in a vehicle interior (4) of a vehicle (2) via a communication link (14), wherein via the communication link (14) vehicle data are received and evaluated, or evaluated vehicle data are received, and, based on the evaluated vehicle data, at least one traffic event is categorized, and control commands for controlling the VR device (10) are generated, through which the user (8) of the VR device (10) is made aware of a reaction of the vehicle (2) to the event, the reaction being unforeseeable for the user (8), characterized in that control commands for controlling the VR device (10) signaling a need of a vehicle passenger (6) to communicate with the user (8) of the VR device (10) are generated.

2. The method according to claim 1, wherein the need of the vehicle passenger (6) to communicate with the user (8) of the VR device (10) is initiated by a vehicle-side input of the vehicle passenger (6).

3. The method according to claim 1 or 2, wherein the control commands are generated for outputting a message to a display of the VR device (10), activating a transparency of the VR device (10), outputting an acoustic or graphic notification, for pausing or ending an application of the VR device (10) and / or for deactivating a noise-canceling function.

4. The method according to any of claims 1 to 3, wherein the user (8) of the VR device (10) is made aware of a reaction of the vehicle (2) to the traffic event in the form of an activation of an emergency brake assistant, an avoidance maneuver of the vehicle (2), an arrival at or an approach to a destination, a triggered fatigue detection, or an object identified by a vehicle sensor (16).

5. The method according to any of claims 1 to 4, wherein the control commands for controlling the VR device (10) are generated by a vehicle-side control device (12) and / or by a processor unit (11) of the VR device (10).

6. The method according to claim 5, wherein vehicle data and / or evaluated vehicle data are transmitted between the vehicle-side control device (12) and the VR device (10) via a communication link (14) in the form of a Bluetooth link or WLAN link.

7. A vehicular arrangement (1), with a vehicle (2) having a vehicle-side control device (12) and at least one VR device (10), wherein the vehicular arrangement (1) is configured for executing a method according to any of the preceding claims.