Method and device for operating data glasses in vehicle

Abstract

The invention relates to a method for operating a display system in a vehicle, having an auxiliary system fixed to the vehicle and having data glasses, comprising the following steps: monitoring whether a display surface has been placed from a parking position into a use position in which the display surface faces the data glasses, the display surface is a dark surface or a black surface, in particular an unstructured surface; transmitting activation information from the auxiliary system to the data glasses if it is identified that the display surface has been placed in the use position; and automatically starting and displaying the media information in the data glasses according to the starting information.

Description

Technical Field

[0001] The present invention relates to the use of augmented reality data glasses in vehicles, in particular for simplifying the display of image or video information.

[0002] The present invention relates to augmented reality (AR) data glasses, and more particularly to a method for operating AR data glasses to display media content in a virtual presentation window. Background Art

[0003] Augmented reality data glasses, also known as head-mounted displays, are known. They can display images on one or two display surfaces in the field of view of the wearer of the data glasses using a display device. The display surfaces can be reflective surfaces that direct the images into the eyes of the wearer. The viewing aperture of the data glasses is transparent, so that the real environment can also be perceived in a conventional manner through the display surface of the data glasses. The display surface is arranged in the viewing aperture so that the information to be displayed (e.g., text, symbols, graphics, video displays, etc.) can be displayed superimposed on the perceived environment.

[0004] The information can be presented to the wearer of the data glasses in a contact-simulated manner, i.e., in the following manner: the information is superimposed as a display object on a specific corresponding environmental object or a specific position or a specific position area in the real environment, or is oriented according to a specific corresponding environmental object or a specific position or a specific position area in the real environment, or the display object to be displayed is displayed in a specific orientation / orientation of the data glasses or the line of sight of their wearer.

[0005] However, in order to display objects on the display surface of data glasses in a contact-simulating manner, it is necessary to know the position of environmental objects in the reference environment and the user's gaze direction. When wearing data glasses, the user's gaze direction is fixedly assigned to their head posture or glasses posture (i.e., the six degrees of freedom (6DoF) corresponding to the 3D position and 3D orientation of the data glasses).

[0006] The eyeglass pose can be determined using an external pose recognition unit (outside-in tracking method). A vehicle interior camera captures the interior space and the head of the eyeglass wearer. The camera image is evaluated using a visual tracking method to determine the pose of the head and derive the eyeglass pose from it, or to determine the eyeglass pose directly by recognizing the shape of the eyeglasses. In these so-called outside-in tracking systems, the vehicle-fixed eyeglass pose is determined, i.e., relative to a vehicle coordinate system fixed to the vehicle.

[0007] Displaying media information, such as images or videos, on data glasses requires a dark viewing background because dark or black pixels of the displayed image appear transparent on the data glasses. This transparent display of black pixels allows for enhanced perception of surrounding objects when they overlap with the displayed object on the data glasses.

[0008] This can be disadvantageous when viewing image information or video information with dark image areas through data glasses, particularly when the viewing background is bright and / or non-uniform or structured. Therefore, to view the image information or video information, the wearer of the data glasses must seek out a dark background surface and maintain their gaze on the dark background while viewing the image information or video information, thereby allowing them to view the image information or video information without interruption. Summary of the Invention

[0009] The object of the present invention is to provide a comfortable option for viewing image or video information in data glasses.

[0010] This object is achieved by a method for operating a display system with data glasses in a motor vehicle according to claim 1 and by a display system according to the independent claim.

[0011] Further refinements are given in the dependent claims.

[0012] According to a first aspect, a method for operating a display system in a vehicle, comprising an assistance system fixed to the vehicle and data glasses, is provided, comprising the following steps:

[0013] - monitoring whether the display surface has been moved from the parking position into the use position, in which the display surface faces the data glasses, the display surface being a dark or black surface, in particular an unstructured surface;

[0014] If it is detected that the display surface has been placed in the use position, a start information is transmitted from the assistance system to the data glasses;

[0015] - According to the startup information, automatically start and display media information in the data glasses.

[0016] Additionally, the following steps can be set:

[0017] - monitoring whether the display surface has been moved from the usage position into the parking position in which the display surface is retracted;

[0018] If it is detected that the display surface has been placed in the parking position, a stop message is transmitted from the assistance system to the data glasses;

[0019] - According to the stop information, automatically stop displaying the media information in the data glasses.

[0020] As mentioned at the outset, the wearer of data glasses requires a dark, unstructured background surface to view image or video information, which is only available in a few areas of the vehicle. In particular, the vehicle may include a device that can be flipped open or otherwise brought into a use position to provide a dark background surface in a comfortable viewing direction. The data glasses can then be manipulated accordingly to display image or video information, which the user can view by directing their gaze toward the dark background surface. The display surface could, for example, be the back of a sun visor, which can be flipped down into the use position. The user can then activate the display of the image or video information using the data glasses and direct their gaze toward the corresponding display surface of the sun visor.

[0021] Provision can be made to monitor whether the display surface has been moved into the use position or the parking position by evaluating the camera image of the interior camera using gesture recognition, wherein the use position or the parking position is occupied by detecting a respectively specific hand gesture.

[0022] The above-described method provides for the use of an interior camera configured to track the posture of the data glasses or other vehicle applications, such as fatigue detection, in order to monitor the user's operating movements. This can be achieved, for example, using conventional methods for gesture recognition. If the interior camera and corresponding evaluation detect that the display surface is in a use position (e.g., by tilting the sun visor downward or upward, or otherwise manually moving a corresponding element on the dashboard), the corresponding information can be transmitted to the data glasses. This information can be used in the data glasses to automatically display image or video information, eliminating the need for the user to perform further interaction to trigger or activate functions in the data glasses.

[0023] Furthermore, if it is detected that the presentation surface has been placed in the use position, the media information to be displayed can also be transmitted from the assistance system to the data glasses in addition to the start information.

[0024] For example, the display surface can be located on the rear side of the tiltable sun visor.

[0025] It can be provided that the media information is displayed on the presentation surface in a contact-analog manner with respect to a vehicle-fixed coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The following further describes the implementation method with the help of the accompanying drawings.

[0027] Figure 1 A schematic diagram showing a display system in a vehicle;

[0028] Figure 2 A schematic diagram showing a display system; and

[0029] Figure 3 A flow chart is shown for illustrating a method for operating a display system. DETAILED DESCRIPTION

[0030] Figure 1 A schematic top view of a display system of a vehicle 1 is shown, wherein a vehicle occupant N is wearing data glasses 3. Data glasses 3 are connected to an auxiliary system 2 fixed to the vehicle via a communication connection 4. Communication connection 4 is designed as a data transmission channel, for example, in the form of a wireless or wired communication connection. Communication connection 4 enables the transmission of any type of data and information between the auxiliary system 2 and the data glasses 3, for example, via data transmission packaged in data packets. Communication connection 4 can be based on, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy, or similar standardized radio protocols.

[0031] Figure 2 A schematic diagram of a display system is shown.

[0032] The assistance system 2 can be arranged in a stationary manner in the vehicle 1 . The assistance system 2 can be equipped with a communication unit 23 , which enables data transmission via a communication connection 4 between the data glasses 3 and the assistance system 2 .

[0033] The interior space of the vehicle is continuously captured by means of the interior space camera 21 to provide a camera image of the vehicle interior space. The interior space camera 21 may include, for example, an RGB camera, an IR camera, a fisheye camera, a dynamic vision sensor, and the like.

[0034] The assistance system 2 may include a control unit 24 for analyzing the camera image using a pattern recognition method or a visual tracking method and thereby determining absolute eyewear position data of the one or more data glasses 3 in the vehicle. The eyewear position data represent the absolute eyewear position relative to the vehicle coordinate system.

[0035] With the aid of the method for sensor data fusion, the absolute glasses pose data can be used to provide glasses poses for use in data glasses for contact-simulated presentation of display objects.

[0036] The control unit 24 can also analyze the camera image using a visual tracking method and recognize and identify the user's hand gestures. In particular, the control unit 24 can be configured to recognize a gesture with which the user moves the display surface for viewing image or video information from a parking position to a use position, in which the display surface is easily and conveniently visible, or from a use position to a parking position, in which the display surface is retracted and preferably not visible to the user.

[0037] A possible display surface can be provided on the rear side of the sun visor 11 (the side facing the user in the usage position), which display surface is embodied as dark or black and preferably has no visible structuring.

[0038] Furthermore, the assistance system 2 includes a vehicle motion sensor system (IMU sensor system) 25, which can be designed, for example, in the form of a 6-DOF inertial sensor and provides motion data of the vehicle 1 in the form of translational acceleration or angular acceleration or angular velocity.

[0039] The data glasses 3 comprise two transparent viewing windows 32, which are embedded in a known manner in a frame 31. The frame 31 is provided with temples 33, for example, so that the data glasses 3 can be worn on the head of a user.

[0040] Furthermore, one or both viewing windows 32 (spectacles) are provided with a transparent display surface 35, through which a display image for presenting a virtual display object can be projected into the eyes of the wearer of the data glasses 3 by a suitable device, such as a display device 36 arranged on the frame 31. The display device 36 can include a microprocessor or similar computing unit and a display unit, such as a projection device or the like. The display unit can be designed to direct an electronically generated display image onto the display surface 35 and image / present it there.

[0041] Due to the transparent design of the display surface 35, electronically generated images can be superimposed on the real environment that can be perceived through the display surface 35. Virtual display objects, such as texts, symbols, media content, such as image or video information, graphics, etc., can be displayed on one or both display surfaces 35 using the display device 36.

[0042] The data glasses 3 can be worn on the user's head like a typical visual aid device, wherein the data glasses 3 can be placed on the user's nose with the frame 31 and the temples 33 can be placed laterally against the user's head. The user's line of sight is then directed in a straight line through the transparent display surface 35 of the viewing window 32, so that the user's line of sight, which is predetermined by the eye position and the optical visual axis (ocular axis), has a fixed relationship with the position and orientation of the data glasses 3 (i.e., the eye posture).

[0043] To display a display object, corresponding object information in the form of object data is transmitted from the assistance system 2 to the data glasses 3 or is already available in the data glasses 3. The object data specifies the type of display object (e.g., a text object, an icon, or another designation of a display area), one or more viewing areas in which the display object is to be displayed on the display surface 35, and the object position in the environment of the data glasses 3. The object position can be specified with respect to a world-fixed environment coordinate system or a vehicle-fixed vehicle coordinate system.

[0044] A glasses motion sensing mechanism 38 (IMU sensing mechanism, inertial sensing mechanism) can be provided, which is configured, for example, in the form of a 6-DOF inertial sensor. It provides relative or differential glasses motion information of the movement of the data glasses 3 in the form of translational acceleration and angular acceleration or angular velocity. This glasses motion information can be converted into position changes and orientation changes, i.e., differential glasses pose, through corresponding (e.g., quadratic) integration. This allows the glasses pose data to be updated at a high frequency using a time integration method. The absolute glasses pose data determined by the internal space camera can be used for sensor data fusion in the data glasses to compensate for integration errors in the absolute glasses pose determined in the data glasses based on the differential glasses pose.

[0045] The relative or differential eye movement information in the form of position and orientation changes determined by the eye movement sensor system 38 can be used in a known manner to determine world-fixed eye position data for the current eye position of the data glasses 3. To determine vehicle-fixed absolute eye position data, the eye movement information must be corrected based on the (also relative or differential) vehicle movement information so that only the movement of the data glasses 3 in the vehicle can be evaluated. The movement difference between the movement of the data glasses and the movement of the vehicle is determined and evaluated to determine the eye position.

[0046] By means of the control unit 37, for example, object information can be received from the assistance system 2 via the communication device 39 or provided and processed in another way, so that the object information is displayed in a contact-simulated manner in the corresponding viewing area according to the glasses' posture with respect to the environmental coordinate system, and the user of the data glasses 3 looks into the viewing area and can perceive environmental objects therein. That is, when the environmental position of the display object is within the viewing area, the display object given by the object information is displayed on the display surface 35. The media content is presented in a display window. The display window is an area, preferably a rectangular area, which only fills a part of the display surface and can be fixedly positioned in the vehicle, and the area is then displayed in a contact-simulated manner at a specific position in the vehicle. Alternatively, the display window can also be fixedly positioned on the display surface of the data glasses.

[0047] The user N of the data glasses 3 is in the vehicle 1 as a passenger, and the glasses posture of the data glasses 3 can change due to head movements and vehicle movements. This glasses posture is continuously and in real time determined in a manner known per se by means of the above-mentioned sensor mechanism. In order to present a display object on the display surface of the data glasses 3 in a contact-simulated manner relative to an environmental position (i.e., the corresponding object information specifies an environmental position), the data glasses 3 must have absolute glasses posture data relative to the environmental coordinate system. Accordingly, in order to present a display object on the display surface of the data glasses 3 in a contact-simulated manner relative to a position fixed to the vehicle (i.e., the corresponding object information specifies a position relative to the vehicle), the data glasses 3 must have absolute glasses posture data relative to the vehicle coordinate system.

[0048] Based on the user's operation, the data glasses 3 now receive information from the assistance system 2 about display objects, such as media content, which should be displayed in a presentation window that is fixed relative to the vehicle. The position and orientation of the presentation window can be freely selected by the user N of the data glasses 3, and is generally selected in the direction of the view toward the presentation surface.

[0049] exist Figure 3 A method for operating a display system with data glasses is described in more detail with the aid of a flow chart.

[0050] To this end, in step S1, an image of the vehicle interior is recorded using interior camera 21 and the movements of user N wearing data glasses 3 are monitored. This is achieved using a visual tracking method, wherein the movements or gestures of the user's arms (particularly the user wearing data glasses 3) are observed.

[0051] In step S2, a check is performed to determine whether a user action or hand gesture has been detected, which would move the display surface from its rest position to its use position. This display surface could be located, for example, on the back of the sun visor 11 or in the form of a hinged surface, such as on the upper side of the dashboard. The monitored arm movement or gesture of user N of the data glasses 3 corresponds to extending the arm with a corresponding movement, which could correspond to flipping the display surface downward into its use position, for example. If a corresponding arm movement of the user is detected (yes), in step S3, media information, i.e., image information or video information, is transmitted as object information to the data glasses 3, along with a start message that signals the start of the display of the media information in the data glasses 3. In the case of video information, this is transmitted in the form of a data stream (streaming). Otherwise (no), the process returns to step S1.

[0052] After receiving the activation information, the media information is now displayed in the data glasses in step S4. Thus, the media information is automatically played after the display surface is placed in the use position, and the user only needs to direct his or her gaze towards the display surface in order to view the media information without disturbing background perception.

[0053] In step S5 , a check is performed to determine whether a user action or gesture has been detected, which moves the display surface from the use position to the parking position. The monitored arm movements or hand gestures of the user N of the data glasses 3 correspond to extending the arm with a corresponding movement, which can correspond to, for example, turning the display surface over to the parking position.

[0054] If a corresponding movement of the user's arm is detected (selection: yes), in step S6, the transmission of the object information of the media information to the data glasses 3 is stopped, and a stop message is simultaneously transmitted, which signals the data glasses 3 to stop displaying the media information. In the case of video information, the transmission is performed in the form of a data stream (streaming). Otherwise, the process returns to step S1.

[0055] After receiving the stop information, in step S7, the output of the media information is now paused in the data glasses 3. Thus, the playback of the media information is automatically stopped after the presentation surface is placed in the parking position.

[0056] Reference Signs List

[0057] 1 vehicle

[0058] 11 sun visor

[0059] 2Assistance Systems

[0060] 3Data Glasses

[0061] 4 Communication Connection

[0062] 21 Interior Space Camera

[0063] 23 Communication Unit

[0064] 24 control unit

[0065] 25 Vehicle motion sensing mechanism (IMU sensing mechanism)

[0066] 31 Framework

[0067] 32 Observation window

[0068] 33 glasses legs

[0069] 35 display surface

[0070] 36 Display device

[0071] 37 Control Unit

[0072] 38 Eyeglass motion sensing mechanism

[0073] 39 Communication devices

Claims

1. A method for operating a display system in a vehicle, the display system having an auxiliary system (2) fixed to the vehicle and having data glasses (3), the method comprising the following steps: - monitoring (S2) whether the display surface has been moved from a parking position into a use position, in which the display surface faces the data glasses (3), the display surface being a dark or black surface, in particular an unstructured surface; - if it is detected that the display surface has been placed in the use position, a start information is transmitted (S3) from the auxiliary system (2) to the data glasses (3); - According to the startup information, the data glasses (3) are automatically started (S4) and the media information is displayed.

2. The method according to claim 1, wherein the method has the following additional steps: - monitoring ( S5 ) whether the display surface has been placed in a parking position in which the display surface is folded; - if it is detected that the display surface has been placed in the parking position, a stop message is transmitted (S6) from the assistance system to the data glasses (3); - According to the stop information, automatically stop (S7) the display of the media information in the data glasses (3).

3. The method according to claim 1 or 2, wherein: Monitoring whether a display surface has been moved into a use position or a parking position is performed by evaluating a camera image of an interior camera (21) using gesture recognition, wherein the use position or the parking position is occupied by recognizing a respectively specific gesture.

4. The method according to any one of claims 1 to 3, wherein: If it is detected that the display surface has been placed in the use position, the media information to be displayed is transmitted from the assistance system (2) to the data glasses (3) in addition to the start information.

5. The method according to any one of claims 1 to 4, wherein: The display surface is located on the rear side of the tiltable sun visor (11).

6. The method according to any one of claims 1 to 5, wherein: The media information is displayed in a contact-analog manner on or in front of the display surface with respect to a vehicle-fixed coordinate system.

7. A device for operating a display system, in particular an assistance system (2), with data glasses (3) in a vehicle, wherein: The device is configured to: - monitoring whether the display surface has been moved from a parking position into a use position, in which the display surface faces the data glasses (3), the display surface being a dark or black surface, in particular an unstructured surface; If it is detected that the display surface has been placed in the use position, a start message is transmitted (S3) from the assistance system (2) to the data glasses (3).

Images