Method and device for operating an augmented reality data glasses for displaying media content in a vehicle

Abstract

The invention relates to a method for operating a display system for smart glasses comprising the following steps: - Providing media content to the smart glasses for display in a display window that represents a sub-area of ​​the display surface of the smart glasses; - Providing at least one image segment of a representation of a vehicle environment, which is captured using an environment detection system, in real time, wherein the image segment corresponds to a section of the representation that is determined by the current position of the glasses; - Displaying the display window in the smart glasses; - Partially displaying the image segment in at least one sub-area outside the display window.

Description

The invention relates to augmented reality (AR) smart glasses, and more particularly to methods of operating AR smart glasses to display media content in a virtual display window. The present invention further relates to measures for avoiding motion sickness (kinetosis) when using smart glasses in a vehicle.Augmented reality smart glasses, also called head-mounted displays, are known, which can display an image on one or two display surfaces in the field of view of the wearer of the smart glasses with the aid of a display device. The display surfaces may correspond to reflection surfaces that direct images into the eye of the wearer of the smart glasses. The viewing openings of the smart glasses are transparent, so that the real environment can also be perceived in the usual manner by the display surfaces of the smart glasses. The display areas are arranged in the viewing openings such that information to be displayed, such as text, symbols, graphics, video displays and the like, can be displayed superimposed on the perception of the environment.The information can be displayed contact-analog to the wearer of the smart glasses, i.e. it can be displayed such that the information is superimposed as display objects on a specific assigned surrounding object or a specific position or a specific spatial area in the real environment or is oriented at the same, or that the display object to be displayed is displayed in a specific orientation / orientation of the smart glasses or their wearer.When using AR and VR smart glasses for viewing media contents in a vehicle, a feeling of nausea (motion sickness or kinetosis) may occur due to a deviation of the visual perception from the perceived accelerations during the movements of the vehicle. Since media contents have to be viewed in an illustration window in AR data glasses and for optimum image display in front of dark, unstructured background areas, no or only slight visual perception of the movement of the vehicle takes place during the viewing of the media contents. However, the accelerations of the vehicle are detected via the sense of equilibrium. This leads predominantly to the increased development of motional sickness, i.e. a sense of nausea, in the case of vehicle occupants in the rear seat rows of a vehicle, since media contents are generally viewed in front of a viewing surface on the rear side of the seat back of the front vehicle seats, which viewing surface almost completely occupies the field of vision.It is an object of the present invention to reduce as much as possible the generation of motion sickness when viewing media contents in smart glasses in a vehicle.This object is achieved by the method for operating a display system with smart glasses in a vehicle according to claim 1 and by a display system for a vehicle according to the subordinate claim.Further embodiments are given in the dependent claims.As mentioned at the beginning, AR smart glasses allow the observation of media contents, such as a computer application, image information and video information. Due to the superimposition of the real environment by the media contents shown, it is necessary to view these media contents against a dark background, which is generally provided by suitable viewing surfaces in the interior of the vehicle. The background is that dark or black pixels of the display are transparently displayed in the smart glasses. For example, vehicle occupants on a rear row of seats of a vehicle may use the rear side of a seat back of front vehicle seats in order to view media contents accordingly. The display window for displaying the media contents is generally stationary with respect to a vehicle coordinate system, i.e. it is displayed contact-analogously to the position of the viewing surface, or it can also be displayed at a fixed position of the display surface of the smart glasses, while the viewing direction of the user is directed at the viewing surface.During viewing, the corresponding display window in which the media contents are displayed in the smart glasses is contact-analogous or fixed and is not influenced by the movement during a trip. If the user of the smart glasses concentrates on the media contents with a fixed viewing direction on the viewing surface in the vehicle, a discrepancy arises between the perceived movement and the perception of the rigid display window. There is a risk that motional skill arises.In order to avoid this effect, the above method provides for complementing the display of the media contents on the display surface of the smart glasses with a display of the vehicle environment. In this case, a vehicle environment of the smart glasses is displayed on the display surface of the smart glasses around the display window for the media contents and / or in front of objects in the interior of the vehicle which are located in the viewing region of the user and which are located within a predefined distance from the user as a pictorial image (as a real image). The representation of the vehicle environment is displayed in real time for the user's direction of gaze. For this purpose, an image of the vehicle environment can be optically captured with the aid of an environment capture device. This image can be transformed perspectively according to the position of the user of the smart glasses in the vehicle interior and can be displayed accordingly on the display surface of the smart glasses with respect to one or more objects in the vehicle interior. For recognizing objects close to the user in the vehicle interior, a 3D detection device may be provided in the vehicle interior, for example, which detects the current position of objects close to the user, i.e., objects that are less than a predetermined distance from the user, and displays a representation of the environment of the vehicle in a contact-analogous manner at the corresponding position in the smart glasses. The contact-analogous representation means that the user perceives the vehicle environment from his viewing angle as if a clear view of the vehicle environment were possible.In this way, the objects close to the user are practically hidden and replaced by the representation of the vehicle environment. This increases the proportion of the area in the visible field of view for which the movement of the vehicle is matched both sensorially for the sense of equilibrium and visually, and thus reduces the risk of motional skill.In principle, it is possible to provide the representation of the vehicle environment in front of user-close objects. This corresponds, for example, in the case of vehicle occupants on a rear row of seats, to the backrest of the front vehicle seat, to an adjacent vehicle door, to a vehicle roof lining and the like.Alternatively or additionally, the presentation window of the media contents can be surrounded by an area in which the vehicle environment is overlaid, regardless of whether an object close to the user is located in this area or not.Embodiments are explained in more detail below with reference to the attached drawings. The following are shown: FIG. 1 is a schematic illustration of a display system in a vehicle for illustrating a virtual image of the rear vehicle environment; FIG. 2 shows a schematic representation of the display system, FIG. 3 is a flow chart illustrating the method of operating the display system; and FIG. 4 is a diagram of an exemplary display image of smart glasses for a vehicle occupant in a rear row of seats.FIG. 1 schematically shows a plan view of a display system having a vehicle 1, in which a vehicle occupant N wears smart glasses 3. The smart glasses 3 are connected to an assistance system 2 fixed to the vehicle via a communication connection 4. The communication connection 4 is designed as a data transmission channel, for example in the form of a wireless or a wired communication connection. The communication link 4 is capable of transmitting any type of data and information between the assistance system 2 and the smart glasses 3, for example based on a packet-bound data transmission. The communication connection 4 can be based, for example, on Wi-Fi, Bluetooth, Bluetooth Low Energy or a comparable standardized radio protocol.FIG. 2 is a schematic illustration of the display system.The assistance system 2 can be provided in particular in a stationary manner in the vehicle 1. The assistance system 2 can be equipped with a communication unit 23 which enables the data transmission via the communication link 4 between smart glasses 3 and the assistance system 2.The assistance system 2 can furthermore be connected to an environment detection system 21 which has a plurality of cameras 22 directed into the external vehicle environment. The cameras can preferably capture the entire rear vehicle environment in the form of camera images. The one or more cameras 22 may include, for example, an RGB, IR, Fisheye camera, a dynamic vision sensor, and the like.The assistance system 2 can have a control unit 24 in order to evaluate the camera image with the aid of a pattern recognition method and to determine an absolute spectacle pose specification of the one or more smart glasses 3 in the vehicle from this. The spectacle pose indication represents a spectacle pose related to a vehicle coordinate system.The control unit 24 can furthermore combine and / or render the simultaneously captured camera images to form an image of the rear vehicle region and provide them in real time.The assistance system 2 furthermore comprises a vehicle movement sensor system (IMU sensor system) 25, which can be designed, for example, in the form of a 6-DOF inertial sensor. This provides movement information of a movement of the vehicle 1 in the form of translatory accelerations or angular accelerations or angular speeds.The interior of the vehicle is continuously captured by means of an interior camera 26. By means of vision tracking methods, an absolute piece of spectacle pose information relating to the vehicle coordinate system, i.e. fixed to the vehicle, can be determined from the camera image of the interior camera 26. The absolute spectacle pose indication can be used with the aid of methods for sensor data fusion to perform the spectacle pose indication which is carried out in the smart glasses for a contact-analogous display of display objects.The smart glasses 3 comprise two transparent viewing panes 32, which are enclosed in a frame 31 in a manner known per se. The frame 31 is provided with eyeglasses frames 33, for example, so that the smart glasses 3 can be worn on the head of a user.One or both viewing panes 32 (spectacle lenses) are furthermore provided with a transparent display surface 35 through which a display image for the display of virtual display objects can be projected into the eye of the wearer of the smart glasses 3 by a suitable device, such as a display device 36 arranged on the frame 31. The display device 36 can have a microprocessor or a comparable arithmetic unit and a display unit, such as a projection device or the like. The display unit can be designed to direct the electronically generated display image onto the display surface 35 and to image / display it there.Due to the transparent configuration of the display surface 35, the electronically generated image can overlap the real environment perceptible by the display surface 35. Using the display device 36, a virtual display object such as a text, a symbol, a media content such as image information or video information, a graphic, or the like can be displayed on one or both display areas 35.The smart glasses 3 can be worn on the head of the user like a typical vision aid, wherein the smart glasses 3 can rest with the frame 31 on the nose of the user and the brackets 33 can rest laterally on the head of the user. The viewing direction of the user in the straight line orientation is then effected through the transparent display surfaces 35 of the viewing panes 32, so that the viewing direction and position of the user, which is predefined by an eye position and an optical viewing axis (eye axis), have a fixed relationship to the position and orientation of the smart glasses 3, i.e. to the eye pose.For displaying display objects, such as media contents and the like, corresponding object information in the form of object data is transmitted from the assistance system 2 to the smart glasses 3 or is already provided in the smart glasses 3. The object data indicate the type of display object, such as a text object, an icon or another identification of a display area, the viewing angle area or the viewing angle areas in which the display object is / is to be displayed on the display area 35 and the object position in the vicinity of the smart glasses 3. The object position can be specified with respect to a spatially fixed environment coordinate system or a vehicle-fixed vehicle coordinate system.A spectacle motion sensor system 38 (IMU sensor system, inertial sensor system) can be provided, which is designed, for example, in the form of a 6-DOF inertial sensor. This provides relative or differential piece of spectacle movement information of a movement of the smart glasses 3 in the form of translatory accelerations and angular accelerations or angular speeds, which can be converted in particular by a respective (inter alia double) integration into a change in position and orientation, i.e. a differential spectacle pose, so that an indication of an absolute spectacle pose can be updated with high frequency by applying a time integration method. The absolute spectacle pose indication determined with the aid of the interior camera is used for sensor data fusion in the smart glasses in order to compensate for integration errors of the absolute spectacle pose determined in the smart glasses on the basis of the differential spectacle poses.The relative or differential piece of spectacle movement information in the form of position and orientation changes ascertained via the spectacle movement sensor system 38 can be used in a manner known per se for determining a world-fixed piece of spectacle pose information for a current spectacle pose of the smart glasses 3. In order to determine an absolute spectacle pose indication fixed to the vehicle, the spectacle movement information must be corrected as a function of the vehicle movement information (likewise relative or differential), so that only the movement of the smart glasses 3 in the vehicle can be evaluated. In this case, the difference in movement between the movement of the smart glasses and the movement of the vehicle is determined and evaluated to determine the eye pose.By means of a control unit 37, object information can be received from the assistance system 2 or provided and processed in some other way via a communication device 39, for example, such that said object information can be displayed contact-analog with respect to the surrounding coordinate system or with respect to the vehicle coordinate system in the respective viewing angle range into which the user of the smart glasses 3 is looking and can be perceived in the surrounding object, depending on the spectacle pose. That is, the display object indicated by the object information is displayed on the display surface 35 when its associated surrounding position or virtual position in the vehicle is in the viewpoint range.Media contents as display objects are displayed in a display window associated with a virtual position in the vehicle. The display window represents a region, preferably a rectangular region, which fills only a partial region of the display area, and is correspondingly positioned fixed to the vehicle and is then displayed in a contact-analogous manner at a specific virtual position in the vehicle. Alternatively, the display window can also be positioned fixedly on the display surface of the smart glasses (fixedly secured to the glasses).The user N of the smart glasses 3 is located as an occupant in the vehicle 1 and the eye pose of the smart glasses 3 can change due to head movements and a movement of the vehicle. This spectacle pose is determined continuously and in real time with the aid of the sensor system described above in a manner known per se. For the contact-analogous display of a display object with respect to a surrounding position, i.e. the corresponding object information is provided with a surrounding position, an absolute piece of spectacle pose with respect to the surrounding coordinate system must be present in the smart glasses 3. Accordingly, for the contact-analogous display of a display object with respect to a position fixed to the vehicle, i.e. the corresponding object information is provided with a position relative to the vehicle, an absolute piece of spectacle pose with respect to the vehicle coordinate system must be present in the smart glasses 3.According to a user operation, the smart glasses 3 now receive from the assistance system 2 information about display objects, such as media contents, which are to be displayed in a display window to be displayed with respect to a position fixed to the vehicle. The position and orientation of the display window can be selected automatically or freely by the user N of the smart glasses 3 with respect to its position and is generally selected by the user in the direction of view toward the front, i.e. parallel to the vehicle longitudinal direction. For vehicle occupants on a rear row of seats in the vehicle 1, a customary position of the display window on the rear side of the backrest of a front vehicle seat is thus obtained. As a rule, this positioning of the display window results in the user not only perceiving the media contents in the display window but also in the rear side of the backrest passing the display window.Alternatively, the display window can also be displayed on the display surface of the smart glasses 3 in a non-contact-analogous manner. Since the user N directs the view forward during a trip, the user automatically positions the display window onto the viewing surface on the rear side of the backrest of the front vehicle seats.In particular, the perception of objects close to the user in the interior of the vehicle 1 frequently leads to the occurrence of motional sickness during a movement of the vehicle.Therefore, a method will be described with reference to the flowchart of FIG. 3 for reducing the perception discontinuity between the acceleration perception and the visual perception, and thus preventing the generation of motional sickness.In step S 1, an image of the vehicle environment is recorded by means of the environment detection system 21. The image of the vehicle environment is segmented according to a spectacle pose of the smart glasses 3, which is determined from the spectacle pose specification determined in the assistance system 2, in order to obtain an image segment of the vehicle environment, which the user N would see in the case of current spectacle pose if no vehicle parts or other objects were to block the view of the vehicle environment.A corresponding image segment, which corresponds to the viewing angle of the user of the smart glasses 3, is transmitted to the smart glasses 3 in step S 2.Furthermore, object information is transmitted to the smart glasses 3 in step S 3. The object information represents a display object that may include media content, such as a screen content of a computer or a mobile device, image information, and / or video information. The media content is assigned to a display window which, with respect to a vehicle coordinate system, is assigned to a specific position in the interior of the vehicle or to a specific position on the display surface of the smart glasses 3.In step S 4, structures in the vehicle interior are also further detected with the aid of the interior camera 26, and objects, surfaces and visually delimiting obstacles in the environment of the user of the smart glasses 3 are thus detected. In addition or optionally as an alternative to the interior camera 26, an interior scanning device can also be provided which recognizes objects which are located within a specific distance from the smart glasses 3 of the user N.Furthermore, in step S 5, the 3D information of the objects close to the user is transmitted to the smart glasses 3. The 3D information includes information about the spatial position and dimensions of the user-close objects in the vehicle interior with respect to the vehicle coordinate system.In the smart glasses 3, the display window is now displayed at the predefined position in step S 6 in a contact-analogous or glasses-fixed manner with the media content.Furthermore, in step S 7, the image segment corresponding to the current vehicle environment in the viewing direction of the user is displayed in the field of view of the user via the objects close to the user, if present, and / or in the environment of the display window. In this way, it is possible, in addition to the static display of the media contents in the field of view of the user, to provide a further surface region of the display surface with moving image information of the current vehicle environment in real time, which visually supports the movement impression of the vehicle. This is based on the fact that motion pitch can be avoided if a further surface region of the visible field of view is occupied with motion image information which corresponds to the movement impression of the sense of equilibrium of the user N.In the smart glasses 3, a display image can thus be generated, wherein the image segment of the vehicle environment is placed over the objects close to the user in the regions around the display window for the media contents and the image segment is not superimposed in the remaining regions. The objects close to the user correspond to all objects in the interior of the vehicle or the object parts thereof which are located within a predetermined distance of, for example, between 10 and 100 cm, preferably between 10 and 80 cm.Alternatively or additionally, an area around or outside the display window can be defined in which the image segment of the vehicle environment is displayed, since when viewing media contents, the gaze of the user of the smart glasses 3 is already directed at the display window and the latter thus preferably perceives the environment of the display window peripherally. In this way, a discrepancy between the user's visual perception and the perception of the movement of the vehicle via the sense of equilibrium is reduced, and motion sickness can be prevented.FIG. 4 shows, by way of example, a display image for the display area 35 of the smart glasses 3 when viewing a display window D having media contents M. It can be seen that the user of the smart glasses 3 looks at the rear side of a backrest 40 of a front vehicle seat, so that the display window D is displayed in the region of a dark area of the backrest. Furthermore, in areas outside the display window D, objects taken by users, such as the back, the image A of the vehicle environment is displayed in the corresponding viewing direction. Outside the visible objects close to the user, no output of the image segment takes place in a further region B of the display surface 35.List of reference characters1 Vehicle 2 Assistance system 3 Smart glasses 4 Communication link 21 Environment detection system 22 Camera 23 Communication unit 24 Control unit 25 Vehicle movement sensor system (IMU sensor system) 26 Interior camera 31 Frame 32 Viewing panes 33 Eye glass bracket 35 Display surface 36 Display device 37 Control unit 38 Eye glass movement sensor system 39 Communication device N User

Claims

Method for operating a display system for smart glasses (3), comprising the following steps: - providing media content to the smart glasses (3) for display there in a display window which represents a part of a display surface (35) of the smart glasses (3); - providing at least one image segment of a mapping of a vehicle environment, which is captured with the aid of an environment capture system (21), in real time, wherein the image segment corresponds to a section of the mapping which is determined by the current smart glasses pose; - displaying the display window (D) in the smart glasses (3); - displaying the image segment in at least one partial region (B) of the display surface (35) outside the display window (D).Method according to Claim 1, wherein the image segment is determined as a function of a spectacle pose indication which indicates the current spectacle pose of the smart glasses (3), wherein the spectacle pose indication is determined with the aid of a vision tracking method of a camera image of an interior camera (26) directed towards the vehicle interior, wherein the image segment corresponds to a representation of the vehicle environment which is / would be perceptible in the case of a current spectacle pose by the user of the smart glasses (3) in the case of a free field of view.Method according to claim 1 or 2, wherein a section of the image segment is displayed in an area around the display window (D) which corresponds to between 5% and 20% of the width and / or the height of the display window (D).Method according to one of Claims 1 to 3, wherein a section of the image segment is displayed in a region outside the display window (D) which overlays objects close to the user in the vehicle interior, wherein objects close to the user correspond to objects at a distance of between 10 and 100 cm from the user.Method according to Claim 4, wherein the objects close to the user are detected with the aid of one or the interior camera (26) or an interior scanning device.Method according to one of Claims 1 to 5, wherein the display window (D) is displayed contact-analogously with respect to a vehicle coordinate system fixed to the vehicle or fixed to the spectacles at a predetermined position on a display surface (35) of the smart glasses (3).Device, in particular assistance system (2), for a display system for smart glasses (3), wherein the device is designed to: - provide media content to the smart glasses (3) for display there in a display window (D) which represents a part of the display surface (35) of the smart glasses (3); and - provide at least one image segment of a mapping of a vehicle environment, which is detected with the aid of an environment detection system (21), in real time, wherein the image segment corresponds to a section of the mapping which is determined by the current spectacle pose.Display system for smart glasses (3), comprising: - a vehicle-mounted assistance system (2) which is designed to: ◯ provide media content to the smart glasses (3) for display there in a display window (D) which represents a part of the display surface (35) of the smart glasses (3); ◯ provide at least one image segment of a mapping of a vehicle environment to the smart glasses, which mapping is detected by means of an environment detection system (21), in real time, wherein the image segment corresponds to a section of the mapping which is determined by the current smart glasses pose; - smart glasses (3) which is designed to ◯ display the display window (D) in the smart glasses (3); and ◯ to display the image segment in at least one partial region (B) of the display area (35) outside the display window (D) in a section.

Images