Controlling display content of an augmented reality head-up display device of a motor vehicle

Abstract

The invention relates to a method, a computer-readable storage medium and a device for controlling the display content of an augmented reality head-up display of a motor vehicle. In a first step (10), the content to be displayed by the augmented reality head-up display is analyzed. In this case, the content to be displayed can optionally be prioritized (11). Subsequently, the position of the eye box of the augmented reality head-up display is adapted (12) according to the content to be displayed.

Description

[0001] The invention relates to a method, a computer-readable storage medium and a device for controlling the display content of an augmented reality head-up display of a motor vehicle. The invention also relates to a motor vehicle in which the method according to the invention or the device according to the invention is used.

[0002] As the technologies and applications of virtual reality and augmented reality continue to be further developed, they are also being used in cars. " refers to the enrichment of the real world by virtual elements, which are correctly recorded in three-dimensional space and allow real-time interaction. Because in the professional field within the German scope, the expression "Augmented Reality" is relative to the expression "erweiterte ” is generally accepted, so the first expression is used next. However, the expression “Mixed Reality” is also used as a synonym.

[0003] Head-up displays (HUDs) provide a feasible technical implementation for enriching the driver's work position with a perspective-correct virtual extension. Here, the light beam of the display installed in the dashboard is folded by multiple mirrors and lenses and reflected into the driver's eyes through a projection surface, so that the driver perceives a virtual image outside the vehicle. In the automotive field, the windshield is usually used as a projection surface, and the curved shape of the windshield must be taken into account when displaying. As an alternative, on the one hand, an additional plate made of glass or plastic is also used, which is arranged on the dashboard between the driver and the windshield. By visually superimposing the display content with the driving scene, less head and eye movements are required to read the information. In addition, the adaptive expenditure for the eyes is reduced, because less or even no adjustment of vision is required depending on the virtual distance of the display content.

[0004] Augmented reality offers a wide range of application possibilities to assist drivers by simulating the marking of lanes and objects. Relatively recent examples are mostly in the field of navigation. Conventional navigation displays usually show schematic views in conventional head-up displays, such as an arrow extending at a right angle to the right as a sign that you should turn right at the next opportunity, while AR displays offer much more effective possibilities. Since the display content can be displayed as "part of the environment", for example, navigation instructions or hazard warnings can be presented to the driver very effectively directly at the real reference position.

[0005] The display area of ​​the head-up display that can display virtual content in the windshield is described by the field of view or Field of View (FOV). The area from which the display content can be seen is called the eyebox. The field of view describes the extension of the virtual image in the horizontal and vertical directions in angles and is essentially limited by the available structural space in the vehicle. In conventional technology, a field of view of about 10°x4° can be achieved. The limited size of the field of view leads to the fact that in many cases, the main display content cannot be displayed or can only be displayed in a limited manner in augmented reality applications.

[0006] A first approach to solving this problem is to increase the size of the field of view, for example by using alternative display technologies. For example, by using holographic components, a larger field of view can be achieved while maintaining the same or even reducing the structural volume.

[0007] In this connection, US2012 / 0224062 A1 describes a head-up display for a motor vehicle. The head-up display uses a laser-based imaging system for virtual images. The imaging system includes at least one laser light source coupled to an imaging optical device in order to provide a light beam with a two-dimensional virtual image. An optical waveguide for widening the exit pupil is optically coupled to the laser-based virtual imaging system in order to receive the light beam and enlarge the eye box of the head-up display for viewing the virtual image.

[0008] Another solution is to adapt the position of the eyebox to the position of the driver's head, so that the driver's eyes are located in the center of the eyebox and effectively provide the largest possible field of view.

[0009] Against this background, DE 10 2015 010 373 A1 describes a method for adapting the position of a virtual image of a head-up display of a motor vehicle to a user's visible area. The head-up display has an adjustable housing which, in response to an operating action by the user, is brought into a desired position in which the virtual image is in the user's visible area.

[0010] In combination with head tracking, it is also possible to implement an individualized and even automatic adjustment of the eye frame according to the driver's head position. Here, dynamic movements of the driver can also be compensated by continuous adaptation or adjustment of the eye frame.

[0011] The object of the present invention is to provide an alternative solution for controlling the display content of an augmented reality head-up display of a motor vehicle, which solution is able to reduce the disadvantages caused by the limited size of the field of view.

[0012] This object is achieved according to the invention by a method for controlling display contents of an augmented reality head-up display of a motor vehicle, by a computer-readable storage medium, by a device for controlling display contents of an augmented reality head-up display of a motor vehicle, and by a motor vehicle having an augmented reality head-up display.

[0013] According to a first aspect of the present invention, a method for controlling display content of an augmented reality head-up display of a motor vehicle comprises the steps of:

[0014] - analyzing content to be displayed via the augmented reality head-up display; and

[0015] - Adapting the position of the eyebox of the augmented reality head-up display according to the content to be displayed.

[0016] According to another aspect of the present invention, a computer program is stored on a computer-readable storage medium, wherein the computer program includes instructions, and when the instructions are executed by a computer, the computer implements the following steps to control the display content of an augmented reality head-up display of a motor vehicle:

[0017] - analyzing content to be displayed via the augmented reality head-up display; and

[0018] - Adapting the position of the eyebox of the augmented reality head-up display according to the content to be displayed.

[0019] The term computer is to be understood broadly here. A computer also includes, in particular, control units and other processor-based data processing devices.

[0020] The computer program may, for example, be made available for electronic retrieval.

[0021] According to another aspect of the present invention, a device for controlling display content of an augmented reality head-up display of a motor vehicle comprises:

[0022] - an analysis module for analyzing content to be displayed by the augmented reality head-up display; and

[0023] - A control module for adapting the position of the eye box of the augmented reality head-up display according to the content to be displayed.

[0024] Due to the optical design of the head-up display, the virtual image can only be recognized when the eye of the observer is in a defined eye box. In the solution according to the invention, the disadvantages caused by the limited size of the field of view are reduced in that the position of the eye box is adjusted according to the virtual content. Therefore, no technology for enlarging the virtual image is required, but instead the limited image area is optimally utilized. Therefore, the solution according to the invention can be realized cost-effectively and does not require the installation space required for adjusting the head-up display.

[0025] According to one aspect of the present invention, the eye movement frame of the augmented reality head-up display is moved vertically according to the content to be displayed. The virtual image rendered for display in the head-up display should always be one buffer larger than the image that the head-up display can display. If, for example, it is assumed that the head-up display has a vertical field of view of 4° and a buffer of 0.5° above and below the image boundary, respectively, the image should be rendered for a field of view of 5°. By moving the vertical position, the field of view is dynamically widened by the range of the buffer. Of course, as an alternative or supplement, horizontal movement of the eye movement frame can also be achieved when buffers are set on the right and left sides of the image boundary.

[0026] According to one aspect of the invention, the position of the eye frame is adapted by adjusting the optical components of the augmented reality head-up display. In many head-up displays, it is already provided that the eye frame can be moved in a vertical orientation by adjusting a mirror in the optics of the head-up display. This serves to adapt the position of the eye frame to the position of the head of the observer. This adjustment possibility can now also be used to adapt the position of the eye frame depending on the content to be displayed. Therefore, no additional adjustment possibility is required.

[0027] According to one aspect of the present invention, when analyzing the content to be displayed by the augmented reality head-up display, the image rendered for the displayed content is evaluated. In particular, the color value of the image can be evaluated here. In order to adjust the eye movement frame in accordance with the situation, the rendered image can be dynamically evaluated. In order to display in the head-up display, an image with a black background is rendered in principle, because black appears transparent in the head-up display. Therefore, the buffer area can be automatically checked for pixels with RGB color values ​​other than (0,0,0). If the result of this test is positive, the eye movement frame is moved. If the content should be displayed in the upper buffer area, the eye movement frame is adjusted upward, and if the content should be displayed in the lower buffer area, the eye movement frame is adjusted downward. Of course, this color evaluation can also be implemented for other color spaces.

[0028] According to one aspect of the invention, when analyzing content to be displayed by an augmented reality head-up display, input data for rendering an image for the displayed content are evaluated. Since the adjustment of the eye frame is usually performed mechanically and is therefore accompanied by a large delay, it is expedient to perform the check of the content to be displayed predictively. Instead of evaluating an already rendered image, the evaluation is performed in this case already before the rendering based on the input data.

[0029] According to one aspect of the invention, the content to be displayed is prioritized. It is preferably ensured that the adjustment of the eye movement frame and the resulting display area of ​​the head-up display does not result in other content to be displayed not being able to be displayed. Therefore, the buffer area is checked in a suitable manner not only with regard to the content to be displayed but also with regard to the display area. The eye movement frame should only be adjusted to such an extent that other content of the rendered image does not fall out of the display area. If not all content to be displayed can be displayed in the display area, the content to be displayed is prioritized. In this way, it can be determined which content to be displayed is cut.

[0030] According to one aspect of the invention, the prioritization is dependent on the driving situation or can be influenced by a user of the augmented reality head-up display.

[0031] For example, it can be provided that when driving on a highway, navigation instructions are given less priority than instructions from a person, whereas when driving in a city, navigation instructions are given greater priority than instructions from a person. It is suitable here to distinguish between people at the edge of the road and people on the road, that is, important warnings about dangerous situations should always be given the highest priority. Preferably, the user of the head-up display can specify which virtual contents should be given priority, so that he can thus adapt the properties of the head-up display to his own preferences.

[0032] The method according to the invention or the device according to the invention is used particularly advantageously in a vehicle, in particular a motor vehicle.

[0033] Other features of the present invention will become apparent from the following description and appended claims taken in conjunction with the accompanying drawings. In the drawings:

[0034] Figure 1 Shows the approach to the intersection from the driver's perspective at a distance from the intersection;

[0035] Figure 2 Shows the approach to the intersection from the driver's perspective at a relatively close distance to the intersection;

[0036] Figure 3 A method for controlling display content of an augmented reality head-up display of a motor vehicle is schematically illustrated;

[0037] Figure 4 A first specific embodiment of a device for controlling display content of an augmented reality head-up display of a motor vehicle is shown;

[0038] Figure 5 A second embodiment of an apparatus for controlling display content of an augmented reality head-up display is shown;

[0039] Figure 6A motor vehicle is schematically shown in which the solution according to the invention is implemented;

[0040] Figure 7 The overall structure of an augmented reality head-up display for a motor vehicle is schematically shown;

[0041] Figure 8 showing tolerance ranges for different positions of a display area of ​​a head-up display and an adjacent eyebox;

[0042] Fig. 9 showing a turning situation which should be explained by means of an augmented reality display;

[0043] Fig.10 an augmented reality display showing the navigation marker without moving the eye box; and

[0044] Fig.11 An augmented reality display is shown with navigation markers having an eye box that moves according to the situation.

[0045] In order to better understand the principle of the present invention, the embodiments of the present invention are described in detail below according to the accompanying drawings. It is self-evident that the present invention is not limited to these embodiments and the features described can also be combined or modified without leaving the scope of the present invention defined in the attached claims.

[0046] Figure 1 The approach to the junction is shown from the driver's perspective at a distance from the junction. The augmented reality head-up display shows, on the one hand, a navigation marker 60 that simulates contact, in this case in the form of a driving carpet, and, on the other hand, an object marker 61 that simulates contact, in this case in the form of a frame around a person. In addition, two different fields of view 62, 62' are shown, the larger field of view 62' corresponding to an angular range of 20°x10° and the smaller field of view 62 corresponding to an angular range of 10°x4°. At this distance, virtual content can be displayed without any problems for both sizes of the fields of view 62, 62'.

[0047] Figure 2 The approach to the intersection is shown from the driver's perspective at a relatively close distance to the intersection. At this distance, the display of the contact-simulated navigation marker 60 and the display of the contact-simulated object marker 61 are severely clipped by the smaller field of view 62. The navigation marker 60 itself is barely recognizable. This effect reduces the added value and user experience of the augmented reality head-up display.

[0048] Figure 3A method for controlling the display content of an augmented reality head-up display of a motor vehicle is schematically shown. In a first step 10, the content to be displayed by the augmented reality head-up display is analyzed. For example, an image rendered for the display content can be analyzed, in particular its color values. Alternatively, input data for rendering an image for the display content can also be evaluated. In addition, the content to be displayed can optionally be prioritized 11. The prioritization can be related to the driving situation or can be influenced by the user. Next, the position of the eye frame of the augmented reality head-up display is adapted 12 according to the content to be displayed. Preferably, the eye frame is moved at least vertically. For example, the position of the eye frame can be adapted 12 by adjusting the optical components of the augmented reality head-up display.

[0049] Figure 4 A simplified schematic diagram of a first embodiment of a device 20 for controlling the display content of an augmented reality head-up display of a motor vehicle is shown. The device 20 has an input 21, through which, for example, image data of a camera 43, data of a sensor system 44 or data of a navigation system 45 can be received. The sensor system 44 can, for example, have a laser scanner or a stereo camera for detecting objects in the surroundings of the motor vehicle. In addition, the device 20 has an analysis unit 22, which can analyze the content to be displayed by the augmented reality head-up display, especially in terms of the displayability of the content in the display area of ​​the augmented reality head-up display. The analysis unit 22 can be configured, for example, to evaluate an image rendered for the display content, especially the color value of the image. Alternatively, input data for rendering an image for the display content can also be evaluated. In addition, the analysis unit 22 can prioritize the content to be displayed. The prioritization can be related to the driving situation or can be influenced by the user. Finally, the control module 23 realizes the adaptation of the position of the eye frame of the augmented reality head-up display according to the content to be displayed. Preferably, the eye frame is moved at least vertically. For example, the position of the eye box can be adapted by adjusting the optical components of the augmented reality head-up display. The control signal of the control module 23 can be output via the output 26 of the device 20, for example to the control device 42 of the augmented reality head-up display.

[0050] The analysis unit 22 and the control module 23 can be controlled by the control unit 24. The settings of the analysis unit 22, the control module 23 or the control unit 24 can be changed as necessary through the user interface 27. The data generated in the device 20 can be stored in the memory 25 of the device 20 when necessary, for example, for subsequent evaluation or for use by the components of the device 20. The analysis unit 22, the control module 23 and the control unit 24 can be implemented as dedicated hardware, for example, as an integrated circuit. But they can also be partially or completely combined or implemented as software, which runs on a suitable processor, such as a GPU. The input end 21 and the output end 26 can be implemented as a separate interface or a combined bidirectional interface. In the example described, the device 20 is an independent component. But the device can also be integrated in the control device 42 of the augmented reality head-up display device.

[0051] Figure 5 A simplified schematic diagram of a second embodiment of a device 30 for controlling the display content of an augmented reality head-up display of a motor vehicle is shown. The device 30 has a processor 32 and a memory 31. The device 30 is, for example, a computer or a control device. Instructions are stored in the memory 31, which, when executed by the processor 32, cause the device 30 to implement the steps according to one of the methods. Therefore, the instructions stored in the memory 31 embody a program that can be executed by the processor 32, which implements the method according to the present invention. The device 30 has an input 33 for receiving information, such as navigation data or data about the surroundings of the motor vehicle. The data generated by the processor 32 is provided via an output 34. In addition, the data can be stored in the memory 31. The input 33 and the output 34 can be integrated into a bidirectional interface.

[0052] Processor 32 may include one or more processor units, such as a microprocessor, a digital signal processor, or a combination thereof.

[0053] The memory 25 , 31 of the described embodiment may have both volatile and non-volatile storage areas and include different storage devices and storage media, for example a hard disk, an optical storage medium or a semiconductor memory.

[0054] Figure 6A motor vehicle 40 in which the solution according to the invention is implemented is schematically shown. The motor vehicle 40 has an augmented reality head-up display 41, which has an associated control device 42. In addition, the motor vehicle 40 has a device 20 for controlling the display content of the augmented reality head-up display 41. The device 20 can of course also be integrated in the augmented reality head-up display 41 or in the control device 42 of the augmented reality head-up display 41. Other components of the motor vehicle 40 are a camera 43 and a sensor system 44 for detecting objects, a navigation system 45, a data transmission unit 46 and a series of auxiliary systems 47, one of which is shown as an example. With the help of the data transmission unit 46, a connection with a service provider can be established, for example for retrieving map data. For storing data, there is a memory 48. The data exchange between the different components of the motor vehicle 40 is realized via a network 49.

[0055] Figure 7 An augmented reality head-up display 41 of a motor vehicle 40 is schematically shown, by means of which content can be displayed on a projection surface 53 of the motor vehicle 40, for example on the windshield or on an additional plate made of glass or plastic, which is arranged on the dashboard between the driver and the windshield. The displayed content is generated by an imaging unit 50 and projected onto the projection surface 53 by means of an optical module 51. In this case, it is usually projected onto the area of ​​the windshield above the steering wheel. With the help of an optical component 52 of the optical module 51, the position of the eye box of the augmented reality head-up display 41 can be adapted. The imaging unit 50 can be, for example, an LCD-TFT display. The augmented reality head-up display 41 is usually installed in the dashboard of the motor vehicle 40.

[0056] The following should be based on Figures 8 to 11 Preferred embodiments of the present invention are described.

[0057] Figure 8 A field of view 62 of a head-up display and adjacent tolerance ranges 63 for different positions of the eye box are shown. Figure 88a) indicates the middle position of the eye frame, 8b) indicates the higher position of the eye frame and 8c) indicates the lower position of the eye frame. Based on the optical design of the head-up display, the virtual image can only be recognized when the observer's eyes are in the eye frame. By adjusting the optical device of the head-up display, for example, by mirror adjustment, this eye frame can be moved in a vertical direction. The available adjustment range is represented by a vertical double arrow and a rectangle displayed with dashed lines. Therefore, by adjusting the optical device, the vertical position of the field of view 62, that is, the top view angle (downward observation angle, that is, the angle of the line of sight axis relative to the road) is defined. If the eye frame is adjusted too high or too low for the driver, the image of the display is cut at the upper edge or lower edge of the field of view 62. On the contrary, when it is adjusted correctly, the driver can fully recognize the image. In addition, a tolerance range 63 is formed above or below the field of view 62. In these ranges, the virtual image can also be recognized when the field of view 62 extends vertically to a greater extent.

[0058] Fig. 9 The turning situation that is to be explained by means of the augmented reality display is shown. The driving carpet is shown reflecting the direction of the turn. This is not actually enhanced by the augmented reality head-up display, but only a virtual display of the driving carpet that fully uses the driver's visual range.

[0059] Fig.10 FIG. 4 shows an augmented reality display of a navigation marker 60 without moving the eye frame. Fig. 9 The form of the navigation mark 60 of the driving carpet shown in FIG. 6 is gradually enhanced. Due to the small distance to the intersection, the view of the contact simulated navigation mark 60 is severely clipped by the field of view 62. The navigation mark 60 itself is almost not recognizable.

[0060] Fig.11 An augmented reality display of a navigation marker 60 with an eyebox that is moved according to the situation is shown. The eyebox is moved downwards to take into account the navigation marker 60 to be displayed, so that a significantly larger portion of the driving carpet can now be recognized. The display of the navigation marker 60 is also significantly improved without a larger vertical extension of the field of view 62. The display of the navigation marker 60 can also be improved additionally by increasing the field of view 62.

[0061] List of Reference Numerals

[0062] 10. Analyze the content to be displayed

[0063] 11. Prioritize content to be displayed

[0064] 12. Adapt the position of the eye frame

[0065] 20 Installation

[0066] 21 Input

[0067] 22 Analysis Module

[0068] 23 Control module

[0069] 24 Control Unit

[0070] 25 Memory

[0071] 26 Output

[0072] 27 User Interface

[0073] 30 Devices

[0074] 31 Memory

[0075] 32 processors

[0076] 33 Input

[0077] 34 Output

[0078] 40 Motor Vehicles

[0079] 41 Augmented Reality Head-Up Display

[0080] 42 Augmented reality head-up display control device

[0081] 43 Camera

[0082] 44 Sensor System

[0083] 45 Navigation System

[0084] 46 Data Transmission Unit

[0085] 47 Auxiliary systems

[0086] 48 Memory

[0087] 49 Network

[0088] 50 Imaging Units

[0089] 51 Optical Module

[0090] 52 Optical components

[0091] 53 Projection surface

[0092] 60 Navigation Markers

[0093] 61 Object Tagging

[0094] 62, 62' field of view

[0095] 63 Tolerance Range

Claims

1. A method for controlling display content of an augmented reality head-up display (41) of a motor vehicle (40), comprising the steps of: - analyzing (10) content (60, 61) to be displayed via the augmented reality head-up display (41); and - adapting (12) the position of the eyebox of the augmented reality head-up display (41) according to the content (60, 61) to be displayed, in, When analyzing (10) content (60, 61) to be displayed by an augmented reality head-up display (41), evaluating input data for rendering an image for said displayed content, Among them, the content to be displayed (60, 61) is prioritized. wherein the eye frame of the augmented reality head-up display is moved vertically according to the content to be displayed (60, 61), In this case, the displayed content can be recognized only when the observer's eyes are in the eye box, wherein no technique for enlarging the virtual image is required, but rather a limited image area is utilized.

2. The method according to claim 1, wherein: The position of the eye box is adapted by adjusting the optical components (52) of the augmented reality head-up display (41).

3. The method according to claim 1, wherein: Color values ​​of an image rendered for the display are evaluated.

4. The method according to claim 1, wherein: The prioritization is dependent on the driving situation or can be influenced by a user of the augmented reality head-up display (41).

5. A computer-readable storage medium having a computer program stored thereon, the computer program having instructions which, when executed by a computer, cause the computer to implement the steps of a method for controlling the display content of an augmented reality head-up display (41) of a motor vehicle (40) as claimed in one of claims 1 to 4.

6. A device (20) for controlling display content of an augmented reality head-up display (41) of a motor vehicle (40), comprising: - an analysis module (22) for analyzing (10) content (60, 61) to be displayed by the augmented reality head-up display (41); and - a control module (23) for adapting (12) the position of the eyebox of the augmented reality head-up display (41) according to the content (60, 61) to be displayed, in, The analysis module is designed to evaluate input data for rendering an image for the displayed content when analyzing (10) content (60, 61) to be displayed via an augmented reality head-up display (41), Among them, the content to be displayed (60, 61) is prioritized. wherein the eye frame of the augmented reality head-up display is moved vertically according to the content to be displayed (60, 61), In this case, the displayed content can be recognized only when the observer's eyes are in the eye box, wherein no technique for enlarging the virtual image is required, but rather a limited image area is utilized.

7. A motor vehicle (40) having an augmented reality head-up display (41), characterized in that: The motor vehicle (40) has a device (20) according to claim 6 or is configured to carry out a method for controlling display contents of an augmented reality head-up display (41) according to one of claims 1 to 4.

Images