Correct positioning of additional information on the vehicle display unit

Abstract

The invention relates to a method for generating a positionally correct display of additional information on a display unit (20) of a vehicle (10), wherein the vehicle (10) is equipped with a number of sensors (130, 150, 170) for detecting the state and surroundings of the vehicle (10). In the method, the data of the sensors (130, 150, 170) are processed cyclically. The method is characterized in that the state of the vehicle (10) is precalculated, wherein the precalculated state is transmitted to a rendering process for calculating an image to be displayed, wherein the position of the display of the additional information in the displayed image is calculated at least based on the precalculated state of the vehicle (10) during the rendering process. In this case, the state of the vehicle (10) is detected again during the rendering process. Therefore, the calculated image is then also post-processed, wherein the image content is transformed, wherein the transformation calculation is determined by the updated state of the vehicle (10).

Description

Technical Field

[0001] The present invention relates to the technical field of driver information systems, also known as infotainment systems. In particular, it relates to the use of a head-up display (HUD) unit, which can be used to display additional information in the driver's field of view. The invention also relates to a correspondingly designed device for carrying out the method and a correspondingly designed vehicle. Background Art

[0002] Research is currently focused on technologies that will enable automated driving in the future. A first approach is to ensure that the driver can take over control of the vehicle at any time, rather than completely relieving them of their duties. Furthermore, the driver is aware of monitoring functions. Newer technologies in the field of driver information systems, such as heads-up displays (HUDs) and data glasses, make it possible to better inform the driver of what is happening in the vehicle's surroundings.

[0003] Therefore, it is assumed that in the near future, comprehensive information about objects (especially vehicles) in the vehicle's visible and hidden / invisible surroundings will be available to systems using newer technologies (vehicle-to-vehicle communication, database utilization, backend connectivity, cloud services, server utilization, vehicle sensors, etc.). In the field of on-board sensors, the following components are mentioned as enabling environmental observation: RADAR devices, which correspond to radio detection and ranging, LIDAR devices, which correspond to light detection and ranging and are primarily used for distance detection / warnings, and cameras with corresponding image processing for object recognition. This environmental data can thus be used as a basis for system-based driving recommendations, warnings, and the like. For example, displays / warnings indicating which direction other surrounding vehicles are attempting to turn (possibly towards the vehicle's own trajectory) are conceivable.

[0004] Vehicle-to-vehicle communication can also be achieved via mobile communications using systems such as Long Term Evolution (LTE) or 5G. The 3GPP organization has adopted a specification called LTE PC5. Alternatively, systems based on WLAN technology, particularly WLANp, which corresponds to IEEE 802.11p, can be used for direct vehicle communication. Such systems are relevant to the areas of cooperative or autonomous driving.

[0005] Due to the current development towards higher levels of automated driving, in which many vehicles are still controlled by the driver, it can be assumed that the corresponding additional information can be used for manually controlled vehicles already in the medium term and not only for highly automated systems in the long term.

[0006] Here, for driver-vehicle interaction, the problem is how to display this information so that it creates real added value for the human driver and he can also quickly or intuitively locate the provided information. Here, the following solutions in this field are already known from the prior art.

[0007] The future vision in the automotive industry is to be able to display virtual elements on the windshield of one's vehicle in order to be able to realize certain advantages for the driver. So-called "augmented reality" (AR) or "mixed reality" (MR) technologies are used. The corresponding German term "erweiterten" is used for the purpose of rendering virtual elements on the windshield of one's vehicle. ” or “gemischten ” is less common. Here, the real environment is filled with virtual elements. This has several advantages: there is no need to look down at a display other than the windshield, because when looking through the windshield, a lot of relevant information is displayed. Therefore, the driver does not have to take his eyes off the road. In addition, the precise positioning of the virtual elements in the real environment allows for lower cognitive effort on the part of the driver, because there is no need to interpret graphics on a separate display. With regard to autonomous driving, added value can also be generated. In this regard, the article "3D-FRC: Depiction of the future roadcourse in the Head-Up Display" by CA Wiesner, M. Ruf D. Sirim and G. Klinker in the 2017 IEEE International Symposium on Mixed and Augmented Reality is mentioned, in which these advantages are explained in more detail.

[0008] Because current technological capabilities are limited, it can be assumed that fully realistic windshields will not be available in vehicles in the medium term. Head-up displays are currently used in vehicles. These vehicles also offer the advantage that HUD images appear closer to the real environment. The display is actually a projection unit that projects the image onto the windshield. However, from the driver's perspective, this image is located several to 15 meters in front of the vehicle, depending on the module's design.

[0009] A major advantage of previously known "augmented reality" displays (AR displays) is that they can be displayed directly within an environment or as part of it. A particularly prominent example relates to navigation. While classic navigation displays (in conventional HUDs) typically show schematic diagrams (e.g., a right-angled arrow indicating a right turn at the next location), AR displays offer significantly more efficient possibilities. Because the display can be shown as "part of the environment," it allows for very quick and intuitive interpretation for the user.

[0010] Such heads-up displays also have the disadvantage that the HUD can only be displayed within a segment of the driver's real field of view defined by the HUD's eyebox. A further problem is that when the driver turns or tilts his head, the segment in which the HUD is displayed moves out of the driver's central viewing area, making it more difficult for the driver to understand the displayed information.

[0011] To address these issues, there are approaches that can also detect the driver's head movements and position the HUD display accordingly. Another approach is to use data glasses for driver assistance systems. In these, the driver's entire field of view can be presented as a virtual display. Data glasses also need to detect head movements, and if necessary, eye movements, to correctly position the AR display relative to the surrounding environment.

[0012] DE 10 2013 016 247 A1 discloses a method for enhancing the display of at least one virtual additional information in a recorded digital image of a vehicle's surroundings, wherein the recorded image is output on a display as a background image. The additional information is output, in particular fixedly positioned / anchored, in the output image at a target location. A first, in particular rough, positioning method is used to determine the prepositioning of the additional information, and a second, in particular fine, positioning method is used to adjust the prepositioning.

[0013] DE 10 2014 208 973 A1 proposes also implementing user movement tracking (user tracking) in the HUD. This can be accomplished using sensors incorporated into the vehicle. Data regarding the user's movements can be used to correctly adjust the representation of AR objects. This can also improve overall latency, so that virtual objects 130 can be appropriately integrated into the environment from the perspective of the driver or passenger.

[0014] DE 10 2015 007 518 A1 discloses a method for predicting the position of a vehicle. Different sensor types are used. A first sensor type detects the position of the vehicle. A second sensor type detects the deviation of the vehicle body from the central position. Sensors 26 of a third sensor type are designed to detect the vehicle's surroundings, in particular to measure the contour of the road surface in front of the vehicle. Based on the vehicle's speed, which is known by the sensors of the first sensor type, it is possible to determine when a road surface irregularity is directly beneath the vehicle, so that this information can be advantageously used by prediction device 12 for the undelayed determination of position 30. Sensors of third sensor type 26 are, for example, stereo cameras, ultrasonic sensors, LIDAR systems, and / or laser triangulation sensors.

[0015] A head-up display for a vehicle is known from DE 10 2015 208 738 A1, wherein the head-up display is designed to compensate for relative movements between the driver's head and the head-up display, thereby reducing relative movements between a virtual image and the driver's head.

[0016] DE 10 2017 001 015 A1 discloses a head-up display with low-latency correction of projected images. For this purpose, the driver is equipped with a camera mounted on a bracket near his head. A camera observing the driver is located in the vehicle interior. Time warping techniques known from the field of virtual reality displays are used to calculate the corrections.

[0017] Even when using HUDs in data glasses, it's crucial that the display speed is high and that the delay between the system's detection of movement and the adjustment of the display image is short. If this isn't the case, it can lead to the aforementioned problems, such as motion sickness and misorientation between the physical and virtual worlds. Road characteristics also play a role, as they can cause vibrations, which can also impair the accuracy of the AR display.

[0018] Other challenges arise for different reasons:

[0019] It's known from the field of virtual and augmented reality that if images displayed to the viewer are generated at a rate of at least 90 Hz (≈11 milliseconds per image) and with a system latency of less than 20 ms, a lag-free user experience is achieved for the viewer. Research has shown that these requirements can be transferred to augmented reality displays in heads-up displays.

[0020] Due to the complex structure of these systems with different sensors and the coupling of the systems with different computing units for processing sensor data, the need to model vehicle motion, the need to predict image content due to vehicle motion and motion during display generation, etc., the latency of the overall system is however significantly greater than 20 milliseconds.

[0021] Known solutions have various disadvantages, which are identified within the scope of the present invention. In currently known HUDs, there is a need for further improvements in the HUD display while taking into account vehicle movements and driver's head movements.

[0022] Due to the long system chain, it takes some time (~150 to 250 ms) until the driver-relevant image is displayed. This results in a delay that manifests as a "floating" or "shaking" virtual image. The time delay associated with HUD display refers to the time offset between the image displayed in the HUD and the human perception of the surroundings.

[0023] A significant amount of computing effort is required to display images correctly positioned on the road from the user's perspective. This applies to the entire system chain, from position determination to displaying the rendered image on the HUD screen. On the other hand, there's the problem that lag-free AR displays aren't possible simply with fast computers and control units. Summary of the Invention

[0024] The technical problem to be solved by the present invention is to provide a solution, which can improve the above-mentioned problems.

[0025] This solution is based on a suitable strategy for synchronizing multiple processes. On the one hand, sensor data is detected, on the other, the vehicle status is predicted, and further, an augmented reality display is generated and output on the display. The overall system can therefore be divided into three components: sensors, applications, and displays.

[0026] This solution is particularly interesting for HUD display units. However, it can also be used for HMD display units (head-mounted displays), in particular data glasses.

[0027] In a preferred embodiment, the solution comprises a method for generating a positionally accurate display of additional information on a display unit of a vehicle, wherein the vehicle is equipped with a number of sensors for detecting the vehicle's state and surroundings. The sensor data is processed periodically. The method is characterized by precalculating the vehicle's state, wherein the precalculated state is transmitted (or transferred) to a rendering process for calculating the image to be displayed. The position (or position, orientation, or position) of the additional information in the displayed image is calculated based on at least the precalculated vehicle state during the rendering process. The calculated image is then post-processed. A transformation of the rendered image is performed, wherein the transformation calculation is determined by the updated vehicle state. The vehicle state is also updated during the rendering process, so that the vehicle's state is already detected again at the end of the rendering process. By precalculating the vehicle's state and using the updated state for post-processing, the overall latency of the display system can be improved. Consequently, "floating" or "shaking" of the image containing the additional information can be avoided.

[0028] It is possible to display a still image that is correctly positioned in the vehicle's surroundings with respect to the position of the vehicle's passengers, thereby also avoiding incorrect interpretation of the additional information and confusion for the driver.

[0029] Particularly advantageous is the precalculation of the state of the vehicle at the point in time at which the display of the additional information is displayed on the display unit, at which point in time the passenger will be looking at the display and the display should then be positioned correctly.

[0030] For this purpose, it may also be advantageous if the viewing direction (or line of sight) of a vehicle passenger is detected by at least one sensor and is taken into account in a rendering process for calculating the image to be displayed, wherein the driver's viewing direction is predicted, wherein the time at which the driver's viewing direction is predicted corresponds to the time at which the additional information is displayed on the display unit. Knowledge of the viewing direction is important in head-up displays, since the additional information cannot be displayed with the correct position across the entire field of view.

[0031] Another advantageous measure is that a front-buffer rendering process is used to generate the additional information for calculating the image, wherein the image is rendered directly into the video memory of the display unit. This significantly contributes to reducing latency.

[0032] In addition, in order to produce the appearance of additional information, at least the display unit can be operated in a "low persistence" (low persistence or short afterglow) display mode, in which the display of the image is dimmed (dunkel getastet or dark control) while the data is written to the video memory, and after the data is written, the image is displayed during a shorter phase of the image cycle than in normal operation until the start of the next writing process.

[0033] In a preferred variant, this can be achieved using an LCD display, whose background lighting is switched off during the writing of data into the video memory and switched on during short display periods, or using an OLED display, whose lighting phases are controlled so that it emits light only during short display periods.

[0034] In a specific example, the display operates at a refresh rate of at least 90 Hz (approximately 11 milliseconds per image) and is "triggered" by an application with a preset clock signal. In addition, the display operates in a "low persistence" mode, in which the display emits light for only 2 milliseconds (a total of 11 milliseconds at 90 Hz in this example). As a result, the appearance of motion blur can be reduced, and the "dimming" of the display can be used to build up the image. This corresponds to a "front buffer rendering" technique by the application. As a result, the latency (compared to conventional systems) can be reduced by a full frame (here 11 milliseconds).

[0035] In an extended variant, a calculation model is used to precalculate the vehicle's state, taking into account the time at which the sensor data is acquired and the system runtime for processing it. The minimum update rate for the application (including rendering) depends on the refresh rate of the display, but it also sets the timing for all other components in the system. Internal application processes precisely determine the time at which data from different sensors is required for further processing. Because the runtimes of the individual data streams are known, the unique "just-in-time" delivery of sensor data ensures that applications always operate with the latest data without incurring any additional delays.

[0036] By appropriately scheduling the individual application cycles based on knowledge of the runtimes of the participating system components, combined with rapid sensor-based model-based predictions of the vehicle state, the overall system latency can be reduced to a few milliseconds. This significantly improves the acceptance of such AR displays in HUDs.

[0037] To determine the viewing direction of a vehicle passenger, it is advantageous to detect the viewing direction using a number of IR cameras installed in the vehicle. The IR cameras determine the position of a number of light spots generated by corresponding light sources, which are mounted on brackets near the passenger's head. This data is then quickly available to the vehicle's computing device, which performs the prediction of the viewing direction.

[0038] Different sensor types operate with their own clock signals. However, when sensors provide redundant information and are combined at the application level, they are always synchronized with each other in time. This allows multiple sensors / data sources to describe the road surface, vehicle position, and so on, for example, to increase accuracy and / or frequency. This time synchronization can be achieved, for example, via a common clock generator, either within the application itself or externally.

[0039] The overall system can be made adaptive by using a clock generator within the application. This means that the time between the application's request for a data stream and its delivery by the sensor / model, and thus the overall system's runtime, can be measured within the application and reacted to. The system can thus even adapt itself to the respective situation (during runtime).

[0040] A device for implementing the method, which has at least one display unit (using which additional information can be displayed in the field of view of a vehicle passenger), is advantageously equipped with a number of sensors for detecting the vehicle state and the vehicle's surroundings and a computing device, which periodically processes the sensor data. In particular, the computing device is designed to precalculate the vehicle state and to perform a rendering process. The rendering process is designed such that the precalculated vehicle state is used to calculate the image to be displayed, the computing device is designed to detect the vehicle state again, and the computing device is further designed to post-process the rendered image, transforming the image content, wherein the transformation calculation is determined by the updated vehicle state.

[0041] A head-up display is generally used as the display unit.

[0042] The present invention can preferably be used in a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail below with reference to the drawings.

[0044] in:

[0045] Figure 1 A diagram showing a cockpit of a vehicle having a HUD display unit;

[0046] Figure 2 shows a diagram of a vehicle's cockpit with an IR camera for outside-inside tracking to detect head pose;

[0047] Figure 3 A block diagram of a vehicle's infotainment system is shown;

[0048] Figure 4 A system proposal showing different periodically running processes that interact for the computation and output of AR content; and

[0049] Figure 5 The time course of various cyclically running processes is shown. DETAILED DESCRIPTION

[0050] This specification illustrates the principles of the disclosure according to the present invention. It is therefore understood that those skilled in the art will be able to design the following different arrangements, which, although not explicitly described herein, embody the principles of the disclosure according to the present invention and should also be protected within their scope.

[0051] Figure 1A typical cockpit of a vehicle 10 is shown. A sedan Pkw is shown. However, any other vehicle is also conceivable as vehicle 10. Examples of further vehicles include buses, commercial vehicles, particularly trucks Lkw, campervans, agricultural machinery, construction machinery, rail vehicles, etc. The present invention can generally be used for land vehicles (including robots and rail vehicles), watercraft, and aircraft (including airplanes and drones).

[0052] The three display units of the infotainment system are shown in the cockpit. These are a touch-sensitive screen 30 located in the center console, an instrument cluster 110 located in the dashboard, and a heads-up display 20 located below the windshield and behind the instrument cluster 110. While driving, the center console is out of the driver's field of view. Therefore, additional information is displayed during driving via the HUD.

[0053] The touch-sensitive screen 30 is used, in particular, to operate functions of the vehicle 10 . For example, this operation can be used to control the vehicle 10 's radio, navigation system, playback of stored music, and / or air conditioning, other electronic devices, or other comfort functions or apps. In short, the term "infotainment system" is often used. In motor vehicles, particularly passenger cars, an infotainment system refers to the combination of the car radio, navigation system, hands-free system, driver assistance systems, and other functions in a central operating unit. The term "infotainment" is a portmanteau of the words "information" and "entertainment." To operate the infotainment system, a touch-sensitive screen 30 ("touch screen") is primarily used, particularly for easy viewing and operation by the driver and passenger of the vehicle 10 . Furthermore, mechanical operating elements, such as buttons, a rotary knob, or a combination thereof, such as a rotary knob, can be arranged below the screen 30 in an input unit 50 . Typically, steering wheel operation is also possible with certain parts of the infotainment system. This unit is not shown separately but is considered part of the input unit 50 .

[0054] Figure 2 The cockpit is shown from different angles. Reference numeral 152 denotes an IR detector installed in the instrument panel. These detectors are part of an exterior-interior tracking system that observes the driver or co-pilot. The purpose is to detect the exact head posture of the driver or co-pilot. This is done by measuring the intensity of the IR beam radiated by an infrared light source placed on the driver's glasses using the IR detector 152. Each time the head is turned, the different intensities of the IR beam are measured. The head posture can be inferred from the measured intensity values. An IR camera can be used as the IR detector. An embodiment is also conceivable in which the driver's head posture is detected using an interior space camera. In this case, there is no need to equip the driver with glasses. However, more complex image processing is required for this. Figure 2Also shown is the AR representation output by the HUD 20. This is a target arrow 22 and a frame marking the destination of the navigation route. Thus, the vehicle 10 has almost reached its destination and should stop at the point indicated by the target arrow 22. The AR representation is calculated so that its position is correctly located within the environment.

[0055] Figure 3 A block diagram of an infotainment system 200 is schematically shown, and some of its subsystems or applications are schematically illustrated. Two display units, a touch-sensitive display unit 30 and a head-up display (HUD) 20, a computing device 40, an operating unit 50, and a memory 60 are shown. The display unit 30 includes a display surface for displaying variable graphical information and an operating surface (touch-sensitive layer) arranged above the display surface for user command input.

[0056] The display units 20 and 30 are connected to the computing device 40 via a data line 70. The data line can be designed according to the LVDS (low voltage differential signal) standard. Via the data line 70, the display unit 30 receives control data for controlling the display surface of the touch screen 30 from the computing device 40. Via the data line 70, the control data of the input command is also transmitted from the touch screen 30 to the computing device 40. The fully rendered display image for the HUD 20 is also transmitted via the data line 70. The operating unit 50 includes the operating elements mentioned above, such as buttons, adjustment knobs, sliding regulators or rotary buttons, with the help of which the operator can make inputs through menu guidance. Input is generally understood to mean calling up selected menu options, as well as changing parameters, starting and stopping functions, etc. The storage device 60 is connected to the computing device 40 via a data line 80.

[0057] The other components of the infotainment system, namely the camera 150, radio 140, navigation device 130, telephone 120, and instrument cluster 110, are connected to the devices used to operate the infotainment system via a data bus 100. A high-speed variant of the CAN bus according to ISO standard 11898-2 is considered as data bus 100. Alternatively, a bus system based on Ethernet technology, such as BroadR-Reach, is also considered. Bus systems that transmit data via optical waveguides can also be used. Examples include the MOST bus (Media Oriented System Transport) or the D2B bus (Digital Home Bus). A vehicle measurement unit 170 is also connected to data bus 100. This vehicle measurement unit 170 is used to detect vehicle movement, particularly vehicle acceleration. This vehicle measurement unit can be configured as a conventional IMU unit, similar to an inertial measurement unit. An IMU unit typically contains an acceleration sensor and a rotational speed sensor, such as a laser gyroscope or a magnetometer gyroscope. It should also be noted that camera 150 can be configured as a conventional video camera. In this case, it records 25 full images per second, which corresponds to 50 half images per second in an interlaced recording mode. Alternatively, special cameras can be used that record more images per second to improve the accuracy of object detection in the case of faster-moving objects, or to record light in a spectrum other than the visible spectrum. Multiple cameras can be used to observe the surroundings. Furthermore, radar or lidar systems can be used, in addition or as an alternative, to implement or expand the observation of the surroundings. Odometer sensors can also be used, for example, to improve the accuracy of determining the vehicle's position. Examples of such sensors are wheel speed sensors, magnetometers, and accelerometers. For inward and outward wireless communication, vehicle 10 is equipped with a communication module 160. This module is also commonly referred to as an on-board connectivity unit (OCU). It can be designed for mobile radio communication, for example, according to the LTE (Long Term Evolution) or SG standards. It can also be designed for WLAN (Wireless LAN) communication, Bluetooth communication, communication with devices of passengers in the vehicle, or vehicle-to-vehicle communication, etc. Reference numeral 152 denotes an IR detector, which is located in the area of the dashboard. The IR detector can be designed as an IR camera. The camera doesn't necessarily produce images with very high resolution. For this purpose, a resolution of 382 x 288 pixels is quite sufficient for this application. Image evaluation can also be performed in the camera. Here, the coordinates of the various light spots recorded by the IR camera are determined. This will now be described in conjunction with data glasses.

[0058] Figure 4A system proposal for different periodically running processes that interact to calculate and output AR content is shown. Identical reference numerals denote the same components as in the previous figures. The flow of the various method steps is indicated by the direction of the arrows and proceeds in a clockwise direction. At the top, the reference numeral UE denotes the eyes of a vehicle occupant, typically the driver. The eye position, i.e., the viewing direction, is detected by the aforementioned IR detector 152. This requires image evaluation of the images recorded by the IR detector 152, as described above. For this purpose, the IR detector 152 can be assigned its own computing device. In a first approximation, the viewing direction can be set to be the same as the head posture. When observing the driver using a camera, more precise image evaluation of the eye portion allows the viewing direction to be determined more precisely by evaluating the eye position. In the block denoted by T152, regular head tracking and, if necessary, eye tracking are performed to detect the driver's viewing direction. Image detection can be performed at a frequency of 25 Hz or 30 Hz. This is insufficient to reduce the overall latency. Therefore, interpolation is performed to calculate intermediate values. This can be done in a similar way as in Hololens, where it is increased to 240Hz for head gestures.

[0059] The block that tracks vehicle position and orientation is designated by the reference symbol VT. This is accomplished by evaluating data in block T130, which is provided by the navigation system 130 and the odometer of vehicle 10. Detecting vehicle position can be performed accurately, but slowly. Detecting rotation in block T170 using the IMU measurement unit 170 can be performed quickly, but drift can occur. The VS block records current vehicle values and states that are important for detecting the vehicle's surroundings. Therefore, these values are also provided to block RW, which performs detection of vehicle 10's surroundings. Block RW includes block RW150, which performs dynamic environmental detection using sensors (camera 150, RADAR, LIDAR, etc.). Block RW also includes block RW130, which performs static environmental measurements by evaluating map material. Processing in blocks VT and RW also occurs periodically at their own frequencies, for example, 50 Hz for detecting vehicle position and 90 Hz for detecting vehicle rotation. The environmental detection may be performed at a frequency of 25 Hz or 30 Hz, for example.

[0060] All information calculated by the tracking blocks T152, VT and RW is forwarded to the computing device 40. The sensors all work with their own clock cycles. However, when the sensors provide redundant information and are merged at the application level, the sensors are always synchronized with each other in time. Thus, multiple sensors / data sources can be used to describe the road surface, vehicle position, etc., in order to increase the accuracy and / or frequency, for example. For example, this time synchronization can be achieved by a common clock generator, for example by the application itself or externally. For this purpose, Figure 4 A block SY is provided in the . In this block, the data provided by the various sensors are adjusted. This block works as follows:

[0061] This block measures the cycle time of the results provided by the sensors. It links all cycle times to a common clock signal preset by the application program being processed in the computing unit 40. It then forwards the sensor data to the appropriate processing component of the application program at the correct time. If this is not possible due to the cycle time, a prediction of the sensor data is made. This prediction is calculated in advance for the time at which the data should be processed by the application program. For this purpose, sensor values from past cycles must be stored and used as the basis for the prediction, allowing for interpolation or extrapolation of intermediate values.

[0062] The application running in the computing unit 40 has the primary purpose of calculating AR content so that it is correctly displayed in the driver's field of view relative to the surrounding position. Rendering of the image to be displayed is also performed as part of the application at the 11ms rhythm mentioned at the beginning. To output the AR image, the "front buffer rendering" and "low persistence display" techniques known from virtual reality are used. Here, the fully rendered image is transmitted using an optimized "rendering pipeline." The "late latch" technique is also used in the rendering process.

[0063] Figure 5 Different processing cycles for generating AR images are also shown. Time is shown along the abscissa. Two consecutive tracking processes T for different sensors are shown in the upper row. i and T i+1 The next row shows two working cycles A of the application program processed by the computing unit 40. i and R i The application is actually divided into two parts. In Phase A i In the process, the state (visibility, position, content of the AR content, etc.) is updated. This description of the display in the virtual scene is independent of the movement of the vehicle and can therefore start in parallel with the sampling by the sensors.

[0064] For example, the sensor-based detection of longitudinal and lateral acceleration is performed by an IMU measurement unit 170. The driver makes requests to the vehicle regarding steering wheel angle, accelerator or brake pedal position, etc. These are also detected by sensors. The application generates a model-based determination / prediction of the vehicle's longitudinal and lateral acceleration. This calculation model is already known and can also be used for other purposes. The predicted data is then compared or even merged with the actually measured data.

[0065] At the same time, the 3D road surface is determined. This can be done using a stereo camera and appropriate image processing. Detailed map material can also be accessed here, based on precise position data, such as from real-time kinematic (RTK) measurements. To increase accuracy, the measured data can be compared / merged with map entries.

[0066] In LL i During the phase, the vehicle's rotation / orientation is predicted. This calculation is also based on a computational model that describes the vehicle's motion. The computational model can also be used to calculate the driver's gaze movement. i+1 At the display time in the , the vehicle's orientation is calculated in advance based on the previously determined longitudinal and lateral accelerations, the 3D road surface and the known system operating time. The same applies to the advance calculation of the viewing direction. i The previously calculated states are written into the memory and replace the previously measured values of these variables. i , the rendering of image i is based on the predicted position and orientation of the vehicle at the display time point and the predicted viewing angle of the driver.

[0067] At the same time, the current position of the vehicle is remeasured using the index i+1 by a sensor with a short cycle time, for example the IMU measurement unit 170. i+1 In this stage, the data is passed to the post-processing (re-projection) of the rendered image. In this stage, the post-processing of the rendered image frame i is still carried out. The operations performed in this stage are also known from VR technology and are generally called "time warping operations". The finished rendered image i is thus converted into image frame i+1. Then, in the subsequent stage FBR i+1In the process, the video data is written to the front cache of the graphics card of the HUD display unit 20. Therefore, the "front cache rendering" technology, which is also known from VR technology, is used. Here, the image is built up during the black screen time of the display. This technology is usually used in combination with the "low persistence display" technology. Here, the black screen time of the display 20 is greatly extended in this technology. The image remains dimmed for almost the entire image period; the image is only brightened at the end of the image period. Therefore, motion blur in the image can be avoided. However, the disadvantage is that the image is displayed darker. In this example, the image is only displayed during a phase of 2ms instead of 11ms. The phase in which the image is finally displayed is at Figure 5 LPD i+1 In summary, it can be said that sensor data detection occurs twice within an image period of 11 ms, and an image rendered using data from the first data detection is post-processed and displayed based on data from the second sensor data detection.

[0068] All examples and conditional references herein should not be construed as limiting the specific examples cited. For example, those skilled in the art will appreciate that the block diagrams shown herein are conceptual diagrams of exemplary circuit arrangements. In a similar manner, it can be seen that the flowcharts, state transition diagrams, pseudocode, etc. shown represent different variations for illustrating processes that can essentially be stored in a computer-readable medium and thus be implemented by a computer or processor. The objects mentioned in the patent claims may also explicitly refer to people.

[0069] It should be understood that the proposed method and related equipment can be implemented in different forms of hardware, software, firmware, special processors or their combination. The special processor may include an application specific integrated circuit (ASIC), a reduced instruction set computer (RISC) and / or a field programmable gate array (FPGA). The proposed method and equipment are preferably implemented as a combination of hardware and software. The software is preferably installed on a program storage device as an application program. Typically, it is a machine based on a computer platform having the following hardware, such as one or more central processing units (CPUs), random access memory (RAM) and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The different processes and functions described herein may be part of an application program or a part implemented by an operating system.

[0070] The present disclosure is not limited to the embodiments described herein, and there is room for various adjustments and modifications that may be considered by those skilled in the art based on their expertise and the content of the present disclosure.

[0071] In the exemplary embodiments, the present invention is explained in more detail using the example of use in a vehicle. Possible uses in aircraft and helicopters, such as during landing maneuvers or search missions, are also mentioned. The present invention is explained using the example of a HUD display unit. However, these principles can also be transferred to use in data glasses or other HMD display units.

[0072] Reference Signs List

[0073] 10 vehicles

[0074] 20 Heads-up display

[0075] 22 AR display

[0076] 30 touch-sensitive display unit

[0077] 40 computing units

[0078] 50 input units

[0079] 60 storage units

[0080] 70 Data line to the display unit

[0081] 80 Data lines to memory cells

[0082] 90 Data lines to input units

[0083] 100 data bus

[0084] 110 Instrument Cluster

[0085] 120 phone number

[0086] 130 Navigation Device

[0087] 140 Radio

[0088] 150 Camera

[0089] 152 IR Camera

[0090] 160 Communication Module

[0091] 170 Vehicle Measurement Unit

[0092] 200 Infotainment System

[0093] VT Vehicle Status Tracking

[0094] T130 Location Tracking

[0095] T152 Observation Direction Tracking

[0096] T170 Vehicle Dynamic Tracking

[0097] RW Environmental Tracking

[0098] RW150 Environmental Sensor Evaluation

[0099] RW130 Environmental Map

[0100] SY Synchronous

[0101] VS Vehicle Status

[0102] UE user's eyes

[0103] A i Application i

[0104] T i Sensor detection

[0105] T i+1 Sensor detection i+1

[0106] LL i Propagation of predicted vehicle states

[0107] LL i+1 Image post-processing

[0108] FBR i+1 Front cache rendering i+1

[0109] LPD i+1 Low persistence display i+1

Claims

1. A method for producing a positionally correct display of additional information on a display unit of a vehicle, wherein: The vehicle (10) is equipped with a number of sensors for detecting the position and orientation of the vehicle (10) and the environment, wherein the data from the sensors are processed periodically, wherein the position and orientation of the vehicle (10) are precalculated, wherein a calculation model is used for precalculating the position and orientation of the vehicle (10), in which the time points for detecting the sensor data and the system operating time for processing the sensor data are taken into account, wherein the precalculated position and orientation are transmitted to a rendering process for calculating an image to be displayed, and wherein the position of the appearance of additional information in the displayed image is calculated at least based on the precalculation of the position and orientation of the vehicle (10) during the rendering process, wherein the position and orientation of the vehicle (10) are detected again during the rendering process, wherein the image rendered during the rendering process is post-processed, wherein the image content is transformed using a time warping operation, wherein the transformation calculation is determined by the updated position and orientation of the vehicle (10).

2. The method according to claim 1, wherein The time point of the predicted position and orientation corresponds to the time point at which the appearance of the additional information is displayed on the display unit (20).

3. The method according to claim 1 or 2, wherein The viewing direction of a passenger of the vehicle (10) is detected by at least one sensor, wherein the viewing direction of the passenger of the vehicle (10) is taken into account in a rendering process for calculating an image to be displayed, wherein the viewing direction of a driver is predicted, wherein the time point at which the viewing direction of the driver is predicted corresponds to the time point at which the additional information is displayed via a display unit (20).

4. The method according to claim 1, wherein For computing the image, a front-buffer rendering process is implemented for generating a representation of the additional information, wherein the image is rendered directly into the video memory of the display unit (20).

5. The method according to claim 1, wherein In order to produce the appearance of the additional information, at least the display unit (20) can be operated in a "low-persistence" display mode.

6. The method according to claim 5, wherein: To implement a "low-persistence" display mode, either an LCD display or an OLED display is used, the backlighting of which is switched off during the writing of data into the video memory and switched on during short display periods; The lighting phases of the OLED display are controlled such that the OLED display emits light only during short display periods.

7. The method according to claim 3, wherein: The viewing direction of a passenger of the vehicle (10) is detected by using a number of IR cameras installed in the vehicle (10). The IR cameras determine the positions of a number of light spots generated by corresponding light sources, which are placed on brackets at the passenger's head.

8. A device for implementing the method according to any one of the preceding claims 1 to 7, the device having at least one display unit (20) with which the additional information can be displayed in the field of vision of a passenger of the vehicle (10); a certain number of sensors for detecting the position and orientation of the vehicle (10) and the surroundings of the vehicle (10) and a computing device (40), with which the data of the sensors are periodically processed, wherein The computing device (40) is designed to pre-calculate the position and orientation of the vehicle (10), using a computing model in which the time points for detecting sensor data and the system runtimes for processing the sensor data are taken into account, wherein the computing device (40) is further designed to perform a rendering process, wherein the rendering process is designed such that the pre-calculated position and orientation of the vehicle (10) are taken into account for calculating the image to be displayed, wherein the computing device (40) is designed to detect the position and orientation of the vehicle (10) again, and wherein the computing device (40) is further designed to post-process the rendered image, in which the image content is transformed using a time warping operation, wherein the transformation calculation is determined by the updated position and orientation of the vehicle (10).

9. The apparatus according to claim 8, wherein The device has at least one sensor for detecting a viewing direction of a passenger of the vehicle (10).

10. The apparatus according to claim 9, wherein At least one sensor is an infrared camera which records light spots of a certain number of infrared light sources which are mounted on a bracket at the head of the passenger.

11. The apparatus according to claim 9, wherein The calculation device (40) has an execution device for a prediction algorithm, using which the viewing direction is predicted, wherein the time at which the viewing direction is predicted corresponds to the time at which the appearance of the additional information is displayed on the display unit (20).

12. The apparatus according to claim 11, wherein The calculation device (40) is designed to calculate an image for the display of the additional information as a function of the predicted viewing direction.

13. The apparatus according to claim 8, wherein The display unit corresponds to a head-up display.

14. A vehicle, characterized in that: The vehicle (10) is equipped with a device according to any one of claims 8 to 13.

Images